A small freedom area RSS

A series of tricks and techniques I learned doing tiny GLSL demos

Sun, 07 Dec 2025 17:48:26 -0000

In the past two months or so, I spent some time making tiny GLSL demos. I wrote an article about the first one, Red Alp. There, I went into details about the whole process, so I recommend to check it out first if you're not familiar with the field.

We will look at 4 demos: Moonlight, Entrance 3, Archipelago, and Cutie. But this time, for each demo, we're going to cover one or two things I learned from it. It won't be a deep dive into every aspect because it would be extremely redundant. Instead, I'll take you along a journey of learning experiences.

Moonlight

Moonlight demo in 460 characters

// Moonlight [460] by bµg
// License: CC BY-NC-SA 4.0
void main(){vec3 o,p,u=vec3((P+P-R)/R.y,1),Q;Q++;for(float d,a,m,i,t;i++<1e2;p=t<7.2?Q:vec3(2,1,0),d=abs(d)*.15+.1,o+=p/m+(t>9.?d=9.,Q:p/d),t+=min(m,d))for(p=normalize(u)*t,p.z-=5e1,m=max(length(p)-1e1,.01),p.z+=T,d=5.-length(p.xy*=mat2(cos(t*.2+vec4(0,33,11,0)))),a=.01;a<1.;a+=a)p.xz*=mat2(8,6,-6,8)*.1,d-=abs(dot(sin(p/a*.6-T*.3),p-p+a)),m+=abs(dot(sin(p/a/5.),p-p+a/5.));o/=4e2;O=vec4(tanh(mix(vec3(-35,-15,8),vec3(118,95,60),o-o*length(u.xy*.5))*.01),1);}

Note

See it on its official page, or play with the code on its Shadertoy portage.

In Red Alp, I used volumetric raymarching to go through the clouds and fog, and it took quite a significant part of the code to make the absorption and emission convincing. But there is an alternative technique that is surprisingly simpler.

In the raymarching loop, the color contribution at each iteration becomes 1/d or c/d where d is the density of the material at the current ray position, and c an optional color tint if you don't want to work in grayscale level. Some variants exist, for example 1/d^2, but we'll focus on 1/d.

1/d explanation

Let's see how it looks in practice with a simple cube raymarch where we use this peculiar contribution:

One glowing and rotating cube

void main() {
    float d, t;
    vec3 o, p,
         u = normalize(vec3(P+P-R,R.y)); // screen to world coordinate

    for (int i = 0; i < 30; i++) {
        p = u * t; // ray position

        p.z -= 3.; // take a step back

        // Rodriguez rotation with an arbitrary angle of π/2
        // and unaligned axis
        vec3 a = normalize(cos(T+vec3(0,2,4)));
        p = a*dot(a,p)-cross(a,p);

        // Signed distance function of a cube of size 1
        p = abs(p)-1.;
        d = length(max(p,0.)) + min(max(p.x,max(p.y,p.z)),0.);

        // Maxed out to not enter the solid
        d = max(d,.001);

        t += d; // stepping forward by that distance

        // Our mysterious contribution to the output
        o += 1./d;
    }

    // Arbitrary scale within visible range
    O = vec4(o/200., 1);
}

Note

The signed function of the cube is from the classic Inigo Quilez page. For the rotation you can refer to Xor or Blackle article. For the general understanding of the code, see my previous article on Red Alp.

The first time I saw it, I wondered whether it was a creative take, or if it was backed by physical properties.

Let's simplify the problem with the following figure:

A ray passing by a radiating object

The glowing object sends photons that spread all around it. The further we go from the object, the more spread these photons are, basically following the inverse square law 1/r^2, which gives the photons density, where r is the distance to the target object.

Let's say we send a ray and want to know how many photons are present along the whole path. We have to "sum", or rather integrate, all these photons density measures along the ray. Since we are doing a discrete sampling (the dots on the figure), we need to interpolate the photons density between each sampling point as well.

Given two arbitrary sampling points and their corresponding distance d_n and d_{n+1}, any intermediate distance can be linearly interpolated with r=\mathrm{mix}(d_n,d_{n+1},t) where t is within [0,1]. Applying the inverse square law from before (1/r^2), the integrated photons density between these 2 points can be expressed with this formula (in t range):

v = \Delta t \int \frac{1}{\mathrm{mix}(d_n,d_{n+1},t)^2} dt

t being normalized, the \Delta t is here to covers the actual segment distance. With the help of Sympy we can do the integration:

>>> a, b, D, t = symbols('a b D t', real=True)
>>> mix = a*(1-t) + b*t
>>> D * integrate(1/mix**2, (t,0,1)).simplify()
 D
───
a⋅b

So the result of this integration is:

v = \frac{\Delta t}{d_{n}d_{n+1}}.

Now the key is that in the loop, \Delta t stepping is actually d_{n+1}, so we end up with:

v = \frac{\Delta t}{d_{n}\Delta t} = \frac{1}{d_n}

And we find back our mysterious 1/d. It's "physically correct", assuming vacuum space. Of course, reality is more complex, and we don't even need to stick to that formula, but it was nice figuring out that this simple fraction is a fairly good model of reality.

Going through the object

In the cube example we didn't go through the object, using max(d, .001). But if we were to add some transparency, we could have used d = A*abs(d)+B instead, where A could be interpreted as absorption and B the pass-through, or transparency.

One glowing, transparent, and rotating cube; A=0.4, B=0.1

I first saw this formula mentioned in Xor article on volumetric. To understand it a bit better, here is my intuitive take: the +B causes a potential penetration into the solid at the next iteration, which wouldn't happen otherwise (or only very marginally). When inside the solid, the abs(d) causes the ray to continue further (by the amount of the distance to the closest edge). Then the multiplication by A makes sure we don't penetrate too fast into it; it's the absorption, or "damping".

This is basically the technique I used in Moonlight to avoid the complex absorption/emission code.

Entrance 3

Entrance 3 demo in 465 characters

// Entrance 3 [465] by bµg
// License: CC BY-NC-SA 4.0
#define V for(s++;d<l&&s>.001;q=abs(p+=v*s)-45.,b=abs(p+vec3(mod(T*5.,80.)-7.,45.+sin(T*10.)*.2,12))-vec3(1,7,1),d+=s=min(max(p.y,-min(max(abs(p.y+28.)-17.,abs(p.z+12.)-4.),max(q.x,max(q.y,q.z)))),max(b.x,max(b.y,b.z))))
void main(){float d,s,r=1.7,l=2e2;vec3 b,v=b-.58,q,p=mat3(r,0,-r,-1,2,-1,b+1.4)*vec3((P+P-R)/R.y*20.4,30);V;r=exp(-d*d/1e4)*.2;l=length(v=-vec3(90,30,10)-p);v/=l;d=1.;V;r+=50.*d/l/l;O=vec4(pow(mix(vec3(0,4,9),vec3(80,7,2),r*r)*.01,p-p+.45),1);}

Note

See it on its official page, or play with the code on its Shadertoy portage.

This demo was probably one of the most challenging, but I'm pretty happy with its atmospheric vibe. It's kind of different than the usual demos for this size.

I initially tried with some voxels, but I couldn't make it work with the light under 512 characters (the initialization code was too large, not the branchless DDA stepping). It also had annoying limitations (typically the animation was unit bound), so I fell back to a classic raymarching.

The first thing I did differently was to use an L-∞ norm instead of an euclidean norm for the distance function: every solid is a cube so it's appropriate to use simpler formulas.

For the light, it's not an illusion, it's an actual light: after the first raymarch to a solid, the ray direction is reoriented toward the light and the march runs again (it's the V macro). Hitting a solid or not defines if the fragment should be lighten up or not.

Mobile bugs

A bad surprise of this demo was uncovering two driver bugs on mobile:

One with tricky for-loop compounds on Snapdragon/Adreno because I was trying hard to avoid the macros and functions.
One with chained assignments on Imagination/PowerVR (typically affect Google Pixel Pro 10).

The first was worked around with the V macro (actually saved 3 characters in the process), but the 2nd one had to be unpacked and made me lose 2 characters.

Isometry

Another thing I studied was how to set up the camera in a non-perspective isometric or dimetric view. I couldn't make sense of the maths from the Wikipedia page (it just didn't work), but Sympy rescued me again:

# Counter-clockwise rotation
a, ax0, ax1, ax2 = symbols('a ax0:3')
c, s = cos(a), sin(a)
k = 1-c
rot = Matrix(3,3, [
    # col 1            col 2              # col 3
    ax0*ax0*k + c,     ax0*ax1*k + ax2*s, ax0*ax2*k - ax1*s, # row 1
    ax1*ax0*k - ax2*s, ax1*ax1*k + c,     ax1*ax2*k + ax0*s, # row 2
    ax2*ax0*k + ax1*s, ax2*ax1*k - ax0*s, ax2*ax2*k + c      # row 3
])

# Rotation by 45° on the y-axis
m45 = rot.subs({a:rad(-45), ax0:0, ax1:1, ax2:0})

# Apply the 2nd rotation on the x-axis to get the transform matrices for two
# classic projections
# Note: asin(tan(rad(30))) is the same as atan(sin(rad(45)))
isometric = m45 * rot.subs({a:asin(tan(rad(30))), ax0:1, ax1:0, ax2:0})
dimetric  = m45 * rot.subs({a:         rad(30),   ax0:1, ax1:0, ax2:0})

Inspecting the matrices and factoring out the common terms, we obtain the following transform matrices:

M_{iso} = \sqrt{2}\sqrt{3}\begin{bmatrix} \sqrt{3} & -1 & \sqrt{2} \\ 0 & 2 & \sqrt{2} \\ -\sqrt{3} & -1 & \sqrt{2} \end{bmatrix} \text{ and } M_{dim} = \frac{4}{\sqrt{2}}\begin{bmatrix} 2 & -1 & \sqrt{3} \\ 0 & \sqrt{6} & \sqrt{2} \\ -2 & -1 & \sqrt{3} \end{bmatrix}

The ray direction is common to all fragments, so we use the central UV coordinate (0,0) as reference point. We push it forward for convenience: (0,0,1), and transform it with our matrix. This gives the central screen coordinate in world space. Since the obtained point coordinate is relative to the world origin, to go from that point to the origin, we just have to flip its sign. The ray direction formula is then:

d_{iso} = -M_{iso} \begin{bmatrix}0 \\ 0 \\ 1\end{bmatrix} = -\frac{\sqrt{3}}{3}\begin{bmatrix}1 \\ 1 \\ 1\end{bmatrix} \text{ and } d_{dim} = -M_{dim} \begin{bmatrix}0 \\ 0 \\ 1\end{bmatrix} = -\frac{1}{4} \begin{bmatrix}\sqrt{6} \\ 2 \\ \sqrt{6}\end{bmatrix}

To get the ray origin of every other pixel, the remaining question is: what is the smallest distance we need to step back the screen coordinates such that, when applying the transformation, the view wouldn't clip into the ground at y=0.

This requirement can be modeled with the following expression:

M \begin{bmatrix}x \\ -1 \\ z\end{bmatrix} > 0

The -1 being the lowest y-screen coordinate (which we don't want into the ground). The lazy bum in me just asks Sympy to solve it for me:

x, z = symbols("x z", real=True)
u = m * Matrix([x, -1, z])
uz = solve(u[1] > 0, z)

We get z>\sqrt{2} for isometric, and z>\sqrt{3} for dimetric.

With an arbitrary scale S of the coordinate we end up with the following:

const float S = 50.;
vec2 u = (P+P-R)/R.y * S; // scaled screen coordinates

float A=sqrt(2.), B=sqrt(3.);

// Isometric
vec3 rd = -vec3(1)/B,
     ro = mat3(B,0,-B,-1,2,-1,A,A,A)/A/B * vec3(u, A*S + eps);

// Dimetric
vec3 rd = -vec3(B,A,B)/A/2.,
     ro = mat3(2,0,-2,-1,A*B,-1,B,A,B)/A/2. * vec3(u, B*S + eps);

The eps is an arbitrary small value to make sure the y-coordinate ends up above 0.

In Entrance 3, I used a rough approximation of the isometric setup.

Archipelago

Archipelago demo in 472 characters

// Archipelago [472] by bµg
// License: CC BY-NC-SA 4.0
#define r(a)*=mat2(cos(a+vec4(0,11,33,0))),
void main(){vec3 p,q,k;for(float w,x,a,b,i,t,h,e=.1,d=e,z=.001;i++<50.&&d>z;h+=k.y,w=h-d,t+=d=min(d,h)*.8,O=vec4((w>z?k.zxx*e:k.zyz/20.)+i/1e2+max(1.-abs(w/e),z),1))for(p=normalize(vec3(P+P-R,R.y))*t,p.zy r(1.)p.z+=T+T,p.x+=sin(w=T*.4)*2.,p.xy r(cos(w)*e)d=p.y+=4.,h=d-2.3+abs(p.x*.2),q=p,k-=k,a=e,b=.8;a>z;a*=.8,b*=.5)q.xz r(.6)p.xz r(.6)k.y+=abs(dot(sin(q.xz*.4/b),R-R+b)),k.x+=w=a*exp(sin(x=p.x/a*e+T+T)),p.x-=w*cos(x),d-=w;}

Note

See it on its official page, or play with the code on its Shadertoy portage.

For this infinite procedurally generated Japan, I wanted to mark a rupture with my red/orange obsession. Technically speaking, it's actually fairly basic if you're familiar with Red Alp. I used the same noise for the mountains/islands, but the water uses a different noise.

The per octave noise curve is w=exp(sin(x)), with the particularity of shifting the x coordinate with its derivative: x-=w*cos(x). This is some form of domain warping that gives the nice effect here. When I say x, I'm really referring to the x-axis position. It is not needed to work with the z-component (xz forms the flat plane) because each octave of the fbm has a rotation that "mixes" both axis, so z is actually backed in x.

w=exp(sin(x))

Note

I didn't come up with the formula, but found it first one this video by Acerola. I don't know if he's the original author, but I've seen the formula being replicated in various places.

Cutie

Cutie demo in 602 characters

// Cutie [602] by bµg
// License: CC BY-NC-SA 4.0
#define V vec3
#define L length(p
#define C(A,B,X,Y)d=min(d,-.2*log2(exp2(X-L-A)/.2)+exp2(Y-L-B)/.2)))
#define H(Z)S,k=fract(T*1.5+s),a=V(1.3,.2,Z),b=V(1,.3*max(1.-abs(3.*k-1.),z),Z*.75+3.*max(-k*S,k-1.)),q=b*S,q+=a+sqrt(1.-dot(q,q))*normalize(V(-b.y,b.x,0)),C(a,q,3.5,2.5),C(q,a-b,2.5,2.)
void main(){float i,t,k,z,s,S=.5,d=S;for(V p,q,a,b;i++<5e1&&d>.001;t+=d=min(d,s=L+V(S-2.*p.x,-1,S))-S))p=normalize(V(P+P-R,R.y))*t,p.z-=5.,p.zy*=mat2(cos(vec4(1,12,34,1))),p.xz*=mat2(cos(sin(T)+vec4(0,11,33,0))),d=1.+p.y,C(z,V(z,z,1.2),7.5,6.),s=p.x<z?p.x=-p.x,z:H(z),s+=H(1.);O=vec4(V(exp(-i/(s>d?1e2:9.))),1);}

Note

See it on its official page, or play with the code on its Shadertoy portage.

Here I got cocky and thought I could manage to fit it in 512 chars. I failed, by 90 characters. I did use the smoothmin operator for the first time: every limb of the body of Cutie is composed of two spheres creating a rounded cone (two sphere of different size smoothly merged like metaballs).

2 spheres merging using the smin operator

Then I used simple IK kinetics for the animation. Using leg parts with a size of 1 helped simplifying the formula and make it shorter.

You may be wondering about the smooth visuals itself: I didn't use the depth map but simply the number of iterations. Due to the nature of the raymarching algorithm, when a ray passes close to a shape, it slows down significantly, increasing the number of iterations. This is super useful because it exaggerate the contour of the shapes naturally. It's wrapped into an exponential, but i defines the output color directly.

What's next

I will continue making more of those, keeping my artistic ambition low because of the 512 characters constraint I'm imposing on myself.

You may be wondering why I keep this obsession about 512 characters, and many people called me out on this one. There are actually many arguments:

A tiny demo has to focus on one or two very scoped aspects of computer graphics, which makes it perfect as a learning support.
It's part of the artistic performance: it's not just techniques and visuals, the wizardry of the code is part of why it's so impressive. We're in an era of visuals, people have been fed with the craziest VFX ever. But have they seen them with a few hundreds bytes of code?
The constraint helps me finish the work: when making art, there is always this question of when to stop. Here there is an intractable point where I just cannot do more and I have to move on.
Similarly, it prevents my ambition from tricking me into some colossal project I will never finish or even start. That format has a ton of limitations, and that's its strength.
Working on such a tiny piece of code for days/weeks just brings me joy. I do feel like a craftsperson, spending an unreasonable amount of time perfecting it, for the beauty of it.
I'm trying to build a portfolio, and it's important for me to keep it consistent. If the size limit was different, I would have done things differently, so I can't change it now. If I had hundreds more characters, Red Alp might have had birds, the sky opening to lit a beam of light on the mountains, etc.

Why 512 in particular? It happens to be the size of a toot on my Mastodon instance so I can fit the code there, and I found it to be a good balance.

Text rendering and effects using GPU-computed distances

Sat, 01 Nov 2025 17:20:06 -0000

Text rendering is cursed. Anyone who has worked on text will tell you the same; whether it's about layout, bi-directional, shaping, Unicode, or the rendering itself, it's never a completely solved problem. In my personal case, I've been working on trying to render text in the context of a compositing engine for creative content. I needed crazy text effects, and I needed them to be reasonably fast, which implied working with the GPU as much as possible. The distance field was an obvious requirement because it unlocks anti-aliasing and the ability to make many great effects for basically free.

In this article, we will see how to compute signed distance field on the GPU because it's much faster than doing it on the CPU, especially when targeting mobile devices. We will make the algorithm decently fast, then after lamenting about the limitations, we will see what kind of effects this opens up.

Progressive build of the 'あ' glyph from the Mochiy Pop One font

Extraction of glyph outlines

Non-bitmap fonts contain glyphs defined by outlines made of closed sequences of lines and (quadratic or cubic) Bézier curves. Extracting them isn't exactly complicated: FreeType or ttf-parser typically expose a way to do that.

For the purpose of this article, we're going to hard code the list of the Bézier curves inside the shader, but of course in a more serious setup those would be uploaded through storage buffers or similar.

Using this tiny program, a glyph can be dumped as series of outlines into a fixed size array:

struct Bezier {
    vec2 p0; // start point
    vec2 p1; // control point 1
    vec2 p2; // control point 2
    vec2 p3; // end point
};

#define N 42
#define NC 2
const int glyph_A_count[2] = int[](33, 9);
const Bezier glyph_A[42] = Bezier[](
    Bezier(vec2(  0.365370,  -0.570817), vec2(  0.374708,  -0.631518), vec2(  0.332685,  -0.687549), vec2(  0.339689,  -0.748249)),
    Bezier(vec2(  0.339689,  -0.748249), vec2(  0.339689,  -0.748249), vec2(  0.344358,  -0.764591), vec2(  0.351362,  -0.771595)),
    Bezier(vec2(  0.351362,  -0.771595), vec2(  0.384047,  -0.822957), vec2(  0.402724,  -0.855642), vec2(  0.442412,  -0.885992)),
    // ...
);

glyph_A_count contains how many Bézier curves there is for each sub-shape composing the glyph, and glyph_A contains that list of Bézier cubic curves.

Even though glyphs are also composed of lines and quadratic curves, we expand them all into cubics: "who can do more can do less".

We use these formulas to respectively expand lines and quadratics into cubics:

\begin{aligned} B_1 &= \begin{bmatrix} P_0 \\ \mathrm{mix}(P_0, P_1, 1/3) \\ \mathrm{mix}(P_0, P_1, 2/3) \\ P_1 \end{bmatrix} \\ B_2 &= \begin{bmatrix} P_0 \\ \mathrm{mix}(P_0, P_1, 2/3) \\ \mathrm{mix}(P_1, P_2, 1/3) \\ P_2 \end{bmatrix} \end{aligned}

Where P_n are the Bézier control points.

For simplicity and because we want to make sure the most complex case is well tested, we will stick to this approach in this article. But it also means there is a lot of room for further optimizations. Since solving linear and quadratics is much simpler, this is left as an exercise for the reader.

Warning

You may be tempted to upload the polynomial form of these curves directly to save some computation in the shader. Don't. You will lose the exact stitching property because one evaluated polynomial end B_n(1) will not necessary match the next polynomial start B_{n+1}(0). This makes artificial "precision holes" that will break rendering in obscure ways.

Signed distance to the shape

In the previous article, we saw how to get the distance to a cubic Bézier curve. Each glyph being composed of multiple outlines, we can simply run over all of them and pick the shortest distance.

float get_distance(vec2 p, Bezier buf[N], int counts[NC]) {
    int base = 0;
    float dist = 1e38;

    for (int j = 0; j < NC; j++) {
        int count = counts[j];

        for (int i = 0; i < count; i++) {
            Bezier b = buf[base + i];
            float d = bezier_sq(p, b.p0, b.p1, b.p2, b.p3);
            dist = min(dist, d);
        }

        base += count;
    }

    return sqrt(dist);
}

Where bezier_sq() is the distance to the Bézier curve, squared, as defined in the previous article.

Distance to the 'A' glyph from the Virgil font

This works just fine, but as you can imagine, it's not cheap to solve so many distances per pixel. A first straightforward optimization would be to ignore any curve with a bounding box further than our currently best distance, because none of them can give a shorter one:

Box distance optimization

Where each box encloses a Bézier curve like this:

Most naive/conservative bounding box

We could use a tighter bound but it would require more computation so this felt like a good trade-off.

Implementing this in the inner loop is pretty simple:

for (int i = 0; i < count; i++) {
    Bezier b = buf[base + i];
    vec2 p0=b.p0, p1=b.p1, p2=b.p2, p3=b.p3;

    // Distance to box (0 if inside), squared
    vec2 q0 = min(p0, min(p1, min(p2, p3)));
    vec2 q1 = max(p0, max(p1, max(p2, p3)));
    vec2 v = max(abs(q0+q1-p-p)-q1+q0, 0.)*.5;
    float h = dot(v,v);

    // We can't get a shorter distance than h if we were to compute the
    // distance to that curve
    if (h > dist)
        continue;

    float d = bezier_sq(p, p0, p1, p2, p3);
    dist = min(dist, d);
}

The distance to the bounding box formula comes from this explicative video by Inigo Quilez (the basic one, without the inside distance), adapted to the Bézier control point coordinates.

This saves a lot of computation in certain cases, but the worst case is still pretty terrible, as shown by the heat map of this 'C' glyph:

Heat map of how many distances are evaluated

Indeed sometimes, it takes a long time to reach a good Bézier curve that is small enough to disregard most of the others. We observe it the further we go away from the beginning of the shape.

So the next step is to find a good initial candidate. One cheap way to do that is to first compute the distance to the center point of each curve, and pick the smallest:

// Find a good initial guess
int best = 0;
float boxd = 1e38;
for (int i = 0; i < count; i++) {
    Bezier b = buf[base + i];
    vec2 p0=b.p0, p1=b.p1, p2=b.p2, p3=b.p3;
    vec2 q0 = min(p0, min(p1, min(p2, p3)));
    vec2 q1 = max(p0, max(p1, max(p2, p3)));
    vec2 v = (q0+q1)*.5 - p;
    float h = dot(v,v);
    if (h < boxd)
        best=i, boxd=h;
}

// Initial guess
Bezier bb = buf[base + best];
dist = min(dist, bezier_sq(p, bb.p0, bb.p1, bb.p2, bb.p3));

for (int i = 0; i < count; i++) {
    if (i == best) // We already computed this one
        continue;

    Bezier b = buf[base + i];
    vec2 p0=b.p0, p1=b.p1, p2=b.p2, p3=b.p3;
    // ...

This optimization is immediately reflected on the heat map, where only the central point seems to become a critical point (this glyph is a pathological case as it forms a circle):

Heat map with a rough initial guess

Winding number

The last step is to figure out whether we are inside or outside the shape. There are two schools here, the even-odd and the non-zero rules. We'll pick the latter because that's the expectation in the case of font rendering.

In deconstructing Bézier curves, we explained the theory of that specific algorithm so we're not going to dive into the details again. The basic idea is to strike a ray in one direction from our current position, and get how many times we cross a given curve. Here we will arbitrarily choose a horizontal ray line y = P_y where P is our current coordinate.

The topology of each curve can hint us on whether it's worth considering it or not. For example, if every control point is above or below our current position, it can be ignored. We can store all the signs in a mask and bail out as soon as the ray is either completely below or completely above the bounding box of the curve:

int signs = int(p0.y < p.y)
          | int(p1.y < p.y) << 1
          | int(p2.y < p.y) << 2
          | int(p3.y < p.y) << 3;
if (signs == 0 || signs == 15) // all signs are identical
    return 0;

Each sign indicates the position of the control point with regard to the ray. We can use the relative position of the starting point as a reference for the overall orientation (if there is a crossing, we know it will come from below or above):

int inc = (signs & 1) == 0 ? 1 : -1;

We also need to convert the Bézier curves to the usual polynomial at^3+bt^2+ct+d:

vec2 a = -p0 + 3.*(p1 - p2) + p3,
     b = 3. * (p0 - 2.*p1 + p2),
     c = 3. * (p1 - p0),
     d = p0 - p;

Then we can find the y-roots and check every point on the x-axis. For every crossing point (at most 3), we switch the sign:

    float r[5];
    int count = root_find3(r, a.y, b.y, c.y, d.y);
    vec3 t = vec3(r[0], r[1], r[2]);
    vec3 v = ((a.x*t + b.x)*t + c.x)*t + d.x;
    if (count > 0 && v.x >= 0.) w += inc;
    if (count > 1 && v.y >= 0.) w -= inc;
    if (count > 2 && v.z >= 0.) w += inc;

Since we already have a 5th degree root finder from the previous article, we just have to build a tiny version for the 3rd degree:

int root_find3(out float r[5], float a, float b, float c, float d) {
    float r2[5];
    int n = root_find2(r2, 3.*a, b+b, c);
    return cy_find5(r, r2, n, 0., 0., a, b, c, d);
}

Note

Our root finder doesn't return roots outside [0,1] so no filtering is required.

To summarize:

int bezier_winding(vec2 p, vec2 p0, vec2 p1, vec2 p2, vec2 p3) {
    int w = 0;
    int signs = int(p0.y < p.y)
              | int(p1.y < p.y) << 1
              | int(p2.y < p.y) << 2
              | int(p3.y < p.y) << 3;
    if (signs == 0 || signs == 15)
        return 0;
    int inc = (signs & 1) == 0 ? 1 : -1;
    vec2 a = -p0 + 3.*(p1 - p2) + p3,
         b = 3. * (p0 - 2.*p1 + p2),
         c = 3. * (p1 - p0),
         d = p0 - p;
    float r[5];
    int count = root_find3(r, a.y, b.y, c.y, d.y);
    vec3 t = vec3(r[0], r[1], r[2]);
    vec3 v = ((a.x*t + b.x)*t + c.x)*t + d.x;
    if (count > 0 && v.x >= 0.) w += inc;
    if (count > 1 && v.y >= 0.) w -= inc;
    if (count > 2 && v.z >= 0.) w += inc;
    return w;
}

For every sub-shape, we can accumulate the winding number, and use it at the end to decide whether we're inside or outside:

float get_distance(vec2 p, Bezier buf[N], int counts[NC]) {
    int w = 0;
    int base = 0;
    float dist = 1e38;

    for (int j = 0; j < NC; j++) {
        int count = counts[j];

        // Get the sign of the distance
        for (int i = 0; i < count; i++) {
            Bezier b = buf[base + i];
            w += bezier_winding(p, b.p0, b.p1, b.p2, b.p3);
        }

        // ...
    }

    // Positive outside, negative inside
    return (w != 0 ? -1. : 1.) * sqrt(dist);
}

And voilà:

Signed distance to the 'A' glyph from the Virgil font

Warning

This winding number logic might be too fragile: it doesn't cover potential degenerate cases such as horizontal tangents / duplicated roots. But for some reason, while I fought these issues for years, none of the weird corner cases seemed to glitch in my extensive tests, probably because the root finder is more resilient than what I was using before.

Limitations

Wicked curves

This may look satisfying, but it's only the beginning of the problems. For example, variadic fonts are typically following chaotic patterns:

The glyph 'e' in the Quicksand font

In addition to the self overlapping part, notice the reverse folding triangle on the right.

This completely wreck the distance field:

Glyph with a broken SDF due to overlaps

Even with a simple character display (meaning something that doesn't exploit the wide range of effects available with an SDF), it starts to glitch:

Glitching glyph due to broken SDF

Little "cracks" should appears around the overlaps. This can be mitigated by lowering the distance by a tiny constant to avoid the zero-crossing, but it impacts the overall glyph (it gets more bold).

And it's not just because of variadic problem, sometimes designers rely on overlaps for simplicity:

The glyph 't' in the Quicksand font

And sometimes... well let's say they have a legitimate reason to do it:

A Bengali glyph

This is not something that can be addressed easily.

For example, take these two overlapping shapes:

Distance inside two overlapping shapes

We see that the actual distance (white circle) is not the smallest distance to either shape, and it's not even the smallest distance to any edge: it is at an intersection point between two curves, which we do not have. Here we're dealing with line segments, but with cubic curves, the problem explodes in complexity.

At this point, we need another strategy, like feeding the GPU renderer with preprocessed outline-only curves. Many people rely on curves flattening to address this issue. This is unfortunately yet another field of research that we're not going to explore this time.

Inigo talked about the combination of signed distance if you want some ideas, but aside from the first one (giving up), none seems particularly applicable here.

Atlas and overlapping distances

Some effects such as blur or glow expand beyond the boundaries of the characters, so the distance field needs to be larger than the glyph itself. This means when an effect spread too large, there will be an overlap. If we're making an effect on a word, the distance field must be the unified version of all the word glyphs (or sometimes even the sentence). The classic approach of an atlas of glyph distances will not work reliably.

In the following illustration, a geometry per glyph is used, each geometry is enlarged to account for the larger distance field, and we end up with potential overlaps when applying effects.

Overlapping character geometries due to larger distance

Rounded corners

Like all distance maps, it suffers from the same limitations. The most common one is the rounded corners problem. This is typically addressed using a multi-channel signed distance field generator, but it's hard for me to tell how accessible it is for a portage on the GPU.

msdfgen demonstration of corners improvement

Note

This problem only appears with intermediate textures. When computing exact distances like here directly in the shaders, this is not an issue.

Effects

Despite all these limitations, we can already do so much, so let's close this article on a positive note. This is not done here, but all of these effects are free as soon as we have the distance field stored in an intermediate texture.

First, we have anti-aliasing / blur:

AA / blur effect

I wrote a dedicated article on the subject of AA (and blur) on SDF if you want more information on how to achieve that.

The shape can also be drastically altered with a simple operator such as "rounding":

d -= rounding;

Rounding effect

This is the same technique we suggested to cover up for the overlap glitch earlier, just rebranded as an effect.

In the same spirit we can also create an outline stroke (on the outer edge to preserve the original glyph design):

Outline effect

This is sooo useful because it makes it possible for our text to be visible no matter what the background is. So many editors don't have this feature because it's hard and expensive to do correctly. Given a distance field though, all we have to do is this (which also includes anti-aliasing on every border):

float aa = fwidth(d); // pixel width estimates
float w = aa * .5; // half diffuse width
vec2 b = vec2(0,1)*outline - d; // inner and outer boundaries; vec2(-1,0) for inner, vec2(-.5,.5) for centered
float inner_mask = smoothstep(-w, w, b.x); // cut-off between the outline and the outside (whole shape w/ outline)
float outer_mask = smoothstep(-w, w, b.y); // cut-off between the fill color and the outline (whole shape w/o outline)
float outline_mask = outer_mask - inner_mask;
vec3 o = (inner_color*inner_mask + outline_color*outline_mask) * outer_mask;

We can also dig into our character with d = abs(d)-ring:

Ring effect

And maybe apply some glow to create a neon effect:

Ring combined with a neon/glow effect

float glow_power = glow * exp(-max(d, 0.) * 10.);
o += glow_color * glow_power;

We could also do drop shadows, all sorts of distortions, or so many other creative way exploiting this distance. You get the idea: it is fundamental as soon as you want fast visual effects.

Conclusion

This article is the last of the series on 2D rendering for me. I've wanted to share this experience and knowledge after many years of struggling (mostly alone) on these issues. I wish I could have succeeded in providing a good free and open-source text effects rendering engine to compete with the industry standards. (Un)fortunately for me, the adventure stops here, but I hope this will benefit creators and future tinkerers interested in the subject.

Fast calculation of the distance to cubic Bezier curves on the GPU

Sat, 18 Oct 2025 09:21:56 -0000

Bézier curves are a core building block of text and 2D shapes rendering. There are several approaches to rendering them, but one especially challenging problem, both mathematically and technically, is computing the distance to a Bézier curve. For quadratic curves (one control point), this is fairly accessible, but for cubic (two control points) we're going to see why it is so hard.

A glyph from the Virgil font, composed of multiple Bézier curves

Having this distance field opens up many rendering possibilities. It's hard, but it's possible; here is a live proof:

Distance to a cubic Bézier curve

In this visualization, I'm borrowing your device resources to compute the distance to the curve for every single pixel. The yellow points are the control points of the curve (in white) and the blue zone is a representation of the distance field.

Note

All the demos and code in this article are self-contained GLSL fragment shaders. Most of the code can be found in the article, but feel free to inspect the source code of any of these WebGL demo for the complete code. They can be run verbatim using ShaderWorkshop.

The basic maths

In a previous article, we explained that a Bézier curve can be expressed as a polynomial. In our case, a cubic polynomial:

B_3(t) = \textbf{a}t^3 + \textbf{b}t^2 + \textbf{c}t + \textbf{d}

Where a, b, c and d are the vector coefficients derived from the start (P_0), end (P_3), and control points (P_1, P_2) using the following formulas (you can refer to the previous article for details):

\begin{aligned} \textbf{a} &= -P_0 + 3(P_1-P_2) + P_3 \\ \textbf{b} &= 3P_0 - 6P_1 + 3P_2 \\ \textbf{c} &= -3P_0 + 3P_1 \\ \textbf{d} &= P_0 \end{aligned}

For a given point p in 2D space, the distance to that Bézier curve can be expressed as a length between our curve and p:

\begin{aligned} d(t) &= ||B_3(t) - \textbf{p}|| \\ &= ||\textbf{a}t^3 + \textbf{b}t^2 + \textbf{c}t + \textbf{d} - \textbf{p}|| \end{aligned}

Our goal is to find the t value where d(t) is the smallest.

The length formula has an annoying square root, so we start with the distance squared for simplicity, which we are going to unroll:

\begin{aligned} D(t) &= d(t)^2 \\ &= ||\textbf{a}t^3 + \textbf{b}t^2 + \textbf{c}t + \textbf{d} - \textbf{p}||^2 \\ &= (a_xt^3 + b_xt^2 + c_xt + d_x - p_x)^2 + (a_yt^3 + b_yt^2 + c_yt + d_y - p_y)^2 \end{aligned}

The derivative of that function will allow us to identify critical points: that is, points where the distance starts growing or reducing. Said differently, solving D'(t)=0 will identify all the maximums and minimums (we're interested in the latter) of D(t) (and thus d(t) as well).

It is a bit convoluted in our case but straightforward to compute:

\begin{aligned} D'(t) &= 2(3a_xt^2 + 2b_xt + c_x)(a_xt^3 + b_xt^2 + c_xt + d_x - p_x) \\ &+ 2(3a_yt^2 + 2b_yt + c_y)(a_yt^3 + b_yt^2 + c_yt + d_y - p_y) \\ &= 6a_x^2t^5 + 10a_xb_xt^4 + (8a_xc_x + 4b_x^2)t^3 + 6(a_xd_x + b_xc_x - a_xp_x)t^2 + (4b_x(d_x-p_x) + 2c_x^2)t + 2c_x(d_x-p_x) \\ &+ 6a_y^2t^5 + 10a_yb_yt^4 + (8a_yc_y + 4b_y^2)t^3 + 6(a_yd_y + b_yc_y - a_yp_y)t^2 + (4b_y(d_y-p_y) + 2c_y^2)t + 2c_y(d_y-p_y) \\ &= t^5 6(a_x^2+a_y^2) \\ &+ t^4 10(a_xb_x+a_yb_y) \\ &+ t^3 (8(a_xc_x+a_yc_y)+4(b_x^2+b_y^2)) \\ &+ t^2 6(a_x(d_x-p_x)+a_y(d_y-p_y) + b_xc_x+b_yc_y) \\ &+ t (4(b_x(d_x-p_x)+b_y(d_y-p_y)) + 2(c_x^2+c_y^2)) \\ &+ 2(c_x(d_x-p_x)+c_y(d_y-p_y)) \\ \end{aligned}

A polynomial, this time of degree 5, emerges here. For conciseness, we can express D'(t) polynomial coefficients as a bunch of dot products:

\begin{aligned} D'(t) &= t^5 6(\textbf{a}\cdot\textbf{a}) \\ &+ t^4 10(\textbf{a}\cdot\textbf{b}) \\ &+ t^3 (8(\textbf{a}\cdot\textbf{c})+4(\textbf{b}\cdot\textbf{b})) \\ &+ t^2 6(\textbf{a}\cdot(\textbf{d}-\textbf{p}) + \textbf{b}\cdot\textbf{c}) \\ &+ t (4(\textbf{b}\cdot(\textbf{d}-\textbf{p})) + 2(\textbf{c}\cdot\textbf{c})) \\ &+ 2(\textbf{c}\cdot(\textbf{d}-\textbf{p})) \\ \end{aligned}

Finally, we notice that solving D'(t)=0 is equivalent to solving D'(t)/2 = 0, so we simplify the expression:

\begin{aligned} D'(t)/2 &= t^5 3(\textbf{a}\cdot\textbf{a}) \\ &+ t^4 5(\textbf{a}\cdot\textbf{b}) \\ &+ t^3 (4(\textbf{a}\cdot\textbf{c})+2(\textbf{b}\cdot\textbf{b})) \\ &+ t^2 3(\textbf{a}\cdot(\textbf{d}-\textbf{p}) + \textbf{b}\cdot\textbf{c}) \\ &+ t (2(\textbf{b}\cdot(\textbf{d}-\textbf{p})) + \textbf{c}\cdot\textbf{c}) \\ &+ \textbf{c}\cdot(\textbf{d}-\textbf{p}) \\ \end{aligned}

Assuming we are able to solve this equation, we will get at most 5 values of t, among which we should find the shortest distance from p to the curve. Since t is bound within 0 and 1 (start and end of the curve), we will also have to test the distance at these locations.

Note

We could also compute the 2nd derivative in order to differentiate minimums from maximums, but simply evaluating the 5(+2) potential t values and keeping the smallest works just fine.

Distance from a random point to critical testing points of the curve

The red dot in the blue field is a random point in space. The red lines show which distances are evaluated (at most 5+2) to find the smallest one.

Translated to GLSL code

Transposing these formulas into code gives us this base template code:

float bezier_distance(vec2 p, vec2 p0, vec2 p1, vec2 p2, vec2 p3) {
    // Start by testing the distance to the boundary points at t=0 (p0) and t=1 (p3)
    vec2 dp0 = p0 - p,
         dp3 = p3 - p;
    float dist = min(dot(dp0, dp0), dot(dp3, dp3));

    // Bezier cubic points to polynomial coefficients
    vec2 a = -p0 + 3.0*(p1 - p2) + p3,
         b = 3.0 * (p0 - 2.0*p1 + p2),
         c = 3.0 * (p1 - p0),
         d = p0;

    // Solve D'(t)=0 where D(t) is the distance squared
    vec2 dmp = d - p;
    float da = 3.0 * dot(a, a),
          db = 5.0 * dot(a, b),
          dc = 4.0 * dot(a, c) + 2.0 * dot(b, b),
          dd = 3.0 * (dot(a, dmp) + dot(b, c)),
          de = 2.0 * dot(b, dmp) + dot(c, c),
          df = dot(c, dmp);

    float roots[5];
    int count = root_find5(roots, da, db, dc, dd, de, df);
    for (int i = 0; i < count; i++) {
        float t = roots[i];
        // Evaluate the distance to our point p and keep the smallest
        vec2 dp = ((a * t + b) * t + c) * t + dmp;
        dist = min(dist, dot(dp, dp));
    }

    // We've been working with the squared distance so far, it's time to get its
    // square root
    return sqrt(dist);
}

Note

dot(dp,dp) is a shorthand for the length squared, of course cheaper than computing length() which contains a square root.

Warning

We assume here the root finder only returns the roots that are within [0,1].

root_find5() is our 5th degree root finder, that is the function that gives us all the t (at most 5) which satisfy:

at^5+bt^4+ct^3+dt^2+et+f = 0

But before we are able to solve that, we need to study the simpler 2nd degree polynomial solving:

at^2+bt+c = 0

Solving quadratic polynomial equations

Diving into the rabbit hole of solving polynomial numerically will lead you to insanity. But we still have to scratch the surface because superior degree solvers usually rely on it.

My favorite quadratic root finding formula is the super simple one introduced by 3Blue1Brown, which involves locating a mid point m from which you get the 2 surrounding roots r:

\begin{aligned} m &= -\frac{b}{2a} \\ r &= m \pm \sqrt{m^2-\frac{c}{a}} \end{aligned}

In GLSL, a code to cover most common corner cases would look like this:

// Return true if x is not a NaN nor an infinite
// highp is probably mandatory to force IEEE 754 compliance
bool isfinite(highp float x) { return (floatBitsToUint(x) & 0x7f800000u) != 0x7f800000u; }

// Quadratic: solve ax²+bx+c=0
int root_find2(out float r[5], float a, float b, float c) {
    int count = 0;
    float m = -b / (2.*a);
    float d = m*m - c/a;
    if (!isfinite(m) || !isfinite(d)) { // a is (probably) too small
        // Linear: solve bx+c=0
        float s = -c / b;
        if (isfinite(s))
            r[count++] = s;
        return count;
    }
    if (d < 0.) // no root
        return count;
    if (d == 0.) {
        r[count++] = m; // single root
        return count;
    }
    float z = sqrt(d);
    r[count++] = m - z;
    r[count++] = m + z;
    return count;
}

Not quite as straightforward as the math formula, isn't it?

We cannot know in advance whether the division is going to succeed, so we do run divisions and only then check if they failed (and assume a reason for the failing). This is much more reliable than an arbitrary epsilon value. We also try to avoid duplicated roots.

Note

The roots are automatically sorted because z is always positive.

Warning

isfinite() may not be as reliable because in GLSL "NaNs are not required to be generated", meaning some edge case may not be supported depending on the hardware, drivers, and the current weather in Yokohama.

As much as I like it, this implementation might not be the most stable numerically (even though I don't have have strong data to back this claim). Instead, we may prefer the formula from Numerical Recipes:

\begin{aligned} \delta &= b^2-4ac \\ q &= -\frac{1}{2} (b + \mathrm{sign}(b)\sqrt{\delta}) \\ r_0 &= \frac{q}{a} \\ r_1 &= \frac{c}{q} \end{aligned}

Leading to the following alternative implementation:

int root_find2(out float r[5], float a, float b, float c) {
    int count = 0;
    float d = b*b - 4.*a*c;
    if (d < 0.)
        return count;
    if (d == 0.) {
        float s = -.5 * b / a;
        if (isfinite(s))
            r[count++] = s;
        return count;
    }
    float h = sqrt(d);
    float q = -.5 * (b + (b > 0. ? h : -h));
    float r0 = q/a, r1 = c/q;
    if (isfinite(r0)) r[count++] = r0;
    if (isfinite(r1)) r[count++] = r1;
    return count;
}

This is not perfect at all (especially with the b²-4ac part). There are actually many other possible implementations, and this HAL CNRS paper shows how near impossible it is to make a correct one. It is an interesting but depressing read, especially since it "only" covers IEEE 754 floats, and we have no such guarantee on GPUs. We also don't have fma() in WebGL, which greatly limits improvements. For now, it will have to do.

Solving quintic polynomial equations: attempt 1

Solving polynomials of degree 5 cannot be solved analytically like quadratics. And even if they were, we probably wouldn't do it because of numerical instability. Typically, in my experience, analytical 3rd degree polynomials solver do not provide reliable results.

The first iterative algorithm I picked was the Aberth–Ehrlich method. Nowadays, more appropriate algorithms exist, but at the time I started messing up with these problems (several years ago), it was a fairly good contender. This video explores how it works.

The convergence to the roots is quick, and it's overall simple to implement. But it's not without flaws. The main problem is that it works in complex space. We can't ignore the complex roots because they all "respond" to each others. And filtering these roots out at the end implies some unreliable arbitrary threshold mechanism (we keep the root only when the imaginary part is close to 0).

The initialization process also annoyingly requires you to come up with a guess at what the roots are, and doesn't provide anything relevant. Aberth-Ehrlich works by refining these initial roots, similar to a more elaborate Newton iterations algorithm. Choosing better initial estimates leads to a faster convergence (meaning less iterations).

The Cauchy bound specifies a space by defining the radius of a disk (complex numbers are in 2D space) where all the roots of a polynomial should lie within. We are going to use it for the initial guess, and more specifically its "tight" version (which unfortunately relies on pow()).

Since Aberth-Ehrlich is a refinement and not just a shrinking process, we define and use an inner disk that has half the area of Cauchy bound disk. That way, we're more likely to start with initial guesses spread in the "middle" of the roots; this is where the √2 comes from in the formula below.

#define K5_0 vec2( 0.951056516295154,  0.309016994374947)
#define K5_1 vec2( 0.000000000000000,  1.000000000000000)
#define K5_2 vec2(-0.951056516295154,  0.309016994374948)
#define K5_3 vec2(-0.587785252292473, -0.809016994374947)
#define K5_4 vec2( 0.587785252292473, -0.809016994374948)

int root_find5_aberth(out float roots[5], float a, float b, float c, float d, float e, float f) {
    // Initial candidates set mid-way of the tight Cauchy bound estimate
    float r = (1.0 + max_5(
        pow(abs(b/a), 1.0/5.0),
        pow(abs(c/a), 1.0/4.0),
        pow(abs(d/a), 1.0/3.0),
        pow(abs(e/a), 1.0/2.0),
            abs(f/a))) / sqrt(2.0);

    // Spread in a circle
    vec2 r0 = r * K5_0,
         r1 = r * K5_1,
         r2 = r * K5_2,
         r3 = r * K5_3,
         r4 = r * K5_4;

The circle constants are generated with the following script:

import math
import sys

n = int(sys.argv[1])
for k in range(n):
    angle = 2 * math.pi / n
    off = math.pi / (2 * n)
    z = angle * k + off
    c, s = math.cos(z), math.sin(z)
    print(f"#define K{n}_{k} vec2({c:18.15f}, {s:18.15f})")

Next, it's basically a simple iterative process. Unrolling everything for degree 5 looks like this:

#define close_to_zero(x) (abs(x) < eps)

// This also filters out roots out of the [0,1] range
#define ADD_ROOT_IF_REAL(r) if (close_to_zero(r.y) && r.x >= 0. && r.x <= 1.) roots[count++] = r.x

#define SMALL_OFF(off) (dot(off, off) <= eps*eps)

/* Complex multiply, divide, inverse */
vec2 c_mul(vec2 a, vec2 b) { return mat2(a, -a.y, a.x) * b; }
vec2 c_div(vec2 a, vec2 b) { return mat2(a, a.y, -a.x) * b / dot(b, b); }
vec2 c_inv(vec2 z)         { return vec2(z.x, -z.y) / dot(z, z); }

// Compute f(x)/f'(x): complex polynomial evaluation (y) divided by their
// derivatives (q) using Horner's method in one pass
vec2 c_poly5d4(float a, float b, float c, float d, float e, float f, vec2 x) {
    vec2 y =       a*x  + vec2(b, 0), q =       a*x  + y;
         y = c_mul(y,x) + vec2(c, 0); q = c_mul(q,x) + y;
         y = c_mul(y,x) + vec2(d, 0); q = c_mul(q,x) + y;
         y = c_mul(y,x) + vec2(e, 0); q = c_mul(q,x) + y;
         y = c_mul(y,x) + vec2(f, 0);
    return c_div(y, q);
}

vec2 sum_of_inv(vec2 z0, vec2 z1, vec2 z2, vec2 z3, vec2 z4) { return c_inv(z0 - z1) + c_inv(z0 - z2) + c_inv(z0 - z3) + c_inv(z0 - z4); }

int root_find5_aberth(out float roots[5], float a, float b, float c, float d, float e, float f) {
    if (close_to_zero(a))
        return root_find4_aberth(roots, b, c, d, e, f);

    // Code snip: see previous snippet
    // float r = ...
    // vec2 r0, r1, r2, ... 

    for (int m = 0; m < 16; m++) {
        vec2 d0 = c_poly5d4(a, b, c, d, e, f, r0),
             d1 = c_poly5d4(a, b, c, d, e, f, r1),
             d2 = c_poly5d4(a, b, c, d, e, f, r2),
             d3 = c_poly5d4(a, b, c, d, e, f, r3),
             d4 = c_poly5d4(a, b, c, d, e, f, r4);

        vec2 off0 = c_div(d0, vec2(1,0) - c_mul(d0, sum_of_inv(r0, r1, r2, r3, r4))),
             off1 = c_div(d1, vec2(1,0) - c_mul(d1, sum_of_inv(r1, r0, r2, r3, r4))),
             off2 = c_div(d2, vec2(1,0) - c_mul(d2, sum_of_inv(r2, r0, r1, r3, r4))),
             off3 = c_div(d3, vec2(1,0) - c_mul(d3, sum_of_inv(r3, r0, r1, r2, r4))),
             off4 = c_div(d4, vec2(1,0) - c_mul(d4, sum_of_inv(r4, r0, r1, r2, r3)));

        r0 -= off0;
        r1 -= off1;
        r2 -= off2;
        r3 -= off3;
        r4 -= off4;

        if (SMALL_OFF(off0) && SMALL_OFF(off1) && SMALL_OFF(off2) && SMALL_OFF(off3) && SMALL_OFF(off4))
            break;
    }

    int count = 0;
    ADD_ROOT_IF_REAL(r0);
    ADD_ROOT_IF_REAL(r1);
    ADD_ROOT_IF_REAL(r2);
    ADD_ROOT_IF_REAL(r3);
    ADD_ROOT_IF_REAL(r4);
    return count;
}

When the main coefficient is too small, we fall back on the 4th degree (and so on until we reach the analytic quadratic). The 4th and 3rd degree versions of this function are easy to guess (they're pretty much identical, just removing one coefficient at each degree).

We're also hardcoding a maximum of 16 iterations here because it's usually enough. To have an idea of how many iterations are required in practice, following is a visualization of the heat map of the number of iterations for every pixel:

Heat map of the iterations of the Aberth-Ehrlich algorithm

The big picture and the weaknesses of the algorithm should be pretty obvious by now. Among all drawbacks of this approach, there are also surprising pathological cases where the algorithm is not performing well. Fortunately, there were some progress on the state of the art in recent years.

Solving quintic polynomial equations: the state of the art

In 2022, Cem Yuksel published a new algorithm for polynomial root solving. Initially I had my reservations because the official implementation had a few shortcomings on some edge cases, which made me question its reliability. It's also optimized for CPU computation and is, to my very personal taste, overly complex.

Fortunately, Christoph Peters showed that it was possible on the GPU by implementing it for very large degrees, and without any recursion. Inspired by that, I decided to unroll it myself for degree 5.

One core difference with Aberth approach is that it is designed for arbitrary ranges. In our case this is actually convenient because, due to how Bézier curves are defined, we are only interested in roots between 0 and 1. We will need to adjust the Quadratic function to work in this range, as well as keeping the roots ordered:

     }
     float h = sqrt(d);
     float q = -.5 * (b + (b > 0. ? h : -h));
-    float r0 = q/a, r1 = c/q;
-    if (isfinite(r0)) r[count++] = r0;
-    if (isfinite(r1)) r[count++] = r1;
+    vec2 v = vec2(q/a, c/q);
+    if (v.x > v.y) v.xy = v.yx; // keep them ordered
+    if (isfinite(v.x) && v.x >= 0. && v.x <= 1.) r[count++] = v.x;
+    if (isfinite(v.y) && v.y >= 0. && v.y <= 1.) r[count++] = v.y;
     return r;
 }

The core logic of the algorithm relies on a cascade of derivatives for every degree. Christoph Peters provides an analytic formula to obtain the derivative for any degree. This is a huge helper when we need to work for an arbitrary degree, but in our case we can just differentiate manually:

\begin{aligned} f_5(x) &= ax^5+bx^4+cx^3+dx^2+ex+f \\ f_4(x) &= 5ax^4+4bx^3+3cx^2+2dx+e \\ f_3(x) &= 20ax^3+12bx^2+6cx+2d \\ f_2(x) &= 60ax^2+24bx+6c \end{aligned}

Since we're only interested in the roots, similar to what we did to D(t), we can simplify some of these expressions:

\begin{aligned} f_5(x) &= ax^5+bx^4+cx^3+dx^2+ex+f \\ f_4(x) &= 5ax^4+4bx^3+3cx^2+2dx+e \\ f_3(x) &= 10ax^3+6bx^2+3cx+d \\ f_2(x) &= 10ax^2+4bx+c \end{aligned}

The purpose of that cascade of derivatives is to cut the curve into monotonic segments. In practice, the core function looks like this:

int root_find5_cy(out float r[5], float a, float b, float c, float d, float e, float f) {
    float r2[5], r3[5], r4[5];
    int n = root_find2(r2,          10.*a, 4.*b,    c);            // degree 2
    n = cy_find5(r3, r2, n, 0., 0., 10.*a, 6.*b, 3.*c,   d);       // degree 3
    n = cy_find5(r4, r3, n,     0.,  5.*a, 4.*b, 3.*c, d+d, e);    // degree 4
    n = cy_find5(r,  r4, n,             a,    b,    c,   d, e, f); // degree 5
    reutnr n;
}

We could unroll cy_find3, cy_find4, and cy_find5, but to keep the code simple, the degree 3 to 5 will share the same function, with leading coefficients set to 0 (hopefully the compiler does its job properly).

cy_find5 relies on roots found (at most 4) at previous stages to define intervals of search:

Such an approach has the nice side effect of keeping the roots ordered.

The solver itself is not that complex either:

float poly5(float a, float b, float c, float d, float e, float f, float t) {
     return ((((a * t + b) * t + c) * t + d) * t + e) * t + f;
}

// Quintic: solve ax⁵+bx⁴+cx³+dx²+ex+f=0
iint cy_find5(out float r[5], float r4[5], int n, float a, float b, float c, float d, float e, float f) {
    int count = 0;
    vec2 p = vec2(0, poly5(a,b,c,d,e,f, 0.));
    for (int i = 0; i <= n; i++) {
        float x = i == n ? 1. : r4[i],
              y = poly5(a,b,c,d,e,f, x);
        if (p.y * y > 0.)
            continue;
        float v = bisect5(a,b,c,d,e,f, vec2(p.x,x), vec2(p.y,y));
        r[count++] = v;
        p = vec2(x, y);
    }
    return count;
}

The last brick of the algorithm is the Newton bisection, the slowest part:

// Newton bisection
//
// a,b,c,d,e,f: 5th degree polynomial parameters
// t: x-axis boundaries
// v: respectively f(t.x) and f(t.y)
float bisect5(float a, float b, float c, float d, float e, float f, vec2 t, vec2 v) {
    float x = (t.x+t.y) * .5; // mid point
    float s = v.x < v.y ? 1. : -1.; // sign flip
    for (int i = 0; i < 32; i++) {
        // Evaluate polynomial (y) and its derivative (q) using Horner's method in one pass
        float y = a*x + b, q = a*x + y;
              y = y*x + c; q = q*x + y;
              y = y*x + d; q = q*x + y;
              y = y*x + e; q = q*x + y;
              y = y*x + f;

        t = s*y < 0. ? vec2(x, t.y) : vec2(t.x, x);
        float next = x - y/q; // Newton iteration
        next = next >= t.x && next <= t.y ? next : (t.x+t.y) * .5;
        if (abs(next - x) < eps)
            return next;
        x = next;
    }
    return x;
}

And that's pretty much it. Looking at its heat map, it has a completely different look than Aberth:

Heat map of the iterations of Cem Yuksel algorithm

The number of iterations might be larger but it is much faster (I observed a factor 3 on my machine), the code is shorter, and actually more reliable.

Note

The scale used to represent the heat map is not the same as the one used in Aberth, but it is identical with the method presented in the next section.

Exploring ITP convergence

The bisection being the hot loop, it is interesting to ponder how to make this faster. A while back, Raph Levien hypothesized about how the ITP method could perform. Out of curiosity, I gave it a chance. The function is designed to work like a bisection, claiming to be as performant in the worst case.

There isn't a lot of code, and the paper provides a pseudo-code. But implementing it was actually challenging in many ways.

First of all, the authors didn't seem to find relevant to mention that it only works if f(a)<0<f(b). If f(a)>0>f(b), you're pretty much on your own. It requires just 2 lines of adjustments but figuring out this shortcoming of the algorithm was particularly unexpected.

Another bothering aspect concerns the parameters: K_1, K_2 and n_0. The paper proposes those:

\begin{aligned} K_1 &= 0.1 \\ K_2 &= 0.98(1+\frac{1+\sqrt{5}}{2})\approx 2.56567 \\ n_0 &= ? \end{aligned}

I played with them for a while and couldn't find any set that would make a real difference, so I ended up with the following:

For performance reasons, reducing K_2 to a value of 2 saves a call to pow().
For K_1, CRAN seems to suggest \frac{0.2}{b-a} so I went along with it
And for n_0, well 1 or 2 seem to be the usual parameter.

In the end, the function looks like this:

// ITP algorithm (2020) by Oliveira & Takahashi
// "An Enhancement of the Bisection Method Average Performance Preserving Minmax Optimality"
//
// a,b,c,d,e,f: 5th degree polynomial parameters
// t: x-axis boundaries (a and b in the paper)
// v: respectively f(a) and f(b) in the paper (evaluation of the function with t.x and t.y)
float itp5(float a, float b, float c, float d, float e, float f, vec2 t, vec2 v) {
    float diff = t.y-t.x;

    // K1 and n0 suggested by CRAN
    float K1 = .2 / diff;
    int n0 = 1;

    // The paper has the assumption that f(a)<0<f(b) but we want to
    // support f(a)>0>f(b) too, so we keep a sign flip
    float s = v.x < v.y ? 1. : -1.;

    // Using log(ab)=log(a)+log(b): log2(x/(2ε)) <=> log2(x/ε)-1
    int nh = int(ceil(log2(diff/eps)-1.)); // n_{1/2} (half point)
    int n_max = nh + n0;

    // ε 2^(n_max-k) = ε 2^n_max 2^-k = ε 2^n_max ½^k
    // ½^k is done iteratively in the loop, simplifying the arithmetic
    float q = eps * float(1<<n_max);

    while (diff > eps+eps) {
        // Interpolation
        float xf = (v.y*t.x - v.x*t.y) / (v.y-v.x); // Regula-Falsi

        // Truncation
        float xh = (t.x+t.y) * .5; // x half point
        float sigma = sign(xh - xf);
        float delta = K1*diff*diff; // save a pow() by forcing K2=2
        float xt = delta <= abs(xh - xf) ? xf + sigma*delta : xh; // xt: truncation of xf

        // Projection
        float r = q - diff*.5;
        float x = abs(xt-xh) <= r ? xt : xh-sigma*r;

        // Updating
        float y = poly5(a,b,c,d,e,f, x);
        float side = s*y;
        if      (side > 0.) t.y=x, v.y=y;
        else if (side < 0.) t.x=x, v.x=y;
        else                return x;

        diff = t.y-t.x;
        q *= .5;
    }
    return (t.x+t.y) * .5;
}

This function can be used as a drop'in replacement for bisect5.

I had a lot of expectations about it, but in the end it requires more iterations than the bisection we implemented. The paper claims to perform at least as good as a bisection, but our bisect5 is driven by the derivatives so it converges much faster. Here is the heat map with itp5 instead of bisect5:

Heat map of the iterations of Cem Yuksel algorithm with ITP method

Conclusion

The naive unrolled version of Cem Yuksel paper definitely is, so far, the best choice for our problem. I have still concerns about how to implement a good quadratic formula, and I have my reservations about various edge cases. There is also still room for improvements in the cubic solver (degree 3) because it's still a special case where analytical formulas exist, but in general this implementation is satisfying.

The next step is to work with chains of Bézier curves to make up complex shapes (such as font glyphs). It will lead us to build a signed distance field. This is not trivial at all and mandates one or several dedicated articles. We will hopefully study these subjects in the not-so-distant future.

Code golfing a tiny demo using maths and a pinch of insanity

Mon, 29 Sep 2025 13:30:50 -0000

A few weeks ago, I made a tiny demo that fits into 448 characters:

Red Alp GLSL demo in 448 characters

void main(){vec3 c,p,K=vec3(3,1,0);for(float z,i,a,g=1.,t,h,d,w,k=.15;i++<1e2;d=max(max(d-3.
,-d),a=z)*k,w=g-g/exp(h>.001?a++,d/.4:h*3e2),g-=a*=w,c+=a*d*4.5+(d>z?z:h/2e2)*K,a=min(p.y+2.
,1.),c.r+=w*a*a*.1,t+=min(h*.2,k/=.985))for(p=normalize(vec3(P+P-R,R.y))*t,p.xz*=mat2(cos(
sin(T*.2)+K.zyxz*11.)),p.z+=T*.3,d=p.y,h=d+.5,a=.01;a<1.;a+=a)p.xz*=mat2(8,6,-6,8)*.1,d+=abs
(dot(sin((p/a+T)*.3),p-p+a)),h+=abs(dot(sin(p.xz*.6/a),P-P+a));O=vec4(tanh(c),1);}

Note

The number of characters was 464 characters at first, but thanks to the community it got reduced further, and the article updated accordingly.

There is no texture, no mesh, no 3D helper: it's simply a procedural mathematical formula evaluated at each pixel assigning them a color. Code golfing is about making it as short as possible, and thus is part of the art performance.

To put things into perspective, the 853x480 JPEG thumbnail of this article is 167x larger than this code.

You can watch a larger version on its main dedicated page, or a portage on Shadertoy (484 chars). If your device is not powerful enough (I'm sorry for the lag on this page) or doesn't support WebGL2, a short preview video can be seen on Mastodon.

I'm guessing the wizardry of the code has confused many people so we're going to dive through the making-of together. Overall, this demo is a particularly dense and entangled compilation of different techniques, where each aspect could mandate a dedicated article. For that reason, some parts will prefer to link to external resources when the literacy is verbose on the subject.

Warning

Some demos in this article will start "decaying" over time due to floating point variables getting too large. Reloading the page should fix that.

The base template

The code is written in GLSL and is executed for each pixel (technically each fragment) on a simple quad geometry (to be accurate it's even a single big triangle). There is no geometry aside from that, it's basically just a fragment shader.

The fragment receives 3 different inputs:

the canvas resolution vec2 R
the time float T
the pixel position vec2 P (basically gl_FragCoord.xy)

And it has to output a sRGB color in out vec3 O. The code has to be written in a void main() function, and that's pretty much all we need to start.

If you're curious about the glue to setup WebGL2, just look at the source code on the dedicated page. There is no external dependency and the canvas setup code is pretty simple.

Development setup

For development, people usually directly use Shadertoy. I prefer to use my own local live coding environment: ShaderWorkshop. It can be run without setting up anything, just uv run --with shader-workshop sw-server (assuming the uv Python package manager is installed on the machine). Aside from the comfort of being able to use your favorite code editor, it allows instancing live controls for uniforms very easily, making it smooth to interact with any value and get an immediate feedback.

Red Alp demo with user controls as seen from ShaderWorkshop

Noise

One of the core primitive we need is a noise function: it is required for the mountains, the fog, and the clouds.

In a recent article, I talked about gradient noise. We could technically use that, but it will have a lot of drawbacks. First of all, it's super expensive. I know because I made a demo using it the other day, and it was awfully slow. Once per pixel would be fine, but in our case it will have to be evaluated a hundred times, so we need something faster.

Secondly, we're trying to make it as short as possible, and the 2D gradient noise, even minified, is already twice as big as the size of the full demo. We will also need a 3D noise for the clouds and fog, which is even larger and more expensive. And that's not even accounting for the fbm signal combination code.

Inigo Quilez, in his famous Rainforest, used value noise. It is faster, but it still won't do for us for the same reasons, just somehow mitigated. And since we're professionals, we're not going to cheat by sampling a noise texture.

Fortunately, while reverse engineering some Shadertoy demos, in particular the ones from diatribes, I came across some code that made use of this incredible technique of accumulating sine waves.

Combining sin waves

Let's say we want to combine two sine waves in order to get a height map as a 3rd dimension. There are multiple ways of achieving that. For example, we can multiply them:

z = \sin x \times \sin y

But we could also add them together:

z = \sin x + \sin y

The surprising take here is that... it's pretty much equivalent. It doesn't give the same result for sure, but visually it could be considered the same, just with a frequency and amplitude a bit different, and rotated on the z axis by 45°.

Similarly, you may think using cosines instead of sinusoids would make a difference, but no, even when combined together, they always give the same base pattern we just saw.

z = \sin x + \cos y

So let's pick one, let's say z=\sin x + \sin y. But this time, we're going to take the absolute value to transform the up and down pattern into bumps:

z = |\sin x + \sin y|

These bumps are the perfect base for clouds, but not so much for spiky mountains going through aggressive erosion. But with the help of this weird little trick, we can just flip the shape upside down to get sharp edges:

z = -|\sin x + \sin y|

We now have the basis for both our clouds and mountains, but it's not yet convincing. The next step is to use the fbm loop as if we were dealing with Gaussian or value noise: we accumulate several frequencies of our signal together:

z = S \sum_{i=0}^{N-1} F(\begin{bmatrix}x \\ y\end{bmatrix} \cdot l^{i}) g^{i}

S is the sign (-1 for spiky, 1 for bobby)
i is the octave identifier going from 0 to N-1 (included).
F(x,y) is usually the noise signal function, in our case it's the sinusoid combination function, we choose |\sin x + \sin y| here.
l is the lacunarity factor, that is how frequency changes at each octaves; this is usually a multiply by 2 or a close value.
g is the gain, that is by how the amplitude changes at each octaves; this is usually a multiply by 0.5 or a close value.

Without surprise this is still very periodic, but we can see a glimpse of chaos emerging. The final touch does all the magic: all we have to do now is simply rotate each layer by like, 30° or something (I'll pick 0.5 radians here, or about 29°):

The symmetry around the origin is still noticeable, but the illusion will work as we will move away from it. It's also possible to add some phase or offsetting (arbitrary addition within the sin or between each layer).

I implemented this in a Desmos 3D scene with all the parameters if one wants to play with it. The formula there has a few more controls, for example the vertical location, an optional transition offset in addition to the rotation, and controls for the base frequency and amplitude.

If this mathematical gibberish is above your head, a GLSL code for the 2D noise could look like this with a lacunarity of 2, a gain of 0.5 and 5 octaves:

float noise(vec2 p) {
    float v = 0.0;
    float amplitude = 1.0;
    for (int i = 0; i < 5; i++) {
        p = rotate(0.5) * p; // rotate our space (more on this in the next section)
        v += abs(sin(p.x) + sin(p.y)) * amplitude; // accumulate noise
        p *= 2.0; // double the frequency at each octave
        amplitude *= 0.5; // half the amplitude at each octave
    }
    return v;
}

One cool trick here: abs(sin(p.x)+sin(p.y)) could also be written abs(dot(sin(p),vec2(1))). This is interesting because now we can operate on the two components of p, easing the possibility to modify them at once (for example doing p*A+B). The dot trick doesn't work with sin(p.x)*sin(p.y), but fortunately, as we saw before, multiply and addition are similar and could be swapped in various situations.

Rotations

We needed some rotation for the noise, and they will be required again soon, so we need to have a closer look to them. Let's start with the formula most people are familiar with:

M = \begin{bmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{bmatrix}

A matrix can be seen as a function, so mathematically writing p'=M \cdot p would be equivalent to the code p=rotate(angle)*p with:

// Matrix for a counter-clockwise rotation
mat2 rotate(float a) {
    return mat2(
        cos(a), sin(a), // column 1
       -sin(a), cos(a)  // column 2
    );
}

Doing p'=M \cdot p is rotating the space p lies into, which means it gives the illusion the object is rotating clockwise. Though, in the expression p=rotate(angle)*p, I can't help but be bothered by the redundancy of p, so I would prefer to write p*=rotate(angle) instead. Since matrices are not commutative, this will instead do a counter-clockwise rotation of the object. The inlined rotation ends up being:

p *= mat2(cos(a),sin(a),-sin(a),cos(a)); // counter-clockwise rotation of object at point p

Note

To make the rotation clockwise, we can of course use -a, or we can transpose the matrix: mat2(cos(a),-sin(a),sin(a),cos(a)).

This is problematic though: we need to repeat the angle 4 times, which can be particularly troublesome if we want to create a macro and/or don't want an intermediate variable for the angle. But I got you covered: trigonometry has a shitton of identities, and we can express every sin according to a cos (and the other way around).

For example, here is another formulation of the same expression:

p *= mat2(cos(a + vec4(0,3,1,0)*PI/2.0));

Now the angle appears only once, in a vectorized cosine call.

GLSL has degrees() and radians() functions, but it doesn't expose anything for \pi nor \tau constants. And of course, it doesn't have sinpi and cospi implementation either. So it's obvious they want us to use \arccos(-1) for \pi and \arccos(0) for \pi/2:

p *= mat2(cos(a + vec4(0,3,1,0)*acos(0.)));

Note

To specify a as a normalized value, we can use mat2(cos((a*4.+vec4(0,3,1,0))*acos(0.))).

On his Unofficial Shadertoy blog, Fabrice Neyret goes further and provide us with a very cute approximation, which is the one we will use:

p *= mat2(cos(a + vec4(0,11,33,0)));

I checked for the best numbers in 2 digits, and I can confirm they are indeed the ones providing the best accuracy.

Comparison of the 2 rotations matrices

On this last figure, the slight red/green on the outline of the circle represents the loss of precision.

Note

With 3 digits, 344 and 699 can respectively be used instead of 11 and 33.

This is good when we want a dynamic rotation angle (we will need that for the camera panning typically), but sometimes we just need a hardcoded value: for example in the rotate(0.5) of our combined noise function.

mat2(cos(.5+vec4(0,11,33,0))) is fine but we can do better. Through Inigo's demos I found the following: mat2(.8,.6,-.6,.8). It makes a rotation angle of about 37° (around 0.64 radians) in a very tiny form. Since 0.5 was pretty much arbitrary, we can just use this matrix as well. And we can make it even smaller (thank you jolle):

p *= mat2(8,6,-6,8)*.1; // rotate p counter-clockwise by about 37° without any trigo

One last rotation tip from Fabrice's bag of tricks: rotating in 3D around an axis can be done with the help of GLSL swizzling:

p.xz *= rotate(0.5); // 3D rotation around y-axis (the absent component)

We will use this too.

Note

p.zy *= rotate(.5) is the same p.yz *= rotate(-.5), if we need to save one character and can't transpose the matrix.

Camera (and axis) setup

One last essential before going creative is the camera setup.

We start with the 2D P pixel coordinates which we are going to make resolution independent by transforming them into a traditional mathematical coordinates system:

// 1:1 ratio with [-1,1] along the shortest axis (horizontal or vertical)
vec2 u = (2.0*P - R) / min(R.x, R.y);

Since we know our demo will be rendered in landscape mode, dividing by R.y is enough. We can also save one character using P+P:

// 1:1 ratio with [-1,1] along the vertical axis
vec2 u = (P+P - R) / R.y;

To enter 3D space, we append a third component, giving us either a right or a left-handed Y-up coordinates system. This choice is not completely random.

Indeed, it's easier/shorter to add a 3rd dimension at the end compared to interleaving a middle component. Compare the length of vec3(P, z) to vec3(P.x, z, P.y) (Z-up convention). In the former case, picking just a plane remains short and easy thanks to swizzling: p.xz instead of p.xy.

To work in 3D, we need an origin point (ro for ray origin) and a looking direction (rd for ray direction). ro is picked arbitrarily for the eye position, while rd is usually calculated thanks to a lookAt helper:

// Right-hand with Y-up (like Godot)
mat3 lookAt(vec3 origin /* where we are */, vec3 target /* where we look */) {
    vec3 w = normalize(target - origin);
    vec3 u = normalize(cross(w, vec3(0,1,0)));
    vec3 v = normalize(cross(u, w)); // Note: normalize() can be ditched here
    return mat3(u, v, w);
}

Right-hand Y-up 3D coordinates system

Which is then used like that, for example:

vec2 u = (P+P - R) / R.y;
vec3 target = /* ... */;
vec3 ro = /* ... */;
vec3 rd = normalize(lookAt(ro, target) * vec3(u, 1));

Note

I made a Shadertoy demo to experiment with different 3D coordinate spaces if you are interested in digging this further.

All of this is perfectly fine because it is flexible, but it's also way too much unnecessary code for our needs, so we need to shrink it.

One approach is to pick a simple origin and straight target point so that the matrix is as simple as possible. And then later on apply some transformations on the point. If we give ro=vec3(0) and target=vec3(0,0,1), we end up with an identity matrix, so we can ditch everything and just write:

vec3 rd = normalize(vec3((P+P - R) / R.y, 1));

This can be shorten further: since the vector is normalized anyway, we can scale it at will, for example by a factor R.y, saving us precious characters:

vec3 rd = normalize(vec3(P+P - R, R.y));

And just like that, we are located at the origin vec3(0), looking toward Z+, ready to render our scene.

Mountain height map

It's finally time to build our scene. We're going to start with our noise function previously defined, but we're going tweak it in various ways to craft a mountain height map function.

Here is our first draft:

const float mountain_y = -0.5; // mountain y-axis position
const float mountain_f = 0.6; // mountain base frequency

float mountain_height_map(vec2 p) {
    float h = mountain_y;
    for (float a = 1.0; a > 0.01; a /= 2.0) {
        p *= rotate(0.5);
        h += abs(dot(sin(p*mountain_f / a), vec2(1))) * a; // dot(sin(v),1) -> sin(v.x)+sin(v.y)
    }
    return -h; // minus for the spiky version of the noise
}

We're exploiting one important correlation of the noise function: at every octave, the amplitude is halving while the frequency is doubling. So instead of having 2 running variables, we just have an amplitude a getting halved every octave, and we divide our position p by a (which is the same as multiplying by a frequency that doubles itself).

I actually like this way of writing the loop because we can stop the loop when the amplitude is meaningless (a>0.01 acts as a precision stopper). Unfortunately, we'll have to change it to save one character: a/=2. is too long for the iteration, we're going to double instead by using a+=a which saves one character. So instead the loop will be written the other way around: for (float a=.01; a<1.; a+=a). It's not exactly equivalent, but it's good enough (and we can still tweak the values if necessary).

We're going to inline the constants and rotate, and use one more cool trick: vec2(1) can be shortened: we just need another vec2. Luckily we have p, so we can simply replace it with p/p.

Finally, we can get rid of the braces of the for loop by using the , in its local scope:

float mountain_height_map(vec2 p) {
    float h = -.5;
    for (float a=.01; a<1.; a+=a)
        p *= mat2(8,6,-6,8)*.1,
        h += abs(dot(sin(p*.6/a), p/p))*a;
    return -h;
}

p/p works fine as long as it's not zero. In this particular case, we can instead use vec2(0) (obtained with p-p) and then include the a amplitude multiplier within the expression: abs(dot(sin(p*.6/a), p-p+a)). (p-p+a is the same as vec2(a) when p is a vec2). We end up with the following safer version:

float mountain_height_map(vec2 p) {
    float h = -.5;
    for (float a=.01; a<1.; a+=a)
        p *= mat2(8,6,-6,8)*.1,
        h += abs(dot(sin(p*.6/a), p-p+a));
    return -h;
}

Mountain height map in 2D (rescaled for display)

To render this in 3D, we are going to do some ray-marching.

Solid ray-marching

The main technique used in most Shadertoy demos is ray-marching. I will assume familiarity with the technique, but if that's not the case, An introduction to Raymarching (YouTube) by kishimisu and Painting with Math: A Gentle Study of Raymarching by Maxime Heckel were good resources for me.

In short: we start from a position in space called the ray origin ro and we project it toward a ray direction rd. At every iteration we check the distance to the closest solid in our scene, and step toward that distance, hoping to converge closer and closer to the object boundary.

We end up with this main loop template:

float t = 0.0;

vec3 ro = vec3(0); // ray origin
vec3 rd = normalize(vec3(P+P - R, R.y)); // ray direction

// 100 iterations should be enough to hit something if there is any
for (int i = 0; i < 100; i++) {
    vec3 p = ro + rd*t; // t amount in rd direction from ro origin
    float h = distance_to_solid(p); // 3D distance function
    if (h < 0.001) { // we converged close enough to a solid
        // Here we assign a color according to where p is
        // [...]
        break;
    }
    t += h; // there is no solid closer than h so we step by that much
}

This works fine for solids expressed with 3D distance fields, that is functions that for a given point give the distance to the object. We will use it for our mountain, with one subtlety: the noise height map of the mountain is not exactly a distance (it is only the distance to what's below our current point p):

float distance_to_solid(vec3 p) { // positive outside, negative inside
    return p.y - mountain_height_map(p.xz);
}

Because of this, we can't step by the distance directly, or we're likely to go through mountains during the stepping (t += h). A common workaround here is to step a certain percentage of that distance to play it safe.

Technically we should figure out the theorical proper shrink factor, but we're going to take a shortcut today and just arbitrarily cut. Using trial and error I ended up with 20% of the distance.

After a few simplifications, we end up with the following (complete) code:

float mountain_height_map(vec2 p) {
    float h = .5;
    for (float a=.01; a<1.; a+=a)
        p *= mat2(8,6,-6,8)*.1,
        h += abs(dot(sin(p*.6/a), p-p+a));
    return -h;
}

float distance_to_solid(vec3 p) {
    return p.y - mountain_height_map(p.xz);
}

void main() {
    vec3 rd = normalize(vec3(P+P - R, R.y));

    float t = 0.0, color = 0.0;
    for (int i = 0; i < 100; i++) {
        vec3 p = rd*t;

        p.z += T*.2; // move forward

        float h = distance_to_solid(p);
        if (h < 0.001) {
            color = exp(-t*t*.01); // depth map like "coloring"
            break;
        }
        t += h * 0.2;
    }

    O = vec4(vec3(pow(color, 3.0/2.2)), 1);
}

Basic ray-marching of the mountain height map

We start at ro=vec3(0) so I dropped the variable entirely.

You may be curious about the power at the end; this is just a combination of luminance perception with gamma 2.2 (sRGB) transfer function. It only works well for grayscale; for more information, see my previous article on blending.

Clouds and fog

Compared to the mountain, the clouds and fog will need a 3 dimensional noise. Well, we don't need to be very original here; we simply extend the 2D noise to 3D:

float noise3(vec3 p) {
    float v;
    for (float a=.01; a<1.; a+=a)
        p.xz *= mat2(8,6,-6,8)*.1,
        v += abs(dot(sin(p*.3/a + T*.3), p-p+a));
    return v;
}

The base frequency is lowered to 0.3 to make it smoother, and the p goes from 2 to 3 dimensions. Notice how the rotation is only done on the y-axis, the one pointing up): don't worry, it's good enough for our purpose.

We also add a phase (meaning we are offsetting the sinusoid) of T*0.3 (T is the time in seconds, slowed down by the multiply) to slowly morph it over time. The base frequency and time scale being identical is a happy "coincidence" to be factored out later (I actually forgot about it until jolle reminded me of it).

You also most definitely noticed v isn't explicitly initialized: while only true WebGL, it guarantees zero initialization so we're saving a few characters here.

Volumetric ray-marching

For volumetric material (clouds and fog), the loop is a bit different: instead of calculating the distance to the solid for our current point p, we do compute the density of our target "object". Funny enough, it can be thought as a 3D SDF but with the sign flipped: positive inside (because the density increases as we go deeper) and negative outside (there is no density, we're not in it).

const float clouds_y = 3.0; // vertical position

float clouds_density(vec3 p) {
    float n = noise3(p);     // random value associated with a 3D position in space
    float h = -clouds_y + n; // similar to mountain_height_map() but 3d and bobby
    float d = p.y - h;       // similar to distance_to_solid()
    d = -d;                  // flip sign: distance to density
    // We are only interested in the density within the material,
    // the density will be considered 0 when outside of it.
    return max(d, 0.0);
}

For simplicity, we're going to rewrite the function like this:

const float clouds_y = 3.0;

float clouds_density(vec3 p) {
    float n = noise3(p);
    float d = -p.y - cloud_y + n;
    return max(d, 0.0);
}

Compared to the solid ray-marching loop, the volumetric one doesn't bail out when it reaches the target. Instead, it slowly steps into it, damping the light as the density increases:

const float absorption = 0.15;
const float radiance   = 1.0;

void main() {
    float step_len = 0.15;
    float t;

    vec3 rd = normalize(vec3(P+P-R,R.y));
    vec3 color;

    float transmittance = 1.0; // remaining visibility
    for (int i = 0; i < 100; i++) {
        vec3 p = rd*t;

        // Move camera forward
        p.z += T * 1.5;

        // How many particules of the material we can find at that position
        // If negative, we're not in the element yet, otherwise it's the density
        // (getting higher as we go deeper into it typically).
        float d = clouds_density(p);

        // Integrate the density discretely: we assume the segment of length
        // we're walking has the same point density all along
        d *= step_len;

        // The fraction of light that survives through this segment (Beer-Lambert law)
        // The denser, the closer to 0 this gets
        float attenuation = exp(-d*absorption);

        float emission = d*radiance; // how much light is emitted along the segment (glow)
        float alpha = 1.0 - attenuation; // fraction of light removed for that given density segment

        float weight = alpha * transmittance;

        // Accumulate color emission
        color += weight * emission;

        transmittance -= weight; // could also be written transmittance *= attenuation

        // Normal volumetric marching (step_len) clamped to the distance to the
        // solid (mountain)
        t += step_len;

        // Larger volumetric steps as we go far
        step_len *= 1.015;
    }

    O = vec4(pow(color, vec3(3.0/2.2)), 1);
}

The core idea is that the volumetric material emit some radiance but also absorbs the atmospheric light. The deeper we get, the smaller the transmittance gets, til it converges to 0 and stops all light. All the threshold you see are chosen by tweaking them through trial and error, not any particular logic. It is also highly dependent on the total number of iterations.

Note

Steps get larger and larger as the distance increases; this is because we don't need as much precision per "slice", but we still want to reach a long distance.

Basic volumetric ray-marching of the clouds density map

We want to be positioned below the clouds, so we're going to need a simple sign flip in the function.

The fog will take the place at the bottom, except upside down (the sharpness will give a mountain-hug feeling) and at a different position. clouds_density() becomes:

const float clouds_y = 3.0;
const float fog_y    = 0.0;

float clouds_fog_density(vec3 p) {
    float n = noise3(p);

    float clouds_d = p.y - clouds_y + n;
    float fog_d    = p.y - fog_y    + n;

    // Pick the element with the highest density (they don't overlap anyway)
    float d = max(clouds_d, -fog_d);

    return max(d, 0.0);
}

Both clouds and fog with volumetric ray-marching

For more resources on volumetric rendering, following are the ones I studied the most:

Volumetric Rendering in 2 parts, by Chris
Real-time dreamy Cloudscapes with Volumetric Raymarching, by Maxime Heckel again
Volumetric Raymarching, by Xor

Combining ray-marching

Having a single ray-marching loop combining the two methods (solid and volumetric) can be challenging. In theory, we should stop the marching when we hit a solid, bail out of the loop, do some fancy normal calculations along with light position. We can't afford any of that, so we're going to start doing art from now on.

We start from the volumetric ray-marching loop, and add the distance to the mountain:

for (int i = 0; i < 100; i++) {
    vec3 p = rd*t;

    // ...

    float d = clouds_fog_density(p);
    float h = distance_to_solid(p);

    // ...
}

If h gets small enough, we can assume we hit a solid:

bool solid = h < 0.001;

In volumetric, the attenuation is calculated with the Beer-Lambert law. For solid, we're simply going to make it fairly high:

-    float attenuation = exp(-d*absorption);
+    float attenuation = solid ? 0.95 : exp(-d*absorption);

This has the effect of making the mountain like a very dense gas.

We're also going to disable the light emission from the solid (it will be handled differently down the line):

-    float emission = d*radiance;
+    float emission = solid ? 0.0 : d*radiance;

The transmittance is not going to be changed when we hit a solid as we just want to accumulate light onto it:

-    transmittance -= weight;
+    if (!solid) transmittance -= weight;

Finally, we have to combine the volumetric stepping (t += step_len) with the solid stepping (t += h*0.2) by choosing the safest step length, that is the minimum:

-    t += step_len;
+    t += min(h*0.2, step_len);

We end up with the following:

Combination of volumetric and solid ray-marching

We can notice the mountain from negative space and the discrete presence of the fog, but it's definitely way too dark. So the first thing we're going to do is boost the radiance, as well as the absorption for the contrast:

-const float absorption = 0.15;
-const float radiance   = 1.0;
+const float absorption = 2.5;
+const float radiance   = 4.5;

This will make the light actually overshoot, so we also have to replace the current gamma 2.2 correction with a cheap and simple tone mapping hack: tanh():

-    O = vec4(pow(color, vec3(3.0/2.2)), 1);
+    O = vec4(tanh(color), 1);

Tonemapping the scene

The clouds and fog are much better but the mountain is still trying to act cool. So we're going to tweak it in the loop:

emission += 0.1;

This boosts the overall emission.

While at it, since the horizon is also sadly dark, we want to blast some light into it:

color += d == 0.0 ? 0.005*h : 0.0;

mkbosmans from HackerNews noticed that the opposite of d==0.0 is actually d>0.0 due to the max(...,0). So we could write it more simply:

color += d > 0.0 ? 0.0 : 0.005*h;

When the density is null (meaning we're outside clouds and fog), an additional light is added, proportional to how far we are from any solid (the sky gets the most boost basically).

More atmospheric light

The mountain looks fine but I wanted a more eerie atmosphere, so I changed the attenuation:

-    float attenuation = solid ? 0.95 : exp(-d*absorption);
+    float attenuation = exp(solid ? -h*300.0 : -d*absorption);

Now instead of being a hard value, the attenuation is correlated with the proximity to the solid (when getting close to it). This has nothing to with any physics formula or anything, it's more of an implementation trick which relies on the ray-marching algorithm. The effect it creates is those crack-like polygon edges on the mountain.

To add more to the effect, the emission boost is tweaked into:

-    emission += 0.1;
+    float e = min(p.y - mountain_y + 1.5, 1.0);
+    emission += e*e * 0.1;

This makes the bottom of the mountain darker quadratically: only the tip of the mountain would have the glowing cracks.

Making mountains eerie

Color

We've been working in grayscale so far, which is a usually a sound approach to visual art in general. But we can afford a few more characters to move the scene to a decent piece of art from the 21st century.

Adding the color just requires very tiny changes. First, the emission boost is going to target only the red component of the color:

-    emission += e*e * 0.1;
     float alpha = 1. - attenuation;
     float weight = alpha * transmittance;
     color += weight * emission;
+    color.r += weight * e*e * 0.1;

And similarly, the overall addition of light into the horizon/atmosphere is going to get a redish/orange tint:

-    color += d > 0.0 ? 0.0 : 0.005*h;
+    color += (d > 0.0 ? 0.0 : 0.005*h) * vec3(3,1,0);

Add a red/orange tint

Last tweaks

We're almost done. For the last tweak, we're going to add a cyclic panning rotation of the camera, and adjust the moving speed:

p.xz *= mat2(cos(sin(T*.2)+vec4(0,11,33,0)));
p.z += T*.3;

Note

I'm currently satisfied with the "seed" of the scene, but otherwise it would have been possible to nudge the noise in different ways. For example, remember the sin can be replaced with cos in either or both volumetric and mountain related noises. Similarly, the offsetting +T could be changed into -T for a different morphing effect. And of course the rotations can be swapped (either by changing .xz into .zx or transposing the values).

Code golfing

At this point, our code went through early stages of code golfing, but it still needs some work to reach perfection. Stripped out of its comments, it looks like this:

// Reference code: 1278 chars (unnecessary spaces and line breaks are not counted)
const float fog_y      = 0.0;
const float clouds_y   = 3.0;
const float mountain_y = -0.5;
const float absorption = 2.5;
const float radiance   = 4.5;

float noise3(vec3 p) {
    float v;
    for(float a=.01; a<1.; a+=a)
        p.xz *= mat2(8,6,-6,8)*.1,
        v += abs(dot(sin(p*.3/a + T*.3), vec3(1)))*a;
    return v;
}

float clouds_fog_density(vec3 p) {
    float n = noise3(p);
    float clouds_d = p.y-clouds_y+n;
    float fog_d    = p.y-fog_y+n;
    float d = max(clouds_d, -fog_d);
    return max(d, 0.0);
}

float mountain_height_map(vec2 p) {
    float h = -mountain_y;
    for (float a=.01; a<1.; a+=a)
        p *= mat2(8,6,-6,8)*.1,
        h += abs(dot(sin(p*.6/a), vec2(1)))*a;
    return -h;
}

float distance_to_solid(vec3 p) {
    return p.y - mountain_height_map(p.xz);
}

void main() {
    float step_len = 0.15;
    float t;

    vec3 color;

    float transmittance = 1.0;
    vec3 rd = normalize(vec3(P+P-R,R.y));
    for (int i = 0; i < 100; i++) {
        vec3 p = rd*t;

        p.xz *= mat2(cos(sin(T*.2)+vec4(0,11,33,0)));
        p.z += T*.3;

        float d = clouds_fog_density(p);
        float h = distance_to_solid(p);

        bool solid = h < 0.001;
        d *= step_len;
        float attenuation = exp(solid ? -h*300.0 : -d*absorption);
        float emission = solid ? 0.0 : d*radiance;
        float e = min(p.y - mountain_y + 1.5, 1.0);
        float alpha = 1. - attenuation;
        float weight = alpha * transmittance;
        color   += weight * emission;
        color.r += weight * e*e * 0.1;
        color += (d > 0.0 ? 0.0 : 0.005*h) * vec3(3,1,0);
        if (!solid) transmittance -= weight;
        t += min(h*0.2, step_len);
        step_len *= 1.015;
    }

    O = vec4(tanh(color), 1);
}

The first thing we're going to do is notice that both the mountain, clouds, and fog use the exact same loop. Factoring them out and inlining the whole thing in the main function is the obvious move:

// 922 chars
const float fog_y      = 0.0;
const float clouds_y   = 3.0;
const float mountain_y = -0.5;
const float absorption = 2.5;
const float radiance   = 4.5;

void main() {
    float step_len = 0.15;
    float t;

    vec3 color;

    float transmittance = 1.0;
    vec3 rd = normalize(vec3(P+P-R,R.y));
    for (int i = 0; i < 100; i++) {
        vec3 p = rd*t;

        p.xz *= mat2(cos(sin(T*.2)+vec4(0,11,33,0)));
        p.z += T*.3;

        float d = p.y;
        float h = p.y-mountain_y;
        for (float a=.01; a<1.; a+=a)
            p.xz *= mat2(8,6,-6,8)*.1,
            d += abs(dot(sin(p*.3/a + T*.3), vec3(1)))*a,
            h += abs(dot(sin(p.xz*.6/a), vec2(1)))*a;
        d = max(max(d-clouds_y, -(d-fog_y)), 0.0);

        bool solid = h < 0.001;
        d *= step_len;
        float attenuation = exp(solid ? -h*300.0 : -d*absorption);
        float emission = solid ? 0.0 : d*radiance;
        float e = min(p.y - mountain_y + 1.5, 1.0);
        float alpha = 1. - attenuation;
        float weight = alpha * transmittance;
        color   += weight * emission;
        color.r += weight * e*e * 0.1;
        color += (d > 0.0 ? 0.0 : 0.005*h) * vec3(3,1,0);
        if (!solid) transmittance -= weight;
        t += min(h*0.2, step_len);
        step_len *= 1.015;
    }

    O = vec4(tanh(color), 1);
}

Next, we are going to do the following changes:

Rename every variable to single letter or inline them whenever possible
Inline all constants
Remove any explicit zero initialization
Use float instead of int for the iterator and bool for the solid flag
Pack all float and vec3 declarations together
Simplify numbers: 1e2 instead of 100.0, 3. instead of 3.0, etc.
vec*() constructor act like cast, so you can pass down integers
Instead of *x, /(1/x) is sometimes shorter (for example /.4 instead of *2.5) (thanks coyote)

// 491 chars
void main() {
    vec3 c, p;
    for (float i, a, g=1., t, h, d, w, k=.15, x, e; i < 1e2; i++) {
        p = normalize(vec3(P+P-R,R.y))*t;
        p.xz *= mat2(cos(sin(T*.2)+vec4(0,11,33,0)));
        p.z += T*.3;
        d = p.y;
        h = p.y+.5;
        for (a=.01; a<1.; a+=a)
            p.xz *= mat2(8,6,-6,8)*.1,
            d += abs(dot(sin(p*.3/a + T*.3), vec3(1)))*a,
            h += abs(dot(sin(p.xz*.6/a), vec2(1)))*a;
        d = max(max(d-3., -d), 0.);
        x = h < .001 ? 0. : 1.;
        d *= k;
        e = min(p.y+2., 1.);
        w = g * (1. - exp(x==0. ? -h*3e2 : -d/.4));
        c += w * x*d*4.5;
        c.r += w * e*e * .1;
        c += (d > 0. ? .0 : h/2e2) * vec3(3,1,0);
        g -= w * x;
        t += min(h*.2, k);
        k /= .985;
    }
    O = vec4(tanh(c), 1);
}

Last pass of tricks:

Merge and unroll more expressions together
Use alternative forms for vec*(1)
Rely on mathematical equivalences such as e^{-x}=1/e^x
Some symbol names can be reused (see a)
Notice how the rotation matrix coefficients (0,11,33,0) are close to the red factors (3,1,0)? That's right, we can factor that out into a shared constant K.
Iterate i within the condition
We're going to inline k*=1.015 inside the min(): this is not equivalent, but in practice it makes no difference
The first 5 instructions of the main loop go into the initialization placeholder of the inner for, and all the others go into the iteration placeholder of the outer for, so that we can remove all {}
Declare a z to be used instead of 0. since we have a bunch of them (thanks coyote)
The x = h < .001 ? 0. : 1. can also be obtained progressively through some increment trick (thanks coyote)

I'm also reordering a bit some instructions for clarity 🙃

// 448 chars
void main() {
    vec3 c,p,K=vec3(3,1,0);
    for(float z,i,a,g=1.,t,h,d,w,k=.15; i++<1e2;
        d = max(max(d-3.,-d),a=z)*k,
        w = g-g/exp(h>.001?a++,d/.4:h*3e2),
        g -= a*=w,
        c += a*d*4.5+(d>z?z:h/2e2)*K,
        a = min(p.y+2.,1.),
        c.r += w*a*a*.1,
        t += min(h*.2,k/=.985))
        for(p=normalize(vec3(P+P-R,R.y))*t,
            p.xz*=mat2(cos(sin(T*.2)+K.zyxz*11.)),
            p.z+=T*.3,
            d=p.y,h=d+.5,a=.01;a<1.;a+=a)
            p.xz *= mat2(8,6,-6,8)*.1,
            d += abs(dot(sin((p/a+T)*.3),p-p+a)),
            h += abs(dot(sin(p.xz*.6/a),P-P+a));
    O = vec4(tanh(c),1);
}

And here we are. All we have to do now is remove all unnecessary spaces and line breaks to obtain the final version. I'll leave you here with this readable version.

Golfer by Courtney Cook (Unsplash)

Forewords

I'm definitely breaking the magic of that artwork by explaining everything in detail here. But it should be replaced with an appreciation for how much concepts, math, and art can be packed in so little space. Maybe this is possible because they fundamentally overlap?

Nevertheless, writing such a piece was extremely refreshing and liberating. As a developer, we're so used to navigate through mountains of abstractions, dealing with interoperability issues, and pissing glue code like robots. Here, even though GLSL is a very crude language, I can't stop but being in awe by how much beauty we can produce with a standalone shader. It's just... Pure code and math, and I just love it.

Perfecting anti-aliasing on signed distance functions

Sat, 26 Jul 2025 14:29:32 -0000

Doing anti-aliasing on SDF is not as straightforward as it seems. Most of the time, we see people use a smoothstep with hardcoded constants, sometimes with screen space information, sometimes cryptic or convoluted formulas. Even if SDFs have the perfect mathematical properties needed for a clean anti-aliasing, the whole issue has a scope larger than it appears at first glance. And even when trivial solutions exist, it's not always clear why they are a good fit. Let's study that together.

SDF

The article assumes that you are at least a bit familiar with what an SDF is, but if I had to provide a quick and informal definition, I would say something like:

"It's a function (or lookup-table of said function, usually stored in a texture) which returns the signed distance from the specified coordinates to a given shape, where the sign indicates whether you're inside or outside the shape."

A common visualization of it looks like this:

SDF of a moving pie/pacman, using Inigo Quilez formula and colorscheme for visualization

The distance is fancily colored here for illustrative purpose, and the shape is animated to see how it affects the field.

Another way of seeing it is to switch to a 3D view:

SDF of a moving pie/pacman, as seen in 3D

For the sign interpretation, here we're using the convention positive inside and negative outside, as seen for example on the Wikipedia illustration. But this is not always the case, for example, Inigo prefers the opposite: negative inside and positive outside. I personally find the Wikipedia convention to be more intuitive and easy to work with, but that's a matter of preferences so we'll figure out the formulas for both models. Switching from one to the other is just a sign swap, but it's important to know what we are working with.

Linear ramp

A properly crafted SDF has a gradient of length 1, meaning the slope is either going up or down, but always at the same constant rate of 1:

1D side cut of an SDF depicting the gradient/slope

This is an important property since anti-aliasing is all about transitioning smoothly toward (or away from) the shape. For our first attempt at anti-aliasing we will simply follow that ramp and make a straight transition.

Once again, we are going to rely on linear, one of the most useful math formulas:

float linear(float a, float b, float x) { return (x-a)/(b-a); }

And more specifically we will need its saturated version linearstep:

float linearstep(float a, float b, float x) { return clamp(linear(a,b,x), 0.0, 1.0); }

This is the same as the well-known smoothstep, except it's a straight line when transitioning from a to b.

linearstep function

The length of our ramp (the transition zone between a and b) is going to be arbitrary at first, we will call it w (for "width"). It's our diffuse, or blur parameter if you prefer. The height h we are looking for corresponds to the opacity of our shape.

Given a positive inside and negative outside SDF, we will start with the transition centered around the boundary between the shape and its outside.

You might be confused about the relationship between the distance and the transition zone (diffuse width w). The following diagram may help clarifying why:

The relationship between the diffuse width (w) and the signed distance (d)

Remember, the gradient of an SDF is supposed to have a length of 1. This means there is a direct match between the height of the signed distance (y-axis on the figure), and the spacial distance traveled (x-axis on the figure).

Note

This is why Inigo and other folks spend a lot of energy into looking for the perfect formula for the distance to an ellipse. We cannot just stretch a circle as it would distort the SDF, and thus break this important property. A broken AA would be one of the consequences.

The previous figure shows that for a centered transition, when the distance d is within [-w/2,w/2], it represents a transition width of size w around the edge, so we want it to be mapped to an opacity within [0,1]. This can be expressed with:

float h = linearstep(-w/2.0, w/2.0, d);

Which can be unrolled and simplified into the following tiny form:

float h = clamp(0.5 + d/w, 0.0, 1.0);

We can also decide to make the transition on the outside or the inside boundary of the shape:

float h_in  = linearstep(0.0, w, d);  // or simply clamp(    d/w, 0.0, 1.0);
float h_out = linearstep(-w, 0.0, d); // or simply clamp(1.0+d/w, 0.0, 1.0);

And we can be creative and have a cursor indicating where we are on the border. If we give k=0 for inside, k=0.5 for centered and k=1 for outside:

float h = clamp(k + d/w, 0.0, 1.0);

For the negative inside and positive outside SDF, we simply swap the sign:

float h = clamp(k - d/w, 0.0, 1.0);

Any of these one-liners is all we need to have AA for our shape, but the question of what value to use for the ramp width w arises.

"anti-aliasing" with a width w oscillating within [0.1,0.3]

Pixel size

The difference between a blur and anti-aliasing is simply the width value. With AA, it's the size of a "pixel", and with a blur it's typically a user input or an arbitrarily large value.

If we are in 2D and have access to the pixel resolution, we can use it to get the pixel size. Note that this is closely tied to the coordinate space we use to calculate the SDF.

For example, let's say we have a canvas for which we don't know the aspect ratio, we can calculate the screen coordinates like this:

vec2 p = (2.0*gl_FragCoord.xy - resolution) / min(resolution.x, resolution.y);

This will give us a p value within [-1,1] on the shortest axis (the y-axis in landscape mode) and preserve a squared aspect ratio. That means the shortest axis will have an amplitude of 2, which means the number of pixel on that axis correspond to 2 units. As a result, the unit-width that will be used for the signed distance can be obtained with:

float w = 2.0 / min(resolution.x, resolution.y);

Remember that this is true only if the position p we use for the SDF is in that range. Basically, we have to adjust this formula to the coordinate space we are using.

anti-aliasing with a width w of 1 pixel

The same with a x10 resolution to better see the AA:

anti-aliasing with a width w of 1 pixel (resolution x10)

3D and numerical derivatives

But sometimes we might not have access to the resolution, or we may want to map that 2D SDF onto a plane in 3D or some other transformation. For example, a decal or a text on the wall in a video game. In that later case, if we were to use the screen resolution, it would lead to inconsistent anti-aliasing:

An SDF shape viewed from above and in perspective, resolution x4

We can see that, when put in perspective, the edge in the back gets way too sharp while the edge in the front becomes a bit too blurry. Fortunately, there is a magic trick we can use, the numerical derivatives:

float w = fwidth(d);

An SDF shape viewed from above and in perspective, using w=fwidth(d), resolution x4

Now we magically have a smooth pixel-wise anti-aliasing, no matter the perspective. What is this sorcery? 🧙

fwidth calculate the rate of change of a given variable using fragment-based numerical derivatives. Mathematically this is a L1-norm (also known as Taxicab or Manhattan norm) defined as: abs(dFdx(x)) + abs(dFdy(x)).

But how the hell is this providing a good pixel width estimate?

Let's see the simple case where we want observe the rate of change of one variable across one axis. For example, dFdx(px) where px is the pixel coordinate: int px = gl_FragCoord.x. We will have dFdx(px)=1. Why? Because x changes at a constant rate of 1 (exactly like our SDF) from one pixel to another. If we remap px to a value between [-1,1] using p=px/W*2.0-1.0 (where W is the number of pixels on the x axis), we can follow derivation rules and end up with: dFdx(p)=2.0/W. This matches with our initial pixel size computation w in our previous section.

Now when 3D and perspective distortions are involved, this still holds and you may be wondering why. The intuitive answer is that fwidth(d) is the rate of change of the signed distance as seen from the flat pixel screen perspective. In 3D view, in the back of the shape, the distance d changes sharply from one pixel to another (meaning fwidth(d) will be high), while in the front it's way smoother (meaning fwidth(d) will be low). So this numerical derivative is used to scale the distance back to a transition that works smoothly from the 2D pixel point of view.

Numerical derivatives refinement

Instead of fwidth, we could also use the L2-norm (also known as euclidean distance):

float w = length(vec2(dFdx(d), dFdy(d)));

This is more expensive than fwidth, but it can be considered as an alternative. The AA will be slightly different but it's hard to say which one really is better than the other:

L1-norm (left) vs L2-norm (right) for pixel estimate, resolution x8

Straight vs smooth(er) ramp

Instead of a linearstep() some people like to use smoothstep(). The main reason is probably because smoothstep() is available in builtin while linearstep() isn't. But is it a better choice?

Intuitively, to me at least, it makes perfect sense for the alpha value to follow a linear ramp. A few weeks ago I would have adamantly argued that it's actually a faster and more logical choice than its curved version smoothstep.

Well... I did some tests. With a large diffuse, here is what it looks like with a linear ramp:

A blurry shape using a linearstep transition with w=0.3

It looks like there is a brighter highlight around the border (before the fall-off), doesn't it? Well, it's an illusion, it's just our brain noticing the discontinuity and telling us about it.

With a smoothstep things get better:

A blurry shape using a smoothstep transition with w=0.3

I wasn't expecting that, so I stand corrected: smoothstep is actually a better choice. It's also builtin, so we won't need to define our own function.

Of course, one may prefer an even smoother curve, for example smootherstep() using the quintic curve instead of the hermite:

float smootherstep(float a, float b, float x) {
    float t = linearstep(a, b, x);
    return ((6.0*t-15.0)*t+10.0)*t*t*t; // quintic
}

A blurry shape using a smootherstep transition

Note

For pixel-wise anti-aliasing, the discontinuity won't be noticed so using a linear interpolation still is a perfectly valid choice.

Color space

When we have our anti-aliasing value, it's pretty much done. But we still have the question on how to use it. My previous examples were in black and white, but in many cases we need blending between colors. The question of how to blend is probably the most tricky of all, and my previous article was entirely dedicated to this particular issue.

In all the examples from this page, I've been using the OkLab blending because "it's perfect". But the reality is likely to force you to use a simple linear blending. For anti-aliasing, it's honestly just fine, the illusion still works out, but if you're trying to blur I would advise you against it and switch to a better colorspace like OkLab whenever possible.

See here how, because of how the human perception works, the linear blending does feel "bobby" and too large compared to OkLab:

A blurry shape blend using linear (left) or OkLab (right) blending

Summary

Given all these tools we can combine them according to our needs and preferences. As closing words, let me propose a few reference examples:

A "good enough" centered linear ramp working in 2D or 3D

vec2 p = (2.0*gl_FragCoord.xy - resolution) / min(resolution.x, resolution.y);
float d = sdWP(p, ...); // signed distance, positive inside, negative outside SDF (Wikipedia style)
float h = clamp(0.5 + d/fwidth(d), 0.0, 1.0);
vec3 c = mix(c0, c1, h); // this assumes c0 and c1 colors are in linear space

A smooth user blur working in 2D only

float r = min(resolution.x, resolution.y);
vec2 p = (2.0*gl_FragCoord.xy - resolution) / r;
float w = max(u_blur, 2.0/r); // blur should not be smaller than unit size
float d = sdIQ(p, ...); // signed distance, negative inside, positive outside SDF (iQuilez style)
float h = smoothstep(-w/2.0, w/2.0, -d); // smooth centered blur
vec3 c = mix(c0, c1, h); // this assumes c0 and c1 colors are in linear space

An outer anti-aliasing with a more refined unit width estimation

vec2 p = (2.0*gl_FragCoord.xy - resolution) / min(resolution.x, resolution.y);
float d = sdWP(p, ...); // signed distance, positive inside, negative outside SDF (Wikipedia style)
float w = length(vec2(dFdx(d),dFdy(d))); // L2-norm width estimation
float h = smoothstep(-w, 0.0, d); // smooth outer AA
vec3 c = mix(c0, c1, h); // this assumes c0 and c1 colors are in linear space

Anti-aliasing SDFs can be beautifully simple, and now we also understand better the magic behind. ✨

The current technology is not ready for proper blending

Fri, 18 Jul 2025 20:10:43 -0000

The idea that we must always linearize sRGB gradients or work in a perceptually uniform colorspace is starting to be accepted universally. But is it that simple?

When I learned about the subject, it felt like being handed a hammer and using it everywhere. The reality is a bit more nuanced. In this article we will see when to use which, how to use them, and we will then see why the situation is more dire than it looks.

Code snippets

Before we start, since we are going to use GLSL as language, following are the reference functions we will use for the rest of the article.

vec3 s2l(vec3 c) { // sRGB to linear
    return mix(c/12.92, pow((max(c,0.0)+0.055)/1.055,vec3(2.4)), step(vec3(0.04045),c));
}

vec3 l2s(vec3 c) { // linear to sRGB
    return mix(c*12.92, 1.055*pow(max(c,0.0),vec3(1./2.4))-0.055, step(vec3(0.0031308),c));
}

vec3 l2oklab(vec3 rgb) { // linear to OkLab
    const mat3 rgb2lms = mat3(
        +0.4122214708, +0.2119034982, +0.0883024619,
        +0.5363325363, +0.6806995451, +0.2817188376,
        +0.0514459929, +0.1073969566, +0.6299787005);
    const mat3 lms2lab = mat3(
        +0.2104542553, +1.9779984951, +0.0259040371,
        +0.7936177850, -2.4285922050, +0.7827717662,
        -0.0040720468, +0.4505937099, -0.8086757660);
    vec3 lms = rgb2lms * rgb;
    return lms2lab * pow(lms, vec3(1.0/3.0));
}

vec3 oklab2l(vec3 lab) { // OkLab to linear
    const mat3 lab2lms = mat3(
        +1.0000000000, +1.0000000000, +1.0000000000,
        +0.3963377774, -0.1055613458, -0.0894841775,
        +0.2158037573, -0.0638541728, -1.2914855480);
    const mat3 lms2rgb = mat3(
        +4.0767416621, -1.2684380046, -0.0041960863,
        -3.3077115913, +2.6097574011, -0.7034186147,
        +0.2309699292, -0.3413193965, +1.7076147010);
    vec3 lms = lab2lms * lab;
    return lms2rgb * (lms*lms*lms);
}

Also, the output of the pipeline will be expected to be sRGB all the time.

Color gradients

To illustrate how sRGB, linear RGB and OkLab respectively look like, let's interpolate between two colors with each one of them:

Color gradients from top to bottom: sRGB, linear, OkLab

The 3 stripes were generated like this:

vec3 o_srgb   = mix(c0, c1, v);
vec3 o_linear = l2s(mix(s2l(c0), s2l(c1), v));
vec3 o_oklab  = l2s(oklab2l(mix(l2oklab(s2l(c0)), l2oklab(s2l(c1)), v)));

Where v is simply the x coordinate between 0 and 1, c0 the left color, and c1 the right one.

Note

The input colors are considered to be sRGB in input. Similarly, we always make sure to output sRGB at the end (with l2s()) because that's what the pipeline expects.

Key takeaways:

sRGB is not acceptable because of this grayish/brownish zone, which is also perceived darker. In various situations this creates undesirable muddy midtones. In general, it's wrong and broken to do that.
Linear is better from a purely physical point of view as it models the mixing of light energy properly. But from a color perception point of view it's not ideal, for example here it has this transition into pinkish which might not be desirable.
The last one is using OkLab for a perceptually uniform gradient; it is the one providing the best result for our human perception, at a certain performance cost.

The general consensus is as follow: if you need a color transition within a shape or texture, or some sort of color map, OkLab is the best tool, while linear is cheap, physically correct, and usually acceptable visually.

But what about monochrome gradients?

Things are not as obvious as they seem when we work in monochrome. If instead of red and blue we pick black and white, this is what happens:

Grayscale gradients from top to bottom: sRGB, linear, OkLab

Suddenly this tells a whole different story. sRGB becomes perfectly acceptable, while linear favors way too much the lightness, and OkLab remains the best. The linear gradient felt acceptable before, but now it is highly questionable.

Just to be clear, the linear strip is linear, you can see it as linear energy or casually said "wattage", to which our perception does respond non-linearly.

At this point one may even argue that sRGB looks best.

So what can we do about this?

First of all, we always need to question what we are trying to achieve, and fortunately sometimes we can take a few shortcuts. For example, let's say we want to depict a heat map in black and white. In my previous article I had to display 2D noise, so I wanted the observer to experience a linear perception of the "height" of the noise. In this case, working in sRGB (that is, doing zero effort with regards to perception) is actually a better call than mixing between black and white in linear space:

Noise 2D with height as sRGB (left) or linear (right)

Here we are comparing these two:

vec3 o_srgb   = vec3(v);      // equivalent to mix(black, white, v)
vec3 o_linear = l2s(vec3(v)); // equivalent to l2s(mix(black, white, v))

Note

We removed the mix out of the formulas because black=vec3(0) and white=vec3(1), which have the same value when uncompressed to linear space.

To do things right we may want to use OkLab but this feels overkill since this is just a straightforward monochromatic signal. Fortunately, the perceptual lightness can be fairly simple to model. With monochromatic input, OkLab uses L=x³, which is basically equivalent to do a gamma correction with γ=3.

This means that we can simplify the OkLab interpolation we used before to the very simple:

vec3 o_oklab = l2s(vec3(v*v*v));  // equivalent to l2s(oklab2l(vec3(v,0,0)))

Noise 2D with height remapped to human lightness perception

Doing this simple operation is exactly equivalent to interpolating between black and white in OkLab space, except it's just 2 extra multiplications.

We still need to be extra careful if we want to swap the black and white. v needs to be swapped before the gamma encoding, and that means before the sRGB gamma encoding as well:

Top to bottom: srgb(1-v³) (incorrect), 1-srgb(v³) (incorrect), srgb((1-v)³)

One extra trick: combining gammas

sRGB has a curve that closely approximates a gamma correction γ=2.2. So sometimes, instead of using l2s(rgb), we may prefer to use the simpler pow(rgb,vec3(1.0/2.2)). It means we could replace l2s(vec3(v*v*v)) with the following to merge the two operations:

vec3 o_oklab = vec3(pow(v, 3.0/2.2)); // combination of v³ and gamma 2.2 (sRGB-like) encoding

And the white-to-black version:

vec3 o_oklab = vec3(pow(1.0-v, 3.0/2.2));

Warning

Whenever you use pow, make sure your input is positive. Adding a max(v,0.0) for safety might be reasonable in certain cases.

The difference between a proper sRGB conversion and the combined gamma is pretty small:

Top: srgb(v³), bottom: v^(3/2.2)

Alpha blending and pre-multiplication

Sometimes, instead of fading colors into each others, we need to compose shapes, textures, masks, ... This need for compositing, or blending, arises when the pipelines are separated, meaning we are not working in the same fragment shader for everything. For example, we could have a shape generated in a fragment, which we need to overlay onto a surface. That shape might have some non-binary transparency, either for anti-aliasing purposes, a blur, or similar.

An example of a shape partially transparent

If that shape were to be blend onto another colored surface, we would like to have the same effect as the gradient earlier. For well-known reasons, it is likely that this shape would end up as a pre-multiplied color, which would be blend onto one or more layers. If what I just said is confusing, I recommend checking out this good article on alpha compositing from Bartosz Ciechanowski. The literature is quite extensive on the subject so I will assume familiarity with it.

Of course, if we are to do things right, the blending would have to happen in linear space. Do not consider sRGB blending in alpha blending, it's an even more terrible idea than before because of the bilinear filtering, transforms or mipmaps that can happen between the pre-multiplication and the blending itself.

But that means we would end up with the linear gradient shortcomings from earlier, wouldn't we? And this is where things get ugly.

Look at the difference between a linear and an OkLab blending, in black and white:

Blending of a blurry white circle onto black, left is linear, right is OkLab

If we invert the colors:

Blending of a blurry black circle onto white, left is linear, right is OkLab

We have the exact same problem as earlier, but seeing it with an actual blending of shapes makes the problem particularly striking. The white and black OkLab circles look the same size (because they are), and they don't have the unfortunate "bobbing" effect of the linear version (on the white onto black).

Warning

The OkLab blending is done with pre-multiplied Lab colors. It is important not to pre-multiply linear values which are then converted to OkLab, this will give very unexpected results.

The problem is, it is very unlikely that your whole graphics pipeline would switch to OkLab for every textures and buffers. And since most of the time the pipelines are built for more than just black and white, the cube hack suggested earlier has a very limited scope. In the case of shape blending, it is almost certain that the whole pipeline would not be contained in a single shader where you can just mix in OkLab. You're probably thinking of using sRGB, but in a blending pipeline this really is a terrible idea.

Final words

In practice, neither sRGB nor pure linear blending give good results, and using OkLab is not always an option. And unfortunately, I don't have a good answer to this whole situation. My next article is about anti-aliasing where this problem also exists, and I must admit this whole ordeal puts me in quite some distress; I had to talk about this issue first.

Sharing everything I could understand about gradient noise

Fri, 06 Jun 2025 14:45:38 -0000

You've most likely heard about gradient noise through the name Perlin noise, which refers to one particular implementation with various CPU optimizations. Because it's an incredible tool for creative work, it's used virtually everywhere: visual effects, video games, procedural mathematical art, etc. While getting it right can sometimes be subtle, a "broken" implementation can still look good or interesting. After all, "it looks fine, and I'm an artist".

In order to gain a deeper and more meaningful understanding we will start studying the 1D version (a case often omitted in the literature), then slowly climb our way up in dimensions and complexity. We'll also work from a GPU perspective rather than a CPU-based one, hence all code snippets and visuals here are implemented in WebGL2/GLSL (hopefully without being too heavy on performance). They should run on most modern devices; let me know if you run into issues.

Before we begin, credit where it's due: most of the material here are nothing new. This article is the result of weeks of studying and experimenting with the maths from Inigo Quilez's incredible pages and other scattered resources over the Internet. But as rich and valuable these resources are, they sometimes move quickly over the details, assuming they're obvious. This post is an attempt to fill those gaps.

A welcoming wavy 1D gradient noise signal

Hashing function and pseudo-random values

At the most elementary level, we need a deterministic coordinate based pseudo-random system. More specifically, for any given integer coordinate we need a random value, and as uniformly distributed as possible. Something like:

\begin{aligned} h(-3) &= -0.006124 \\ h(-2) &= -0.996686 \\ h(-1) &= 0.200864 \\ h(0) &= -1.000000 \\ h(1) &= 0.053313 \\ h(2) &= -0.893312 \\ h(3) &= 0.854923 \\ \text{...} \end{aligned}

Perlin's implementation relies on a permutation table, which is convenient when working on the CPU, but more awkward for a shader. On the GPU, most people rely on various floating point hacks or sub-optimal bit tricks, which often fall short when exploring the full 32-bit range of inputs.

Note

We can not use a PRNG because it relies on a state where we need determinism (for one coordinate, we always want the same random value). LCG (Linear Congruential Generator) are a sub-classes of PRNG where the state is actually the returned value. But even then, we can not re-use the previously returned value since we need seeking; that is the ability to get the random value associated with our integer coordinate. Trying to feed a PRNG/LCG with our coordinate instead of the expected state would be equivalent to changing the seed at every call, and this may create an important bias in the random distribution. This is the reason why we need integer hashing instead.

So we need a hashing function, and we will have to limit ourselves to 32-bit because we're on the GPU. Fortunately, we won't have to look for very long because in 2018 Chris Wellons found a pretty good one he named lowbias32, which was later refined by TheIronBorn.

This is the first building block we will be using, the hashing function:

uint hash(uint x) {
    x = (x ^ (x >> 16)) * 0x21f0aaadU;
    x = (x ^ (x >> 15)) * 0x735a2d97U;
    return x ^ (x >> 15);
}

This is sweet and all, but a 32-bit unsigned integer is not directly useful by itself, we need a normalized float. We could naively divide by 0xffffffff, but we'll end up with a nonuniform distribution (this is not as important as it may sound to be honest). Instead, we will adapt the technique presented by Vigna Sebastiano for doubles to floats, and in GLSL:

float u2f(uint x) { return float(x >> 8U) * uintBitsToFloat(0x33800000U); }

Combining these two functions into u2f(hash(x)) maps any 32-bit coordinate x to a random float in [0,1).

To match our h(x) function from earlier and make it more "signal-like", we can optionally center it around 0 by remapping it from [0,1) to [-1,1): h(x)=u2f(hash(x))*2.0-1.0.

Expanding the hash function to more dimensions

When we will work in 2 or more dimensions, our input grid coordinates will be more than one value, but we need a way to feed them to our single parameter hash function. One trick is to use nested xor hash:

uint hash(uvec2 x) { return hash(x.x ^ hash(x.y)); }  // for 2D input
uint hash(uvec3 x) { return hash(x.x ^ hash(x.yz)); } // for 3D input

Using the pixel coordinates as input, we can test the hashing function in 2D:

float stepnoise2(vec2 p) {
    ivec2 i = ivec2(floor(p));    // integer coordinate, or lattice
    return u2f(hash(uvec2(i)));   // non-centered h(x)
}

Display h(x,y) by stepping the contiguous coordinates

Warning

We're going through an intermediate signed integer conversion before going to unsigned to avoid issues with negative coordinates. Quoting GLSL 4.60 specifications: "It is undefined to convert a negative floating-point value to an uint". With this conversion we will only have a problem when the coordinates go outside the signed 32-bit range.

Basic signal and white noise

Aside from stepping, we can also assign values to the integer coordinates only and interpolate linearly between them. Let's try this in 1D:

float value(int x) { return u2f(hash(uint(x)))*2.0 - 1.0; } // h(x)

float vnoise1_linear(float p) {
    int i = int(floor(p));                // integer coordinate, or lattice
    float f = fract(p);                   // x-position between the 2 surrounding values
    return mix(value(i), value(i+1), f);  // linear interpolation between the two
}

Feeding this function with the x-axis coordinate, we get the amplitude (height) of the signal:

Basic 1D value noise with linear interpolation

Note

This might not be obvious yet, but the beauty of this is that this "infinite" signal is deterministic and thus seekable. Indeed, we can move forward and backward to any real position p and instantly know how the signal looks like. This is essential for procedural programming.

But the linear interpolation is usually not that great for a signal, so instead of a straight line we will use a smooth fade. The two commonly used functions are:

The cubic Hermite curve, f(t)=3t^2-2t^3 (also used in GLSL smoothstep()) initially used by Ken Perlin in his first Perlin Noise implementation.
The more modern (and complex) quintic curve f(t)=6t^5-15t^4+10t^3 introduced in 2002 by Ken Perlin in his proposed improved version of the Perlin Noise, in order to address discontinuities in the 2nd order derivatives f"(t).

For the record, in GLSL:

float fade_quintic(float t) { return ((6.0*t-15.0)*t+10.0)*t*t*t; }
float fade_hermite(float t) { return (3.0-2.0*t)*t*t; }

We will stick with the quintic for the rest of the article:

#define fade fade_quintic

float vnoise1(float p) {
    int i = int(floor(p));
    float f = fract(p);
    float a = fade(f);
    return mix(value(i), value(i+1), a);
}

1D value noise with quintic interpolation as fading function

1D gradient noise

Still, a signal generated in such a way may not be desirable due to the abrupt changes in slope/frequency; it is too "unstable". So instead of using the random as noise values directly, we interpret them as gradient: this is the gradient noise.

How the gradient values affect the signal shape

Note

If we're pedantic, in 1D they can't exactly be called gradients, we should use the term slopes, or angles. But we will still do it for consistency with 2 and more dimensions.

For each lattice, we decide to give them y=0 at regular interval, and have their random gradient impacts the surrounding curve. This may sound complicated to implement but in practice it's 2 multiplications and 1 subtraction more than the value noise (for 1D at least):

float grad(int x) { // int lattice to random [-1,1)
    return u2f(hash(uint(x))) * 2.0 - 1.0;
}

float noise1(float p) {
    int i = int(floor(p));
    float g0 = grad(i);
    float g1 = grad(i + 1);

    float f = fract(p);
    float v0 = g0 * f;
    float v1 = g1 * (f - 1.0);

    float a = fade(f);
    return mix(v0, v1, a);
}

1D gradient noise with slope indicators on the lattice coordinates

Geometrically speaking, v_0 and v_1 are y-coordinates obtained by extending the 2 slopes (the little sticks at each lattice) around our current target point p and finding where the vertical line x=p intersects them. Then we smoothly interpolate between them using our fade function.

Note

One important property of the gradient noise is that it passes through 0 at every lattice, meaning at a constant rate. In other words, \textbf{noise}_1(p)=0 whenever p is an integer.

Expanding to 2 dimensions

In 2D, each lattice point stores a 2-component gradient vector. To evaluate noise at a point p=(x,y), instead of a simple multiplication we compute the dot product of each gradient vector with the vector from the lattice corner to (x,y), then bilinearly interpolate those four dot products using a 2D fade function.

#define bmix(a,b,c,d,x,y) mix(mix(a,b,x),mix(c,d,x),y) // bilinear interpolation

float noise2(vec2 p) {
    ivec2 i = ivec2(floor(p));
    vec2 g0 = grad(i);
    vec2 g1 = grad(i + ivec2(1, 0));
    vec2 g2 = grad(i + ivec2(0, 1));
    vec2 g3 = grad(i + ivec2(1, 1));

    vec2 f = fract(p);
    float v0 = dot(g0, f);
    float v1 = dot(g1, f - vec2(1.0, 0.0));
    float v2 = dot(g2, f - vec2(0.0, 1.0));
    float v3 = dot(g3, f - vec2(1.0, 1.0));

    vec2 a = fade(f);
    return bmix(v0, v1, v2, v3, a.x, a.y);
}

Since gradients are now in 2D, the random gradient function needs to be extended: we need a normalized vector generator, that is something that gives us a unit vector pointing in any direction.

A first solution would be to call the hash function twice (on itself typically) to obtain random (x,y) coordinates in a square, that we then normalize to get them on a circle:

vec2 grad(ivec2 x) { // ivec2 lattice to random 2D unit vector (normalized square point)
    uint h1 = hash(uvec2(x));
    uint h2 = hash(h1);
    return normalize(vec2(u2f(h1), u2f(h2)) * 2.0 - 1.0);
}

But we can do better with only one hash and some trigonometry:

const float TAU = 6.283185307179586;

vec2 grad(ivec2 x) { // ivec2 lattice to random 2D unit vector (circle point)
    float angle = u2f(hash(uvec2(x))) * TAU;
    return vec2(cos(angle), sin(angle));
}

And this is enough to get our 2D gradient noise:

2D gradient noise (with y-axis coordinates that cover -10 to 10)

Expanding to 3 dimensions

Similarly, for 3D gradient noise we will need these 2 changes:

The interpolation happens between the 8 points of a cube: it's a trilinear interpolation, a combination of 2 bilinear interpolations (itself being a combination of 3 linear interpolations)
The random unit vectors used as gradients need to be distributed evenly on a sphere instead of circle since we are working in 3D

vec3 grad(ivec3 x) { // ivec3 lattice to random 3D unit vector (sphere point)
    uint h0 = hash(uvec3(x));
    uint h1 = hash(h0);
    // use the first random for the polar angle (latitude)
    float c = 2.0*u2f(h0) - 1.0, // c = cos(theta) = cos(acos(2x-1)) = 2x-1
          s = sqrt(1.0 - c*c);   // s = sin(theta) = sin(acos(c)) = sqrt(1-c*c)
    float phi = TAU * u2f(h1);   // use the 2nd random for the azimuth (longitude)
    return vec3(cos(phi) * s, sin(phi) * s, c);
}

#define tmix(a,b,c,d,e,f,g,h,x,y,z) mix(bmix(a,b,c,d,x,y),bmix(e,f,g,h,x,y),z) // trilinear interpolation

float noise3(vec3 p) {
    ivec3 i = ivec3(floor(p));
    vec3 g0 = grad(i);
    vec3 g1 = grad(i + ivec3(1, 0, 0));
    vec3 g2 = grad(i + ivec3(0, 1, 0));
    vec3 g3 = grad(i + ivec3(1, 1, 0));
    vec3 g4 = grad(i + ivec3(0, 0, 1));
    vec3 g5 = grad(i + ivec3(1, 0, 1));
    vec3 g6 = grad(i + ivec3(0, 1, 1));
    vec3 g7 = grad(i + ivec3(1, 1, 1));

    vec3 f = fract(p);
    float v0 = dot(g0, f);
    float v1 = dot(g1, f - vec3(1, 0, 0));
    float v2 = dot(g2, f - vec3(0, 1, 0));
    float v3 = dot(g3, f - vec3(1, 1, 0));
    float v4 = dot(g4, f - vec3(0, 0, 1));
    float v5 = dot(g5, f - vec3(1, 0, 1));
    float v6 = dot(g6, f - vec3(0, 1, 1));
    float v7 = dot(g7, f - vec3(1, 1, 1));

    vec3 a = fade(f);
    return tmix(v0, v1, v2, v3, v4, v5, v6, v7, a.x, a.y, a.z);
}

3D gradient noise on a sphere (with 10x unzoom)

Note

The spherical representation has nothing to do with the selection of a random 3D unit vector on a sphere. The noise in this implementation applies to any 3D geometry. The choice of a sphere for the display was just a simple 3D shape to lay it on. If we were in 2D we could also feed the 3d noise function with p=(x,y,t) where t is the current time, giving us a "cloudy" atmospheric rendering.

The random gradient in 3D might be a bit expensive, so we may consider simpler approaches, like normalizing a random position in a cube, similar to what we've initially suggested for 2D noise. From my observation, it doesn't seem to have any noticeable impact visually, but maybe it would have under certain circumstances.

Fractal Brownian Motion (fBm)

The idea behind fBm is to sum multiple "octaves" of noise together to construct a more refined pattern. In the most common cases, we would raise the frequency by doubling it (called a "lacunarity" factor), and halving the amplitude (called a "gain" or "persistence" factor):

Multiple signals

The algorithm is pretty much the same in all dimensions, it's simply a sum of signals. In 2D for example we can write:

const float LACUNARITY = 1.98;
const float GAIN = 0.51;

float fbm(vec2 p, int octaves) {
    float sum = 0.0;
    float amp = 1.0, freq = 1.0;
    for (int i = 0; i < octaves; i++) {
        sum += amp * noise2(p * freq);
        freq *= LACUNARITY;
        amp *= GAIN;
    }
    return sum;
}

2D gradient noise with 5 octaves (with y-axis coordinates that cover -2.5 to 2.5)

Note

We're not using exactly 2.0 and 0.5 for lacunarity and gain to break correlations.

And in 3D for completeness (aside from the raytracer to get a sphere, the noise code is the same, it just calls noise3(p*freq) and uses a vec3 p input):

3D gradient noise with 5 octaves

Derivatives

Derivatives (that is the rate of change of the signal) are useful in many situations. For example, you may have noticed that the 1D signals in this article tend to have a lighter color when the slope is steep compared to when it's flatter: the derivative is used to interpolate between the 2 colors.

Also in 1D case, to display the curve with correct thickness, I'm also using the derivatives for Inigo's distance to curve trick):

float dist = abs(v - p.y) / sqrt(1.0 + d*d); // v: curve value, p: position, d: derivative of curve

Using this signed distance like any other, we can display the curve smoothly:

1D gradient noise

In higher dimension, we may want to use them to get some lighting: indeed, with the derivatives we can compute the normal, which is then used for reflections:

// Lambertian lighting hack
vec3 normal = normalize(vec3(-d, 1.0)); // normal from the 2D partial derivatives d
float a = 5.0*TAU/8.0; // light from south-west
vec3 light_direction = normalize(vec3(cos(a), sin(a), 2.0));
float lighting = max(dot(normal, light_direction), 0.0);
col *= lighting;

2D gradient noise with (right) and without (left) lighting

If you're into terrain generation, another use case is to fake erosion by scaling the value of each noise layer of the fBm by \frac{1}{1+\|\sum_{i=0}^{octaves}d_i\|^2}.

float fbm2e(vec2 p) {
    float sum = 0.0;
    float amp = 1.0, freq = 1.0;
    vec2 d = vec2(0.0);
    for (int i = 0; i < octaves; i++) {
        vec3 n = noise2d(p * freq);    // adjusted noise2() returning the partial derivatives in .xy
        d += n.xy;                     // cumulated derivatives without frequency scaling
        float w = 1.0/(1.0+dot(d,d));  // gradient/slope based weight
        sum += amp * n.z * w;          // value damped down by the weight
        freq *= LACUNARITY;
        amp *= GAIN;
    }
    return sum;
}

2D gradient noise with (right) and without (left) erosion

The intuitive idea behind the formula here is that each new layer get "muted" when the gradients indicate a steep mountain, preventing them from being "rugged" more by higher frequency noises, and thus giving a sharper feel.

Anyway, these are random things we can do with the derivatives, but there are likely others I'm forgetting. The point is, they are particularly useful so we're going to study them.

Numerical vs analytical derivatives

We have two methods to get the derivatives:

The numerical method, where we compute the rate of change between 2 close points. It "works" with all curves (given enough precision to work with), but it has an accuracy (and sometimes speed) issue.
The analytical method, where we derive the exact mathematical formula.

The numerical method can be implemented on the GPU thanks to the dFdx and dFdy functions. These functions use the local and the neighbor fragments data to calculate a derivative of the specified value. It's one of the rare function that communicate information cross-fragment, and as you can guess it requires synchronization and thus has performance implications.

But for example, let's say we take our last 2D noise scene and compare the numerical vs the analytical derivatives:

2D gradient noise with its partial derivatives (both analytical and numerical)

On the left we have our standard 2D gradient fBm noise, and on the right the length of the derivatives of the noise: first the analytical version, then the numerical one. The latter should appear pixelized.

Numerical derivatives

The numerical derivatives were obtained with the following:

float w = 2.0/resolution.y;           // size of a pixel
float v = noise2(p);                  // noise value at position p
vec2 d = vec2(dFdx(v), dFdy(v)) / w;  // numerical partial derivatives

Note

It is also possible to derive numerically ourselves with finite differences, sampling the noise function for the neighbor positions within the same fragment, but this is going to be extremely expensive.

Analytical derivatives

To get the analytical derivatives we need to combine multiple derivatives found in the noise functions, starting from the fading function derivative. This one is easy:

\begin{aligned} \mathrm{fade}(t) &= 6t^5-15t^4+10t^3 \\ \mathrm{fade}'(t) &= 30t^2(t^2-2t+1) \end{aligned}

But we will also need the derivatives of the interpolation functions for each dimension:

lerp: \mathrm{mix}(a,b,x) = (1-x)a + bx
bilerp: \mathrm{bmix}(a,b,c,d,x,y) = \mathrm{mix}(\mathrm{mix}(a,b,x),\mathrm{mix}(c,d,x),y)
trilerp: \mathrm{tmix}(a,b,c,d,e,f,g,h,x,y,z) = \mathrm{mix}(\mathrm{bmix}(a,b,c,d,x,y),\mathrm{bmix}(e,f,g,h,x,y),z)

So here they are:

\partial_x\mathrm{mix}=b-a
\nabla\mathrm{bmix}=\mathrm{mix}(\begin{bmatrix}b\\c\end{bmatrix}-a,d-\begin{bmatrix}c\\b\end{bmatrix},\begin{bmatrix}y\\x\end{bmatrix})
\nabla\mathrm{tmix}=\mathrm{bmix}(\begin{bmatrix}b\\c\\e\end{bmatrix}-a,\begin{bmatrix}d-c\\d-b\\f-b\end{bmatrix},\begin{bmatrix}f-e\\g-e\\g-c\end{bmatrix},h-\begin{bmatrix}g\\f\\d\end{bmatrix},\begin{bmatrix}y\\x\\x\end{bmatrix},\begin{bmatrix}z\\z\\y\end{bmatrix})

Following is a proof for bilinear and trilinear interpolation function partial derivatives, you can skip them if you're not into nasty orgy of math symbols.

Note

I'm introducing a new notation for the partial derivatives with values because the usual ones are awful to make and read. An example is better than a long explanation, so instead of something like \frac{\partial_f(x,y,z)}{\partial_x}(17,u+v,v^2) or its horrendous vertical bar version, I will instead use: \partial_xf(x=17,y=u+v,z=v^2)

\begin{aligned} \partial_x\mathrm{bmix} &= \partial_a\mathrm{mix}\cdot\partial_x\mathrm{mix} + \partial_b\mathrm{mix}\cdot\partial_x\mathrm{mix} \\ &= \partial_a\mathrm{mix}(a=\mathrm{mix}(a,b,x),b=\mathrm{mix}(c,d,x),x=y) \cdot \partial_x\mathrm{mix}(a=a,b=b,x=x) \\ &+ \partial_b\mathrm{mix}(a=\mathrm{mix}(a,b,x),b=\mathrm{mix}(c,d,x),x=y) \cdot \partial_x\mathrm{mix}(a=c,b=d,x=x) \\ &= (1-y)(b-a) + y(d-c) \\ &= \boxed{\mathrm{mix}(b-a,d-c,y)} \\ \\ \partial_y\mathrm{bmix} &= \partial_x\mathrm{mix}_0(a=\mathrm{mix}(a,b,x), b=\mathrm{mix}(c,d,x), x=y) \\ &= \mathrm{mix}(c,d,x) - \mathrm{mix}(a,b,x) \\ &= \boxed{\mathrm{mix}(c-a,d-b,x)} \\ \\ \partial_x \mathrm{tmix} &= \partial_a\mathrm{mix}\cdot\partial_x\mathrm{bmix} + \partial_b\mathrm{mix}\cdot\partial_x\mathrm{bmix} \\ &= \partial_a\mathrm{mix}(a=\mathrm{bmix}(a,b,c,d,x,y),b=\mathrm{bmix}(e,f,g,h,x,y),z) \\ &\cdot \partial_x\mathrm{bmix}(a=a,b=b,c=c,d=d,x=x,y=y) \\ &+ \partial_b\mathrm{mix}(a=\mathrm{bmix}(a,b,c,d,x,y),b=\mathrm{bmix}(e,f,g,h,x,y),z) \\ &\cdot \partial_x\mathrm{bmix}(a=e,b=f,c=g,d=h,x=x,y=y) \\ &= (1-z)\mathrm{mix}(b-a,d-c,y) + z\mathrm{mix}(f-e,h-g,y) \\ &= \mathrm{mix}(\mathrm{mix}(b-a,d-c,y), z\mathrm{mix}(f-e,h-g,y), z) \\ &= \boxed{\mathrm{bmix}(b-a,d-c,f-e,h-g,y,z)} \\ \\ \partial_y \mathrm{tmix} &= \partial_a\mathrm{mix}\cdot\partial_y\mathrm{bmix} + \partial_b\mathrm{mix}\cdot\partial_y\mathrm{bmix} \\ &= \partial_a\mathrm{mix}(a=\mathrm{bmix}(a,b,c,d,x,y),b=\mathrm{bmix}(e,f,g,h,x,y),z) \\ &\cdot \partial_y\mathrm{bmix}(a=a,b=b,c=c,d=d,x=x,y=y) \\ &+ \partial_b\mathrm{mix}(a=\mathrm{bmix}(a,b,c,d,x,y),b=\mathrm{bmix}(e,f,g,h,x,y),z) \\ &\cdot \partial_y\mathrm{bmix}(a=e,b=f,c=g,d=h,x=x,y=y) \\ &= (1-z)\mathrm{mix}(c-a,d-b,x) + z\mathrm{mix}(g-e,h-f,x) \\ &= \mathrm{mix}(\mathrm{mix}(c-a,d-b,x), z\mathrm{mix}(g-e,h-f,x), z) \\ &= \boxed{\mathrm{bmix}(c-a,d-b,g-e,h-f,x,z)} \\ \\ \partial_z\mathrm{tmix} &= \partial_x\mathrm{mix} \\ &= \partial_x\mathrm{mix}(a=\mathrm{bmix}(a,b,c,d,x,y),b=\mathrm{bmix}(e,f,g,h,x,y),x=z) \\ &= \mathrm{bmix}(e,f,g,h,x,y)-\mathrm{bmix}(a,b,c,d,x,y) \\ &= \boxed{\mathrm{bmix}(e-a,f-b,g-c,h-d,x,y)} \end{aligned}

Finally, given these partial derivatives, we can figure out how to compute the derivatives of the noise itself. It helps to know that fract(p) derivative is 1 (mostly) and floor(p) derivative is 0 (mostly), so things simplify themselves out fairly cleanly. I spare you the details this time:

\begin{aligned} \text{1D: } & \boxed{\textbf{mix}(g_0,g_1,\textbf{fade}(f))+\partial_x\textbf{mix}(v_0,v_1)\cdot\textbf{fade}'(f)} \\ \text{2D: } & \boxed{\textbf{bmix}(g_0,g_1,g_2,g_3,\textbf{fade}(f))+\nabla\mathrm{bmix}(v_0,v_1,v_2,v_3,\textbf{fade}(f))\cdot\textbf{fade}'(f)} \\ \text{3D: } & \boxed{\textbf{tmix}(g_0,g_1,g_2,g_3,g_4,g_5,g_6,g_7,\textbf{fade}(f))+\nabla\mathrm{tmix}(v_0,v_1,v_2,v_3,v_4,v_5,v_6,v_7,\textbf{fade}(f))\cdot\textbf{fade}'(f)} \end{aligned}

Adjusting our gradient noise functions to return the derivatives along with the noise value itself:

vec2 noise1d(float p) {
    int i = int(floor(p));
    float g0 = grad(i);
    float g1 = grad(i + 1);

    float f = fract(p);
    float v0 = g0 * f;
    float v1 = g1 * (f - 1.0);

    float a = fade(f);
    float v = mix(v0, v1, a);

    float g = mix(g0, g1, a);
    float d = v1 - v0;                    // derivative of mix with respect to the interpolant
    float da = ((f-2.0)*f+1.0)*30.0*f*f;  // fade'(t): derivative of quintic interpolant

    return vec2(g + d*da, v);
}

vec3 noise2d(vec2 p) {
    // [...]
    float v = bmix(v0, v1, v2, v3, a.x, a.y);

    vec2 g = bmix(g0, g1, g2, g3, a.x, a.y);
    vec2 d = mix(vec2(v1,v2)-v0, v3-vec2(v2,v1), a.yx); // derivatives of bmix with respect to the interpolant
    vec2 da = ((f-2.0)*f+1.0)*30.0*f*f;

    return vec3(g + d*da, v);
}

vec4 noise3d(vec3 p) {
    // [...]
    float v = tmix(v0, v1, v2, v3, v4, v5, v6, v7, a.x, a.y, a.z);

    vec3 g = tmix(g0, g1, g2, g3, g4, g5, g6, g7, a.x, a.y, a.z);
    vec3 d = bmix(vec3(v1,v2,v4)-v0, vec3(v3-v2,v3-v1,v5-v1),  // derivatives of tmix with
                  vec3(v5-v4,v6-v4,v6-v2), v7-vec3(v6,v5,v3),  // respect to the interpolant
                  a.yxx, a.zzy);
    vec3 da = ((f-2.0)*f+1.0)*30.0*f*f;

    return vec4(g + d*da, v);
}

Note

The derivatives are in the first component so that given n=noise3d(p) we can write vec3 d=n.xyz for x/y/z partial derivatives instead of the more awkward vec3 d=n.yzw.

To get the equivalent derivatives for value noise, we just set g=0 in the final expression of the derivatives.

It is possible to unroll the nested mix expression to (maybe) make it faster, but I find the mix expressions simple, elegant, and likely more numerically stable.

Using the derivatives

There are important things to take into consideration when working with the derivatives. Most notably, it's important to follow the chain rule and the product rule. For example, let's say we're working with a noise function that returns the derivatives:

float freq = 0.1;
vec4 n = noise3d(x*freq);
float v = n.a;            // value
vec2 d = n.xyz * freq;    // partial derivatives

If we multiply the input of our function by the frequency, then we need to multiply the derivatives by the same frequency factor.

Similarly, let's say we want to move v from [-1,1] to [0,1]:

v = (v + 1.0) / 2.0; // [-1,1] -> [0,1]
d = d / 2.0;

The intuitive explanation is that if we are squeezing the value in half its original size, then the derivatives (slopes) are getting flatter as well.

This may not sound very important, but failing to propagate these scale factors correctly often leads to subtle bugs. For example in the fBm, if we want it to return the correct derivatives it is necessary to do something like that (notice the freq):

vec4 fbm3d(vec3 p) {
    vec4 sum = vec4(0.0);
    float amp = 1.0, freq = 1.0;
    for (int i = 0; i < octaves; i++) {
        vec4 n = noise3d(p * freq);
        sum.xyz += amp * n.xyz * freq; // derivatives
        sum.a += amp * n.a; // value
        freq *= LACUNARITY;
        amp *= GAIN;
    }
    return sum;
}

Things get tricky when the noise is being adjusted with the derivatives themselves as we saw before with the erosion, especially since this would imply second order derivatives.

Similarly, there is a technique involving the rotation of p at every iteration of the fBm to reduce the correlations between noise layers. This rotation is a linear transformation, but it requires careful adjustments to the derivatives as well. Implementing these transformations correctly is left as an exercise; careful bookkeeping of derivatives is essential.

Going further

With this introduction we only explored the tip of the iceberg. For example, you'll be interested to know that there are alternative noises such as OpenSimplex, where we interpolate with lattice positions on a simplex grid (triangles in 2D, tetrahedra in 3D) instead of a rectangular grid. It has useful properties, for example in fBm it yields fewer directional artifacts, eliminating the need for ad-hoc rotations.

Speaking of fBm, you may want to check out domain warping where nested fBm together make fancy effects:

Domain warping with fbm2(p+fbm2(p+fbm2(p+t)))

Another thing you'll be tempted to research is how to consistently scale the noise so that it fits into a controlled range of value. Spoiler alert: it's a particularly complex subject.

Similarly, we stopped ourselves at 3D, but 4D is also useful, for example let's say we want to morph the 3D noise with time: p=(x,y,z,t).

I hope this article was able to give a good overview of the concepts, and I'll see you next time for new adventures.

Fixing the iterative damping interpolation in video games

Sat, 18 May 2024 12:22:15 -0000

As I'm exploring the fantastic world of indie game development lately, I end up watching a large number of video tutorials on the subject. Even though the quality of the content is pretty variable, I'm very grateful to the creators for it. That being said, I couldn't help noticing this particular bit times and times again:

a = lerp(a, B, delta * RATE)

Behind this apparent banal call hides a terrible curse, forever perpetrated by innocent souls on the Internet.

In this article we will study what it's trying to achieve, how it works, why it's wrong, and then we'll come up with a good solution to the initial problem.

The usual warning: I don't have a mathematics or academic background, so the article is addressed at other neanderthals like myself, who managed to understand that pressing keys on a keyboard make pixels turn on and off.

What is it?

Let's start from the beginning. We're in a game engine main loop callback called at a regular interval (roughly), passing down the time difference from the last call.

In Godot engine, it looks like this:

func _physics_process(delta: float):
    ...

If the game is configured to refresh at 60 FPS, we can expect this function to be called around 60 times per second with delta = 1/60 = 0.01666....

As a game developer, we want some smooth animations for all kind of transformations. For example, we may want the speed of the player to go down to zero as they release the moving key. We could do that linearly, but to make the stop less brutal and robotic we want to slow down the speed progressively.

Linear (top) versus smooth/exponential (bottom) animation

Virtually every tutorial will suggest updating some random variable with something like that:

velocity = lerp(velocity, 0, delta * RATE)

At 60 FPS, a decay RATE defined to 3.5, and an initial velocity of 100, the velocity will go down to 0 following this curve:

Example curve of a decaying variable

Note

velocity is just a variable name example, it can be found in many other contexts

If you're familiar with lerp() ("linear interpolation") you may be wondering why this is making a curve. Indeed, this lerp() function, also known as mix(), is a simple linear function defined as lerp(a,b,x) = x*(b-a) + a or its alternative stable form lerp(a,b,x) = (1-a)x + bx. For more information, see a previous article about this particular function. But here we are re-using the previous value, so this essentially means nesting lerp() function calls, which expands into a power formula, forming a curve composed of a chain of small straight segments.

Why is it wrong?

The main issue is that the formula is heavily depending on the refresh rate. If the game is supposed to work at 30, 60, or 144 FPS, then it means the physics engine is going to behave differently.

Here is an illustration of the kind of instability we can expect:

Comparison of the curves at different frame rates with the problematic formula

Note that the inaccuracy when compared to an ideal curve is not the issue here. The problem is that the game mechanics are different depending on the hardware, the system, and the wind direction observed in a small island of Japan. Imagine being able to jump further if we replace our 60Hz monitor with a 144Hz one, that would be some nasty pay to win incentive.

We may be able to get away with this by forcing a constant refresh rate for the game and consider this a non-issue (I'm not convinced this is achievable on all engines and platforms), but then we meet another problem: the device may not be able to hold this requirement at all times because of potential lags (for reasons that may be outside our control). That's right, so far we assumed delta=1/FPS but that's merely a target, it could fluctuate, causing mild to dramatic situations gameplay wise.

One last issue with that formula is the situation of a huge delay spike, causing an overshooting of the target. For example if we have RATE=3 and we end up with a frame that takes 500ms for whatever random reason, we're going to interpolate with a value of 1.5, which is way above 1. This is easily fixed by maxing out the 3rd argument of lerp to 1, but we have to keep that issue in mind.

To summarize, the formula is:

not frame rate agnostic ❌
non deterministic ❌
vulnerable to overshooting ❌

If you're not interested in the gory details on the how, you can now jump straight to the conclusion for a better alternative.

Study

We're going to switch to a more mathematical notation from now on. It's only going to be linear algebra, nothing particularly fancy, but we're going to make a mess of 1 letter symbols, bear with me.

Let's name the exhaustive list of inputs of our problem:

Initial value: a_0=\Alpha (from where we start, only used once)
Target value: \Beta (where we are going, constant value)
Time delta: \Delta_n (time difference from last call)
The rate of change: R (arbitrary scaling user constant)
Original sequence: a_{n+1} = \mathrm{lerp}(a_n, \Beta, R\Delta_n) (the code in the main loop callback)
Frame rate: F (the target frame rate, for example 60 FPS)
Time: t (animation time elapsed)

What we are looking for is a new sequence formula u_n (u standing for purfect) that doesn't have the 3 previously mentioned pitfalls.

The first thing we can do is to transform this recursive sequence into the expected ideal contiguous time based function. The original sequence was designed for a given rate R and FPS F: this means that while \Delta_n changes in practice every frame, the ideal function we are looking for is constant: \Delta=1/F.

So instead of starting from a_{n+1} = \mathrm{lerp}(a_n, \Beta, R\Delta_n), we will look for u_n starting from u_{n+1} = \mathrm{lerp}(u_n, \Beta, R\Delta) with u_0=a_0=\Alpha.

Since I'm lazy and incompetent, we are just going to ask WolframAlpha for help finding the solution to the recursive sequence. But to feed its input we need to simplify the terms a bit:

\begin{split} u_{n+1} &= \mathrm{lerp}(u_n, \Beta, R\Delta) \\ &= u_n(1-R\Delta) + \Beta R\Delta \\ &= u_nP + Q \end{split}

...with P=(1-R\Delta) and Q=\Beta R\Delta. We do that so we have a familiar ax+b linear form.

According to WolframAlpha this is equivalent to:

u_n = \Alpha P^n + \frac{Q(P^n-1)}{P-1}

This is great because we now have the formula according to n, our frame number. We can also express that discrete sequence into a contiguous function according to the time t:

f(t) = \Alpha P^{tF} + \frac{Q(P^{tF}-1)}{P-1}

Expanding our temporary P and Q placeholders with their values and unrolling, we get:

\begin{split} f(t) &= AP^{tF} + \frac{Q(P^{tF}-1)}{P-1} \\ &= A(1-R\Delta)^{tF} + \frac{\Beta R\Delta((1-R\Delta)^{tF}-1)}{(1-R\Delta)-1} \\ &= A(1-R\Delta)^{tF} - \Beta((1-R\Delta)^{tF}-1) \\ &= A(1-R\Delta)^{tF} + \Beta(1-(1-R\Delta)^{tF}) \\ &= \mathrm{lerp}(\Beta, \Alpha, (1-R\Delta)^{tF}) \\ &= \mathrm{lerp}(\Beta, \Alpha, (1-R/F)^{tF}) \\ f(t) &= \boxed{\mathrm{lerp}(\Alpha, \Beta, 1-(1-R/F)^{tF})} \end{split}

This function perfectly matches the initial \mathrm{lerp}() sequence in the hypothetical situation where the frame rate is honored. Basically, it's what the sequence a_{n+1} was meant to emulate at a given frame rate F.

Note

We swapped the first 2 terms of \mathrm{lerp}() at the last step because it makes more sense semantically to go from \Alpha to \Beta.

Let's again summarize what we have and what we want: we're into the game main loop and we want our running value to stick to that f(t) function. We have:

v=f(t): the value previously computed (t is the running duration so far, but we don't have it); in the original sequence this is known as a_n
\Delta_n: the delta time for the current frame

We are looking for a function \Eta(v,\Delta_n) which defines the position of a new point on the curve, only knowing v and \Delta_n. It's a "time agnostic" version of f(t).

Basically, it is defined as \Eta(v,\Delta_n)=f(t+\Delta_n), but since we don't have t it's not very helpful. That being said, while we don't have t, we do have f(t) (the previous value v).

Looking at the curve, we know the y-value of the previous point, and we know the difference between the new point and the previous point on the x-axis:

Previous and current point in time

If we want t (the total time elapsed at the previous point), we need the inverse function f^{-1}. Indeed, t = f^{-1}(f(t)): taking the inverse of a function gives back the input. We know f so we can inverse it, relying on WolframAlpha again (what a blessing this website is):

f^{-1}(x) = \frac{\ln{\frac{\Beta-x}{\Beta-\Alpha}}}{F \ln(1-R/F)}

Note

\ln stands for natural logarithm, sometimes also called \log. Careful though, on Desmos for example \log is in base 10, not base e (while its \exp is in base e for some reason).

This complex formula may feel a bit intimidating but we can now find \Eta only using its two parameters:

\begin{split} \Eta(v,\Delta_n) &= f(t + \Delta_n) \\ &= f(f^{-1}(f(t)) + \Delta_n) \\ &= f(f^{-1}(v) + \Delta_n) \\ &= f(\frac{\ln{\frac{\Beta-v}{\Beta-\Alpha}}}{F \ln(1-R/F)} + \Delta_n) \\ &= \mathrm{lerp}(\Alpha, \Beta, 1-(1-R/F)^{(\frac{\ln{\frac{\Beta-v}{\Beta-\Alpha}}}{F \ln(1-R/F)} + \Delta_n) \times F}) \\ &= \mathrm{lerp}(\Alpha, \Beta, 1-(1-R/F)^{\frac{\ln{\frac{\Beta-v}{\Beta-\Alpha}}}{\ln(1-R/F)}} (1-R/F)^{F\Delta_n}) \\ &= \mathrm{lerp}(\Alpha, \Beta, 1-\frac{\Beta-v}{\Beta-\Alpha} (1-R/F)^{F\Delta_n}) \\ &= (1-\frac{\Beta-v}{\Beta-\Alpha} (1-R/F)^{F\Delta_n})(\Beta-\Alpha) + A \\ &= (\Beta-\Alpha) - (\Beta-v) (1-R/F)^{F\Delta_n} + A \\ &= (v-\Beta)(1-R/F)^{F\Delta_n} + \Beta \\ &= \mathrm{lerp}(\Beta, v, (1-R/F)^{F\Delta_n}) \\ \Eta(v,\Delta_n) &= \mathrm{lerp}(v, \Beta, 1-(1-R/F)^{F\Delta_n}) \end{split}

Again we swapped the first 2 arguments of lerp at the last step at the cost of an additional subtraction: this is more readable because \Beta is our destination point.

An interesting property that is going to be helpful here is m^n = e^{n \ln{m}}. For my fellow programmers getting tensed here: pow(m, n) == exp(n * log(m)). Replacing the power with the exponential may not seem like an improvement at first, but it allows packing all the constant terms together:

\begin{split} \Eta(v,\Delta_n) &= \mathrm{lerp}(v, \Beta, 1-(1-R/F)^{F\Delta_n}) \\ &= \mathrm{lerp}(v, \Beta, 1-e^{F\ln(1-R/F)\Delta_n}) \end{split}

F\ln(1-R/F) can be pre-computed because it is constant: it's our rate conversion formula, which we can extract:

\begin{split} R' &= F\ln(1-R/F) \\ \Eta(v,\Delta_n) &= \mathrm{lerp}(v, \Beta, 1-e^{R'\Delta_n}) \end{split}

Rewriting this in a sequence notation, we get:

\begin{split} R' &= F\ln(1-R/F) \\ u_{n+1} &= \mathrm{lerp}(u_n, \Beta, 1-e^{R'\Delta_n}) \end{split}

We're going to make one last adjustment: R' is negative, which is not exactly intuitive to work with as a user (in case it is defined arbitrarily and not through the conversion formula), so we make a sign swap for convenience:

\boxed{\begin{split} R' &= -F\ln(1-R/F) \\ u_{n+1} &= \mathrm{lerp}(u_n, \Beta, 1-e^{-R'\Delta_n}) \end{split}}

The conversion formula is optional, it's only needed to port a previously broken code to the new formula. One interesting thing here is that R' is fairly close to R when R is small.

For example, a rate factor R=5 at 60 FPS gives us R' \approx 5.22. This means that if the rate factors weren't closely tuned, it is probably acceptable to go with R'=R and not bother with any conversion. Still, having that formula can be useful to update all the decay constants and check that everything still works as expected.

Also, notice how if the delta gets very large, -R'\Delta_n is going toward -\infty, e^{-R'\Delta_n} toward 0, 1-e^{-R'\Delta_n} toward 1, and so the interpolation is going to reach our final target \Beta without overshooting. This means the formula doesn't need any extra care with regard to the 3rd issue we pointed out earlier.

Looking at the previous curves but now with the new formula and an adjusted rate:

Comparison of the curves at different frame rates with the new formula

Conclusion

So there we have it, the perfect formula, frame rate agnostic ✅, deterministic ✅ and resilient to overshooting ✅. If you've quickly skimmed through the maths, here is what you need to know:

a = lerp(a, B, delta * RATE)

Should be changed to:

a = lerp(a, B, 1.0 - exp(-delta * RATE2))

With the precomputed RATE2 = -FPS * log(1 - RATE/FPS) (where log is the natural logarithm), or simply using RATE2 = RATE as a rough equivalent.

Also, any existing overshooting clamping can safely be dropped.

Now please adjust your game to make the world a better and safer place for everyone ♥

Going further

As suggested on HN:

For numerical stability it makes sense to use -expm1(x) instead of 1-exp(x)
API wise, proposing a time constant T instead of the rate (where T=1/rate) might be more intuitive
For performance reasons, the exponential could be expanded manually to x+x²/2 for small value of x

Hacking window titles to help OBS

Tue, 06 Jun 2023 09:27:10 -0000

This write-up is meant to present the rationale and technical details behind a tiny project I wrote the other day, WTH, or WindowTitleHack, which is meant to force a constant window name for apps that keep changing it (I'm looking specifically at Firefox and Krita, but there are probably many others).

Why tho?

I've been streaming on Twitch from Linux (X11) with a barebone OBS Studio setup for a while now, and while most of the experience has been relatively smooth, one particularly striking frustration has been dealing with windows detection.

If we don't want to capture the whole desktop for privacy reasons or simply to have control over the scene layout depending on the currently focused app, we need to rely on the Window Capture (XComposite) source. This works mostly fine, and it is actually able to track windows even when their title bar is renamed. But obviously, upon restart it can't find them again because both the window titles and the window IDs changed, meaning we have to redo our setup by reselecting the windows again.

It would have been acceptable if that was the only issue I had, but one of the more advanced feature I'm extensively using is the Advanced Scene Switcher (the builtin one, available through the Tools menu). This tool is a basic window title pattern matching system that allows automatic scene switches depending on the current window. Note that it does seem to support regex, which could help with the problem, but there is no guarantee that the app would leave a recognizable matchable pattern in its title. Also, if we want multiple Firefox windows but only match one in particular, the regex wouldn't help.

OBS native automatic scene switcher

Hacking Windows

One unreliable hack would be to spam xdotool commands to correct the window title. This could be a resource hog, and it would create quite a bunch of races. One slight improvement over this would be to use xprop -spy, but that wouldn't address the race conditions (since we would adjust the title after it's been already changed).

So how do we deal with that properly? Well, on X11 with the reference library (Xlib) there are actually various (actually a lot of) ways of changing the title bar. It took me a while to identify which call(s) to target, but ended up with the following call graph, where each function is actually exposed publicly:

X11 XChangeProperty call tree

From this we can easily see that we only need to hook the deepest function XChangeProperty, and check if the property is XA_WM_NAME (or its "modern" sibling, _NET_WM_NAME).

How do we do that? With the help of the LD_PRELOAD environment variable and a dynamic library that implements a custom XChangeProperty.

First, we grab the original function:

#include <dlfcn.h>

/* A type matching the prototype of the target function */
typedef int (*XChangeProperty_func_type)(
    Display *display,
    Window w,
    Atom property,
    Atom type,
    int format,
    int mode,
    const unsigned char *data,
    int nelements
);

/* [...] */

XChangeProperty_func_type XChangeProperty_orig = dlsym(RTLD_NEXT, "XChangeProperty");

We also need to craft a custom _NET_WM_NAME atom:

_NET_WM_NAME = XInternAtom(display, "_NET_WM_NAME", 0);

With this we are now able to intercept all the WM_NAME events and override them with our own:

if (property == XA_WM_NAME || property == _NET_WM_NAME) {
    data = (const unsigned char *)new_title;
    nelements = (int)strlen(new_title);
}
return XChangeProperty_orig(display, w, property, type, format, mode, data, nelements);

We wrap all of this into our own redefinition of XChangeProperty and… that's pretty much it.

Now due to a long history of development, Xlib has been "deprecated" and superseded by libxcb. Both are widely used, but fortunately the APIs are more or less similar. The function to hook is xcb_change_property, and defining _NET_WM_NAME is slightly more cumbered but not exactly challenging:

const xcb_intern_atom_cookie_t cookie = xcb_intern_atom(conn, 0, strlen("_NET_WM_NAME"), "_NET_WM_NAME");
xcb_intern_atom_reply_t *reply = xcb_intern_atom_reply(conn, cookie, NULL);
if (reply)
    _NET_WM_NAME = reply->atom;
free(reply);

Aside from that, the code is pretty much the same.

Configuration

To pass down the custom title to override, I've been relying on an environment variable WTH_TITLE. From a user point of view, it looks like this:

LD_PRELOAD="builddir/libwth.so" WTH_TITLE="Krita4ever" krita

We could probably improve the usability by creating a wrapping tool (so that we could have something such as ./wth --title=Krita4ever krita). Unfortunately I wasn't yet able to make a self-referencing executable accepted by LD_PRELOAD, so for now the manual LD_PRELOAD and WTH_TITLE environment will do just fine.

Thread safety

To avoid a bunch of redundant function round-trips we need to globally cache a few things: the new title (to avoid fetching it in the environment all the time), the original functions (to save the dlsym call), and _NET_WM_NAME.

Those are loaded lazily at the first function call, but we have no guarantee with regards to concurrent calls on that hooked function so we must create our own lock. I initially though about using pthread_once but unfortunately the initialization callback mechanism doesn't allow any custom argument. Again, this is merely a slight annoyance since we can implement our own in a few lines of code:

/* The "once" API is similar to pthread_once but allows a custom function argument */
struct wth_once {
    pthread_mutex_t lock;
    int initialized;
};

#define WTH_ONCE_INITIALIZER {.lock=PTHREAD_MUTEX_INITIALIZER}

typedef void (*init_func_type)(void *user_arg);

void wth_init_once(struct wth_once *once, init_func_type init_func, void *user_arg)
{
    pthread_mutex_lock(&once->lock);
    if (!once->initialized) {
        init_func(user_arg);
        once->initialized = 1;
    }
    pthread_mutex_unlock(&once->lock);
}

Which we use like this:

static struct wth_once once = WTH_ONCE_INITIALIZER;

static void init_once(void *user_arg)
{
    Display *display = user_arg;
    /* [...] */
}

/* [...] */

wth_init_once(&once, init_once, display);

The End?

I've been delaying doing this project for weeks because it felt complex at first glance, but it actually just took me a few hours. Probably the same amount of time it took me to write this article. While the project is admittedly really small, it still feel like a nice accomplishment. I hope it's useful to other people.

Now, the Wayland support is probably the most obvious improvement the project can receive, but I don't have such a setup locally to test yet, so this is postponed for an undetermined amount of time.

The code is released with a permissive license (MIT); if you want to contribute you can open a pull request but getting in touch with me first is appreciated to avoid unnecessary and overlapping efforts.

Improving color quantization heuristics

Sat, 31 Dec 2022 12:00:43 -0000

In 2015, I wrote an article about how the palette color quantization was improved in FFmpeg in order to make nice animated GIF files. For some reason, to this day this is one of my most popular article.

As time passed, my experience with colors grew and I ended up being quite ashamed and frustrated with the state of these filters. A lot of the code was naive (when not terribly wrong), despite the apparent good results.

One of the major change I wanted to do was to evaluate the color distances using a perceptually uniform color space, instead of using a naive euclidean distance of RGB triplets.

As usual it felt like a week-end long project; after all, all I have to do is change the distance function to work in a different space, right? Well, if you're following my blog you might have noticed I've add numerous adventures that stacked up on each others:

I had to work out the colorspace with integer arithmetic first
...which forced me to look into integer division more deeply
...which confronted me to all sort of undefined behaviours in the process

And when I finally reached the point where I could make the switch to OkLab (the perceptual colorspace), a few experiments showed that the flavor of the core algorithm I was using might contain some fundamental flaws, or at least was not implementing optimal heuristics. So here we go again, quickly enough I find myself starting a new research study in the pursuit of understanding how to put pixels on the screen. This write-up is the story of yet another self-inflicted struggle.

Palette quantization

But what is palette quantization? It essentially refers to the process of reducing the number of available colors of an image down to a smaller subset. In sRGB, an image can have up to 16.7 million colors. In practice though it's generally much less, to the surprise of no one. Still, it's not rare to have a few hundreds of thousands different colors in a single picture. Our goal is to reduce that to something like 256 colors that represent them best, and use these colors to create a new picture.

Why you may ask? There are multiple reasons, here are some:

Improve size compression (this is a lossy operation of course, and using dithering on top might actually defeat the original purpose)
Some codecs might not support anything else than limited palettes (GIF or subtitles codecs are examples)
Various artistic purposes

Following is an example of a picture quantized at different levels:

Original (26125 colors)	Quantized to 8bpp (256 colors)	Quantized to 2bpp (4 colors)

This color quantization process can be roughly summarized in a 4-steps based process:

Sample the input image: we build a histogram of all the colors in the picture (basically a simple statistical analysis)
Design a colormap: we build the palette through various means using the histograms
Create a pixel mapping which associates a color (one that can be found in the input image) with another (one that can be found in the newly created palette)
Image quantizing: we use the color mapping to build our new image. This step may also involve some dithering.

The study here will focus on step 2 (which itself relies on step 1).

Colormap design algorithms

A palette is simply a set of colors. It can be represented in various ways, for example here in 2D and 3D:

A 256 color palette represented in 2D and 3D

To generate such a palette, all sort of algorithms exists. They are usually classified into 2 large categories:

Dividing/splitting algorithms (such as Median-Cut and its various flavors)
Clustering algorithms (such as K-means, maximin distance, (E)LBG or pairwise clustering)

The former are faster but non-optimal while the latter are slower but better. The problem is NP-complete, meaning it's possible to find the optimal solution but it can be extremely costly. On the other hand, it's possible to find "local optimums" at minimal cost.

Since I'm working within FFmpeg, speed has always been a priority. This was the reason that motivated me to initially implement the Median-Cut over a more expensive algorithm.

The rough picture of the algorithm is relatively easy to grasp. Assuming we want a palette of K colors:

A set S of all the colors in the input picture is constructed, along with a respective set W of the weight of each color (how much they appear)
Since the colors are expressed as RGB triplets, they can be encapsulated in one big cuboid, or box
The box is cut in two along one of the axis (R, G or B) on the median (hence the name of the algorithm)
If we don't have a total K boxes yet, pick one of them and go back to previous step
All the colors in each of the K boxes are then averaged to form the color palette entries

Here is how the process looks like visually:

Median-Cut algorithm targeting 16 boxes

You may have spotted in this video that the colors are not expressed in RGB but in Lab: this is because instead of representing the colors in a traditional RGB colorspace, we are instead using the OkLab colorspace which has the property of being perceptually uniform. It doesn't really change the Median Cut algorithm, but it definitely has an impact on the resulting palette.

One striking limitation of this algorithm is that we are working exclusively with cuboids: the cuts are limited to an axis, we are not cutting along an arbitrary plane or a more complex shape. Think of it like working with voxels instead of more free-form geometries. The main benefit is that the algorithm is pretty simple to implement.

Now the description provided earlier conveniently avoided describing two important aspects happening in step 3 and 4:

How do we choose the next box to split?
How do we choose along which axis of the box we make the cut?

I pondered about that for a quite a long time.

An overview of the possible heuristics

In bulk, some of the heuristics I started thinking of:

Should we take the box that has the longest axis across all boxes?
Should we take the box that has the largest volume?
Should we take the box that has the biggest Mean Squared Error when compared to its average color?
Should we take the box that has the axis with the biggest MSE?
Assuming we choose to go with the MSE, should it be normalized across all boxes?
Should we even account for the weight of each color or consider them equal?
what about the axis? Is it better to pick the longest or the one with the higher MSE?

I tried to formalize these questions mathematically to the best of my limited abilities. So let's start by saying that all the colors c of a given box are stored in a N×M 2D-array following the matrix notation:

L₁	L₂	L₃	…	Lₘ
a₁	a₂	a₃	…	aₘ
b₁	b₂	b₃	…	bₘ

N is the number of components (3 in our case, whether it's RGB or Lab), and M the number of colors in that box. You can visualize this as a list of vectors as well, where c_{i,j} is the color at row i and column j.

With that in mind we can sketch the following diagram representing the tree of heuristic possibilities to implement:

Tree of potential heuristics for the Median-Cut algorithm

Mathematicians are going to kill me for doodling random notes all over this perfectly understandable symbols gibberish, but I believe this is required for the human beings reading this article.

In summary, we end up with a total of 24 combinations to try out:

2 axis selection heuristics:
- Cut the axis with the maximum error squared
- Cut the axis with the maximum length
3 operators:
- Maximum measurement out of all the channels
- Product of the measurements of all the channels
- Sum of the measurements of all the channels
4 measurements:
- Error squared, honoring weights
- Error squared, not honoring weights
- Error squared, honoring weights, normalized
- Length of the axis

If we start to intuitively think about which ones are likely going to perform the best, we quickly realize that we haven't actually formalized what we are trying to achieve. Such a rookie mistake. Clarifying this will help us getting a better feeling about the likely outcome.

I chose to target an output that minimizes the MSE against the reference image, in a perceptual way. Said differently, trying to make the perceptual distance between an input and output color pixel as minimal as possible. This is an arbitrary and debatable target, but it's relatively simple and objective to evaluate if we have faith in the selected perceptual model. Another appropriate metric could have been to find the ideal palette through another algorithm, and compare against that instead. Doing that unfortunately implied that I would trust that other algorithm, its implementation, and that I have enough computing power.

So to summarize, we want to minimize the MSE between the input and output, evaluated in the OkLab color space. This can be expressed with the following formula:

\min_{P : |P| = K} \displaystyle\sum_{C \in P} \sum_{c \in C} w_c||c-\mu_C||^2

Where:

P is a partition (which we constrain to a box in our implementation)
C the set of colors in the partition P
w the weight of a color
c a single color
\mu the average color of the set C

Special thanks to criver for helping me a ton on the math area, this last formula is from them.

Looking at the formula, we can see how similar it is to certain branches of the heuristics tree, so we can start getting an intuition about the result of the experiment.

Experiment language

Short deviation from the main topic (feel free to skip to the next section): working in C within FFmpeg quickly became a hurdle more than anything. Aside from the lack of flexibility, the implicit casts destroying the precision deceitfully, and the undefined behaviours, all kind of C quirks went in the way several times, which made me question my sanity. This one typically severely messed me up while trying to average the colors:

#include <stdio.h>
#include <stdint.h>

int main (void)
{
    const int32_t x = -30;
    const uint32_t y = 10;

    const uint32_t a = 30;
    const int32_t b = -10;

    printf("%d×%u=%d\n", x, y, x * y);
    printf("%u×%d=%d\n", a, b, a * b);
    printf("%d/%u=%d\n", x, y, x / y);
    printf("%u/%d=%d\n", a, b, a / b);
    return 0;
}

% cc -Wall -Wextra -fsanitize=undefined test.c -o test && ./test
-30×10=-300
30×-10=-300
-30/10=429496726
30/-10=0

Anyway, I know this is obvious but if you aren't already doing that I suggest you build your experiments in another language, Python or whatever, and rewrite them in C later when you figured out your expected output.

Re-implementing what I needed in Python didn't take me long. It was, and still is obviously much slower at runtime, but that's fine. There is a lot of room for speed improvement, typically by relying on numpy (which I didn't bother with).

Experiment results

I created a research repository for the occasion. The code to reproduce and the results can be found in the color quantization README.

In short, based on the results, we can conclude that:

Overall, the box that has the axis with the largest non-normalized weighted sum of squared error is the best candidate in the box selection algorithm
Overall, cutting the axis with the largest weighted sum of squared error is the best axis cut selection algorithm

To my surprise, normalizing the weights per box is not a good idea. I initially observed that by trial and error, which was actually one of the main motivator for this research. I initially thought normalizing each box was necessary in order to compare them against each others (such that they are compared on a common ground). My loose explanation of the phenomenon was that not normalizing causes a bias towards boxes with many colors, but that's actually exactly what we want. I believe it can also be explained by our evaluation function: we want to minimize the error across the whole set of colors, so small partitions (in color counts) must not be made stronger. At least not in the context of the target we chose.

It's also interesting to see how the max() seems to perform better than the sum() of the variance of each component most of the time. Admittedly my picture samples set is not that big, which may imply that more experiments to confirm that tendency are required.

In retrospective, this might have been quickly predictable to someone with a mathematical background. But since I don't have that, nor do I trust my abstract thinking much, I'm kind of forced to try things out often. This is likely one of the many instances where I spent way too much energy on something obvious from the beginning, but I have the hope it will actually provide some useful information for other lost souls out there.

Known limitations

There are two main limitations I want to discuss before closing this article. The first one is related to minimizing the MSE even more.

K-means refinement

We know the Median-Cut actually provides a rough estimate of the optimal palette. One thing we could do is use it as a first step before refinement, for example by running a few K-means iterations as post-processing (how much refinement/iterations could be a user control). The general idea of K-means is to progressively move each colors individually to a more appropriate box, that is a box for which the color distance to the average color of that box is smaller. I started implementing that in a very naive way, so it's extremely slow, but that's something to investigate further because it definitely improves the results.

Most of the academic literature seems to suggest the use of the K-means clustering, but all of them require some startup step. Some come up with various heuristics, some use PCA, but I've yet to see one that rely on Median-Cut as first pass; maybe that's not such a good idea, but who knows.

Bias toward perceived lightness

Another more annoying problem for which I have no solution for is with regards to the human perception being much more sensitive to light changes than hue. If you look at the first demo with the parrot, you may have observed the boxes are kind of thin. This is because the a and b components (respectively how green/red and blue/yellow the color is) have a much smaller amplitude compared to the L (perceived lightness).

Side by side comparison of the spread of colors between a stretched and normalized view

You may rightfully question whether this is a problem or not. In practice, this means that when K is low (let's say smaller than 8 or even 16), cuts along L will almost always be preferred, causing the picture to be heavily desaturated. This is because it tries to preserve the most significant attribute in human perception: the lightness.

That particular picture is actually a pathological study case:

4 colors	8 colors	12 colors	16 colors

We can see the hue timidly appearing around K=16 (specifically it starts being more strongly noticeable starting the cut K=13).

Conclusion

For now, I'm mostly done with this "week-end long project" into which I actually poured 2 or 3 months of lifetime. The FFmpeg patchset will likely be upstreamed soon so everyone should hopefully be able to benefit from it in the next release. It will also come with additional dithering methods, which implementation actually was a relaxing distraction from all this hardship. There are still many ways of improving this work, but it's the end of the line for me, so I'll trust the Internet with it.