[css-color-4] Premultiplication in cylindrical spaces and mixing

[raphlinus]

The current spec describes premultiplication as not multiplying the hue component in cylindrical spaces. I think I understand the motivation of that for doing interpolation and gradients, but it does not seem to be the correct logic for doing mixing and compositing. I'm wondering what implementations should do, and specifically whether there need to be two forms of premultiplication, one for lerp, one for mixing.

The basic rule for x over y in premultiplied spaces is x + (1 - x.alpha) * y. But this breaks down for cylindrical spaces premultiplied as per the spec, as the sum of the weights on the hue components exceeds 1. (It's not a problem for lerping, as the sum of weights is always 1)

Another way of phrasing this is that the premultiplication method in the Color Level 4 draft mismatches the definition of premultiplication for color-mix in Color Level 5, which does not make an exception for hue.

I can think of several ways of dealing with this:

1. Simply forbid compositing in cylindrical spaces.
2. When compositing in a cylindrical space is requested, actually do it in the associated rectangular space, for example oklab when oklch is requested.
3. Define the over operation as (x.hue * x.alpha + y.hue * y.alpha * (1 - x.alpha)) / (x.alpha + y.alpha * (1 - x.alpha)) for the hue component.
4. Have PremulColor and LerpPremulColor as separate types, with the former premultiplying all components, and the latter holding out hue.

One reason that choices (1) and (2) are on this list is that I'm not sure how useful it is to do compositing in a cylindrical space. I'm happy to be pointed to evidence on this.

The difference in behavior seems subtle, and it's not obvious to me that the CSS specified behavior is clearly more better or more correct than the simpler, compositing-friendly behavior. I searched for discussion where this was decided and couldn't find it. I can generate color ramps to illustrate the difference if that would be useful.

For context, this came up when we were starting to contemplate a color_mix method in our new Rust color crate. My original hope is that the PremulColor type we defined for CSS Color Level 4 interpolation would also efficiently support mixing/compositing, but that is not looking hopeful at the moment.

[facelessuser]

It should be noted if you were to treat the cylindrical space in the rectangular space and apply the premultiplication, you'd still have the same hue, that's why it doesn't make sense to premultiply the hues. The hue doesn't change, just the colorfulness when it is mixed.

>>> color = Color('oklch', [0.5, 0.2, 85], 0.5)
>>> a, b = color.convert('oklab').get(['a', 'b'])
>>> a *= color.alpha()
>>> b *= color.alpha()
>>> Color('oklab', [color['lightness'] * color.alpha(), a, b]).convert('oklch')
color(--oklch 0.25 0.1 85 / 1)

I don't think compositing should really be done in a cylindrical space, but interpolating between two cylindrical colors, which is what color-mix is doing seems fine.

Another way of phrasing this is that the premultiplication method in the Color Level 4 draft mismatches the definition of premultiplication for color-mix in Color Level 5, which does not make an exception for hue.

If I had to guess, I think omitting the statement about not premultiplying hue is just an accident, not an explicit intention.

[facelessuser]

I do realize that, in this scenario, we are referring to this alpha blending as compositing. When I say compositing probably shouldn't be done in a cylindrical space, I mean the browser itself should not apply compositing in this way when rendering colors or overlaying images, etc. due to how hue interpolations work, but there is nothing wrong with interpolating in a cylindrical space if that is what you want to do.

[raphlinus]

Oops, I realize I made a mistake in formulating this question, as I believed that color-mix was capable of representing the Porter-Duff over operator, and, now, looking at it more closely, it seems that it is only capable of representing lerp but with an additional alpha scaling step.

So I think there are two separate concerns. One is the mismatch as pointed out, which I agree with @facelessuser is most likely a spec drafting issue. The second is whether the color representation is suitable for compositing, which is the main question I'm trying to raise right now. We do already have an interpolate method which does the color-mix functionality except for the scaling by 1 / (p1 + p2), and we have a separate mul_alpha which can perform that additional step.

To me, interpolation and compositing are closely related. In particular, I believe compositing color-a / alpha over opaque color-b should match color-mix(color-a alpha, color-b). In rectangular spaces, that is not controversial, but if we allow compositing in cylindrical spaces, it is.

So strike "another way of phrasing this," and the last paragraph should read: For context, this came up when we were starting to contemplate an over method in our new Rust color crate. My original hope is that the PremulColor type we defined for CSS Color Level 4 interpolation would also efficiently support compositing, but that is not looking hopeful at the moment.

Apologies for the confusion.

[facelessuser]

To me, interpolation and compositing are closely related

Sure, interpolation is used in compositing.

I think when defining a general-purpose interpolation function, like what is done in CSS, you need to define clear, sane rules, and the rules for cylindrical spaces seem perfectly reasonable.

Due to how hue interpolation behaves, it doesn't lend itself well to compositing images and layers, but there are plenty of reasons to desire a hue interpolation, such as when doing gradients. I'm not sure there is a reason to specifically forbid alpha blending/compositing in cylindrical spaces as the current rules are quite reasonable. Would I choose to specifically use a cylindrical space to do compositing in an image? Probably not.

[facelessuser]

It probably should be noted that there is a separate discussion talking about exposing blending (possibly) and alpha compositing of colors: #8431. I don't think that discussion goes into specific steps in compositing colors and is more discussing an interface to do so. I think if such an interface is approved, it probably shouldn't allow compositing to be applied in a cylindrical color space.

color-mix is not really a compositing interface, but more an interface for getting a color anywhere between two colors, and yes, cylindrical spaces behave very differently when interpolation is done within them simply due to how they separate the color characteristics. Even if using a rectangular color space, mixing two transparent colors at 50% will not yield the same results as applying Porter Duff compositing on those two colors. The logic is different, even if interpolation is used in Porter Duff compositing. This is why I don't think color-mix should be viewed as a compositing interface, not directly.

I think this is a pretty good read on alpha compositing: https://ciechanow.ski/alpha-compositing/.

[svgeesus]

One reason that choices (1) and (2) are on this list is that I'm not sure how useful it is to do compositing in a cylindrical space. I'm happy to be pointed to evidence on this.

There are three useful spaces for compositing:

1. CIE XYZ (or equivalently, any unbounded linear-light RGB space) to get the right result
2. gamma-encoded sRGB, to get the wrong but web-compatible result
3. gamma-encoded device RGB (because you already have your pixels in that space and want to use the hardware), which also gives a similar wrong result to 2.

[svgeesus]

If I had to guess, I think omitting the statement about not premultiplying hue is just an accident, not an explicit intention.

You guessed correctly

[raphlinus]

There seem to be a few possible issues here. One is whether the over operator is defined over arbitrary color spaces or a select few, and if the former, how it should be defined for cylindrical color spaces. From what @svgeesus has said, the answer is the latter, so that may be the end of the story for this issue. (We may choose to define over in the color crate for more spaces anyway, but that needn't concern us here)

I'm still not convinced that holding out hue from premultiplication is not needless complexity that makes things worse, though it's also possible I'm missing something. In particular, I don't follow the argument above. To me, the interpretation of premultiplied color components has an implicit division by alpha. The premultiplied component vector for oklch(0.5 0.2 85 / 50%) is [0.25, 0.1, 42.5, 0.5], which thus has a hue angle of 85 degrees, not 42.5.

Holding out premultiplication of hue, the result of a 50% lerp of oklch(0.5 0.1 120 / 50%) and oklch(0.5 0.1 240) is oklch(0.5 0.1 180 / 75%). If hue were premultiplied, the result would be oklch(0.5 0.1 200 / 75%), essentially giving more weight to the more opaque color. That actually seems better to me, in that I believe it's a more perceptually uniform ramp. As I say, I'm happy to prepare visual samples to argue for this position.

gamma-encoded sRGB, to get the wrong but web-compatible result

I don't agree with the word "wrong" here, as I believe there is a case to be made that compositing in more perceptually uniform spaces is a useful design tool. In addition, compositing in device RGB is a much closer approximation to layering paint in the Kubelka-Munk model than doing it in a linear space. That may be an additional factor in the staying power of compositing in device color spaces, not just laziness of implementors. But perhaps that's a discussion for another day.

[facelessuser]

One is whether the over operator is defined over arbitrary color spaces or a select few, and if the former, how it should be defined for cylindrical color spaces.

CSS does not currently define composition except in https://www.w3.org/TR/compositing-1/ and only in the context of RGB spaces. It has not been defined or applied outside of that. So over is currently only defined in this context. This does not apply to general interpolation.

I'm still not convinced that holding out hue from premultiplication is not needless complexity that makes things worse, though it's also possible I'm missing something. In particular, I don't follow the argument above.

Premultiplication is used essentially to weight how much each color influences the interpolation. This works well in rectangular coordinate systems. The actual hue of a color does not change in this process (speaking relative to the color space). That's why when you convert said cylindrical color to rectangular coordinates and apply premultiplication, then convert back, hue is unchanged. You should not premultiply hue because if you do, you are now interpolating something very different.

If you cherry pick certain cases, it could look okay if we apply premultiplication to hues. For instance, you gave a specific case in OkLCh which does seem to look okay:

But if we use some other colors, it quickly falls apart:

[tabatkins]

Premultiplication is used essentially to weight how much each color influences the interpolation. This works well in rectangular coordinate systems.

Right, premultiplication is a weighting method designed to model compositing. It is only a meaningful operation when applied to vector coordianates, that have a meaningful notion of "scaling". Hue, in a cylindrical space, is not such a coordinate; there is no privileged zero point. I think the technical term is that the hue is an affine coordinate, which doesn't allow addition or scaling. You can only add/scale differences between hues; the set of hue differences forms a vector space. This is why you can interpolate hue (you're scaling a hue difference, and adding it to a hue, which is also allowed).

So the concept of premultiplying a cylindrical space simply isn't coherent. You have to do premult in a rectangular space, and as Chris says, if you're premultiplying in order to composite, there's only a handful of color spaces intended for that.

[raphlinus]

I'm still needing to be convinced, both the example and the argument from math.

The huge discrepancy in the second example is caused by hue fixup. That is most definitely a complication (and I hadn't thought about that in my initial analysis), but I don't think it's a showstopper. Hue fixup can be defined in terms of un-premultiplying, and then in turn can be evaluated efficiently without division with some algebraic manipulation (basically it's h1 * a2 - h2 * a1 > 180 * a1 * a2 being equivalent to h1 / a1 - h2 / a2 > 180).

And I don't quite follow the argument about affine spaces, and don't think compositing is restricted to coordinates that have a meaningful concept of scaling. Ultimately, the final output is a weighted sum of values, and if the weights sum to 1 then I don't see a fundamental mathematical problem.

Of course, the hue fixup is a real source of trouble, and the fact it can create discontinuities is a good reason to not consider it form of compositing. But there are ways to get discontinuities from blend modes also.

[facelessuser]

I'm still needing to be convinced, both the example and the argument from math.

Fundementally, I don't think polar interpolation models the kind of compositing that premultiplication was designed to help mimic in rectangular spaces. Premultiplication helps model how more or less light makes it to the eye in transparent cases. The entire concept doesn't really translate to hue shifts in polar spaces. More or less transparency doesn't control the bending of light.

It seems you really want an analogy to how compositing works in the polar space, which is fine, but you'll likely have to make a strong argument on why this is useful and is needed. I'm not convinced it is needed or is worth the added complexity, but I'm also not who you need to convince 🙂.

I think I'll bow out of the conversation as I think I've probably said enough.

[svgeesus]

I agree with @facelessuser and @tabatkins that there are no articulated use cases for doing compositing in a polar space, beyond "it makes our code path simpler". The option in the first post

When compositing in a cylindrical space is requested, actually do it in the associated rectangular space, for example oklab when oklch is requested.

seems like an ideal way forward, if you are coding up a generalized multi-color-space compositing codebase. I might be tempted to throw in a warning when that happens, too.

[tabatkins]

And I don't quite follow the argument about affine spaces, and don't think compositing is restricted to coordinates that have a meaningful concept of scaling. Ultimately, the final output is a weighted sum of values, and if the weights sum to 1 then I don't see a fundamental mathematical problem.

You can probably composite in any space in theory (I'm not sure, but I'll take your word for it), but you can't do premultiplication in color spaces with affine coordinates (this I'm 100% sure of). Pre-multiplication is fundamentally a scaling operation, and you can't scale affine coordinates; there is no value of hue angle that means "0deg red, but half as much".

As a very specific example, there is no value of "pre-multiplied" hue angle that you can use for transparent that will have the same effect as doing pre-multiplied interpolation in RGB (that is, every color in the transition has the other endpoint's hue). This is why we have the notion of "missing" hue in the Color 4/5 interpolation rules, so that we can at least handle the common/trivial cases (a literal transparent, etc) as expected, even if it doesn't scale to "nearby" (mostly-transparent) colors.

[raphlinus]

If we're talking about mathematical principles, then I think we'll agree to disagree. That's pretty much academic, as I think we've pretty much converged on not providing compositing methods in cylindrical spaces (partly because of this complexity, partly because it inevitably creates discontinuities). Here's my reasoning though.

There are two interpretations of premultiplied alpha. One is as a filter passing a certain fraction (1 - alpha) of light of the underlying layer, plus adding some additional light. The other is as a convenient mathematical representation of a weighted point in some space, where the actual point can be recovered by dividing by alpha. It's valuable because many different flavors of weighted sum can be computed using only multiplication.

While the former interpretation is appealing, and is certainly meaningful in linear color spaces, I don't think it's particularly valid more generally. Even in nonlinear spaces, it has no real physical interpretation, though it can be considered a (fairly poor) approximation of the linear interpretation. More to the point, as long as what you're computing is a weighted sum of some sort (as is the case both for lerp and over), the two interpretations are the same. And indeed, the result is mathematically invariant to any affine transformation of the coordinates.

In the weighted sum interpretation, I consider what CSS specifies for hue premultiplication to be subtly wrong, as the hue component is not weighted by the alpha of the associated color, while the others are. That slightly degrades the perceptual linearity of the result. But I think it's fair to characterize this as subtle, as it only affects lerping cases where the endpoints have different alpha.

Of course, the concept of a weighted sum of hue is dodgy in any case; the average of orange and magenta can be either red or cyan depending on which way round you take. But there's no denying it's useful.

I'll also argue that the weighted sum interpretation is potentially useful for compositing in color spaces such as Oklab. As I mentioned above, compositing in device RGB is a better approximation than linear compositing to paint layering (Kubelka-Munk). The practical difference is that linear blending of low-key and high-key wipes out the shadow details in the former, while blending in a more perceptually uniform space does not. And Oklab would have better hue linearity. So I think we will implement compositing in arbitrary rectangular spaces. Whether it should be in CSS is a different question, as of course it has costs; anything not supported by system compositors, especially so.

So I think what CSS has is good enough and a bit messy, which probably as it should be for a web standard. Premultiplying hue is cleaner in some ways but also messier in others, particularly the hue fixup logic. So I'm fine with the logic staying the same as it is and the color 5 spec language changed to match color 4.

[facelessuser]

So I think we will implement compositing in arbitrary rectangular spaces.

It probably should be noted that while general alpha compositing will probably work okay in most rectangular spaces, some may not be fully compatible with various blend modes. Blend modes were generally designed with RGB in mind and expect channels to be between 0 - 1. Some blend modes would particularly be more sensitive to these assumptions than others. Spaces that don't fit into this box, or do not scale well into this box, may exhibit some issues.