[Feature] Calibrate HSV measurements

[laurensvalk]

Is your feature request related to a problem? Please describe.
The hues reported by the SPIKE Prime Color sensor, as well as the ones we calculate for the BOOST ColorDistanceSensor aren't really great in the 0--120 degrees region.

For example, yellow is reported at a hue of about 30, while in reality it should be about 50--60.

This is currently not a problem since the matching value is set to 30 in the color map. However, it would be a problem for displaying hsv colors on the status light, because then it would look orange.

Describe the solution you'd like
Pass the RGB and/or HSV values through a mapping that corrects this. This way, measuring a color (e.g. yellow) and showing it on the status light should produce the correct result.

In the case of SPIKE Prime, it means we should no longer use its HSV mode, but compute HSV ourselves from the RGB values. This seems to be what the sensor is doing anyway, so we don't seem to lose any information with this approach.

First steps towards solution
I've been starting to collect some measurents of scanning LEGO bricks. Each object is a stack of at least 3 bricks of the same color, and I was scanning the flat side of the stack. I placed them at such a distance from the sensor such that the V value was constant across all measurements.

This table reports the values we currently read, compared to what they should be according to the Bricklink color guide.

	Bricklink Hue	SPIKE Hue	SPIKE RGB (V=70)	BOOST Hue	BOOST RGB (V=30)	BOOST RGB V=50	NXT Hue
Red	358	350	360, 116, 155	358	132, 15, 21	227, 29, 38	358
Orange	27	3	359, 158, 146	1	132, 22, 16	221, 35, 32	1
Yellow	53	29	364, 276, 188	29	138, 78, 23	220, 133, 39	30
Green	147	155	180, 360, 288	132	43, 134, 62	?, ?, ?	120
Spike Blue	199	197	150, 300, 356	190	35, 116, 136	55, 189, 220	190
Blue	209	215	117, 216, 361	219	22, 60, 137	35, 103, 220	220
Spike Magenta	342	339	362, 153, 228	347	135, 23, 51	222, 42, 83	345

Notes

• The RGB and V values are not currently comparable between SPIKE/BOOST. In the SPIKE case, this is currently the V value obtained from the sensor, which is scaled differently. SPIKE RGB saturates at about 1023, while BOOST RGB peaks at about 440 or 500.

[laurensvalk]

I'd like to tag @tomalphin who looks to be quite the expert on LEGO colors.

The colors in your diagrams seem to be slightly different than the Bricklink colors. If not Bricklink, which reference would you recommend as the standard to pick the right hue for LEGO colors?

EDIT: Ah, I see this comment on your blog:

I extracted colors from a photograph that I took, which was then calibrated to neutral gray using a gray card. At the time, I did not have a more advanced color profiling system, but I now own an x-rite Color Checker Passport, and at some point in the future I plan to photograph the same selection of bricks using multiple cameras, all corrected using the gray card and custom profiles, and then finally averaging the RGB / HSL values.

In case you are interested to be involved, it would be awesome to have accurate reference values for the primary/secondary LEGO colors. It would be pretty cool to calibrate the sensor as a super accurate color scanner which goes at just $15 on the official shop.

[laurensvalk]

A patch for this is applied here. More accurate HSV representation would be nice, but it is less critical now that the user can do their own calibration.

[laurensvalk]

A patch for this is applied here. More accurate HSV representation would be nice, but it is less critical now that the user can do their own calibration.

This is apparently not good enough, so let's reopen it.

Related: pybricks/pybricks-micropython#93. @Novakasa

[Novakasa]

I could take a look at this, at least I can extract some raw RGB data and look at the deviation for some LEGO colors.

Should we then incorporate a mapping already in raw_to_rgb(...) for the specific sensors? I'm guessing that would enable us to catch any sensor dependence at the root. But, depending on the mapping required it might be difficult to do it in RGB space.

[Novakasa]

In theory, there could be a distance-dependent mapping to attempt to compensate for a given distance of the surface. This probably should be provided by the user.

[laurensvalk]

Instead of fixing/equalizing the RGB readings, we could perhaps have some mapping, e. g. like the inverse of https://github.com/pybricks/pybricks-micropython/blob/f69bd694ba481563c980e09fae41b60c0ecea4fc/lib/pbio/src/color/conversion.c#L85

And use sensor type specific parameters if needed to ensure that the rgb-to-hsv conversion produces consistent hsv values.

I don't think we need to match LEGO colors at this level per se, but they are still useful as a reference since all of us can reproduce the same results then.

[laurensvalk]

Would only v be distance dependent in theory?

If we can get H to be consistent across sensors, I think that's a good start.

[Novakasa]

Would only v be distance dependent in theory?

That was my impression when I last did a very superficial test, but I'll look at it more systematically.

[Novakasa]

Today I played around with this setup:

The train car has a sensor pointed to the side and some colored bricks are posted in line. I made the "scanning" somewhat consistent and changed the distance of the colored bricks to the sensor surface in increments of 1 lego plate from 1 to 11 plates. I also did the whole thing with lights on and lights off (admittedly not an extreme change in ambient lighting).

It's still early, but here is a result that kind of shows that in large part only the value changes with the distance, with the sweetspot being 2 plates distance.

In my case the effect of ambient light conditions is rather negligible.

I've pushed the related files to this repo: https://github.com/Novakasa/pybricks-color

[Novakasa]

Also, the largest RGB value I encountered is 473, exceeding the 440 that is used when mapping to 0-255 in raw_to_rgb(...).
I am using PUPDevice to measure the rgb values, but I haven't checked if that would amount the the same data as in raw_to_rgb.

[laurensvalk]

Nice, that is good to know. Which mapping was used to produce this? The default or your PR?

If I recall correctly, this also hold qualitatively for the SPIKE sensor, but the value for saturation and value were a bit different under the same conditions. Perhaps the sensor-type specific mapping from rgb-to-hsv can take care of making them more similar.

Also, the largest RGB value I encountered is 473, exceeding the 440 that is used when mapping to 0-255 in raw_to_rgb(...).

This might well be the case. I think there was an (unfortuate) hard cap of 440 on the SPIKE sensor RGB values coming out of the sensor. As a result, hue calculations break when the brick is too close.

I am using PUPDevice to measure the rgb values, but I haven't checked if that would amount the the same data as in raw_to_rgb.

Yes, these are the RGB values we use.

[Novakasa]

Which mapping was used to produce this? The default or your PR?

I only recorded the RGB values from PUPDevice, the mapping to hsv was done after the fact using the wikipedia algorithm. So, it should be the same as the current mapping in master.

I think there was an (unfortuate) hard cap of 440 on the SPIKE sensor RGB values coming out of the sensor. As a result, hue calculations break when the brick is too close.

So the ColorDistanceSensor uses the same mapping to make its RGB values consistent with the SPIKE sensor? If I see it correctly, we get an int overflow in raw_to_rgb with values above 440.

[laurensvalk]

I only recorded the RGB values from PUPDevice, the mapping to hsv was done after the fact using the wikipedia algorithm. So, it should be the same as the current mapping in master.

Except for the hack that skews some of the values. :) So when posting graphs (which are awesome, keep making them!), maybe add one line directly underneath to say how it was made.

So the ColorDistanceSensor uses the same mapping to make its RGB values consistent with the SPIKE sensor?

It uses the same mapping right now, but that does not result in consistent HSV values. The idea is to make one rgb-to-hsv mapping for the SPIKE sensor, and another for the Boost Color Distance Sensor, such that the end-user HSV values are comparable.

[Novakasa]

Except for the hack that skews some of the values.

Oh yeah, kind of essential for this topic :P

It uses the same mapping right now, but that does not result in consistent HSV values

Hmm, I think I still don't quite get it. If I look at the mapping from raw to rgb both seem to use different approaches. SPIKE vs Color-Distance-Sensor. The ColorDistanceSensor maps from 0-440 to 0-255, while SPIKE maps from 0-1023 to 0-255. As I encountered raw values above 440 using the ColorDistanceSensor, shouldn't we adapt the threshold there?

BTW, In my plot above the hue is in radians, so it is kind of hard to see it really, but the hue didn't really change much with distance.

[laurensvalk]

If I look at the mapping from raw to rgb both seem to use different approaches.

You're right 👍

shouldn't we adapt the threshold there?

We should probably drop every scaling and skewing operation we have, including the one you found (and I forgot about 😄), and replace it with a sensor-type-dependent raw-rgb-to-hsv conversion.

[Novakasa]

For this plot, I took the colors from rebrickable and plotted their RGB values. Then I overlaid my measured dataset at distance=4 (chosen to be the closest match to the rebrickable colors, which is kind of arbitrary because the RGB data is not normalized).

The order of LEGO colors is yellow, green, red, black, white, blue

This plot shows that a mapping in RGB space would have to be nonlocal for each channel (seeing e.g. that the green channel matches for the last color, but doesn't at the first one). Next I will investigate whether a mapping in HSV is more straightforward.

[Novakasa]

I tried a linear mapping in RGB space (a 3x3 matrix) and used scipy.optimize.least_squares to try to fit the "center color" of each peak to its corresponding rebrickable-rgb-value. This is the same plot as above, but with the linear mapping applied:

We can see that it improves in some cases, but can't fit everything simultaneously. I think this shows that at least in RGB, the mapping has to be nonlinear.

The resulting matrix looks like this:

array([[ 1.39618714, -0.04407636, -0.18775358],
       [ 0.02527175,  1.59905428, -0.37466398],
       [-0.21313028,  0.23361669,  1.20627808]]

Since I haven't put any absolute normalization on the data, this matrix can be multiplied by an arbitrary scalar factor to compensate for RGB normalization ( = value calibration)

[Novakasa]

In pybricks/pybricks-micropython#104, I mentioned this:

There is an issue when measuring a white lego brick at the optimal distance of 2 plates, however that seems to be a bug somewhere in the conversion from raw to hsv, as I see the raw RGB values at correct values whereas the hsv values have some suspicious artifacts (saturation jumps from ~0 to ~100 with little change of the raw values).

It just came to me that that is probably due to what I talked about in the above comment #116 (comment). White at distance of 2 plates probably clips the range of 0-440, so the mapping produces values above 255, leading to an int overflow.