Wrong text encoding on lists.openstreetmap.org sites

[gy-mate]

Mailman and/or Pipermail archives and/or shows some accented characters in the wrong encoding on https://lists.openstreetmap.org. See https://lists.openstreetmap.org/pipermail/talk-hu/2021-October/016347.html for example: következő is displayed like kďż˝vetkezďż˝ on macOS Safari 18.1 and Firefox 132.0.2:

Same issue with French text at https://lists.openstreetmap.org/listinfo:

Monthly archives downloaded from https://lists.openstreetmap.org/pipermail/talk-hu/ (e.g. https://lists.openstreetmap.org/pipermail/talk-hu/2024-October.txt.gz) also contain .txt files in ISO 8859-1 encoding which is not detected by macOS apps like BBEdit or TextEdit automatically.

[tomhughes]

It's probably impossible to completely fix this as we have no control over the encoding of individual messages in the archive, and the archiving is way too dumb to do any transcoding. Basically each language has a language set, and each language has an associated encoding and it will assume that everything to do with that list is in that encoding.

It gets even worse with listinfo because that is merging data from different lists that may use different encodings.

Some of the list descriptions are definitely wrong though as they have double encoded characters or unicode replacement characters in them. I've fixed a few of those where I can figure out what they should be but without speaking the relevant languages I can't always work it out.

[gy-mate]

I've fixed a few of those where I can figure out what they should be

@tomhughes Thank you very much!

but without speaking the relevant languages I can't always work it out.

I think talk-latam (French) is missing an é character but that's all I know.

[gy-mate]

the archiving is way too dumb to do any transcoding.

How could this be improved?

[tomhughes]

I think talk-latam (French) is missing an é character but that's all I know.

I doubt it will be French. Spanish is more likely...

the archiving is way too dumb to do any transcoding.

How could this be improved?

Realistically it can't. I mean obviously moving to mailman 3 might help but it's a huge amount of work and the mailing lists are gradually dying out anyway.

I suspect in practice that forcing utf-8 for everything would probably improve things a lot but I haven't had time yet to figure out how to do that.

[gy-mate]

I doubt it will be French. Spanish is more likely...

@tomhughes My bad, I mixed it up with talk-sn. Sénégal should be Sénégal (French), I think.

[gy-mate]

moving to mailman 3 might help but it's a huge amount of work and the mailing lists are gradually dying out anyway.

Yeah you're right. This is a low priority task then.
Should I leave the issue open anyway?