Reject document if the JSON is not UTF-8 encoded

[bernhardreiter]

Currently documents that are not UTF-8 encoded will be downloaded and not (most liely) flagged by the checker.

We found examples of latin-1 encoding.

But JSON on the internet must be utf-8 encoded.

The standard https://datatracker.ietf.org/doc/html/rfc8259#section-8.1 is clear on this:

JSON text exchanged between systems that are not part of a closed
ecosystem MUST be encoded using UTF-8

curl -O https://ftp.suse.com/pub/projects/security/csaf/suse-su-2024_1139-1.json
python3
Python 3.11.2 (main, Sep 14 2024, 03:00:30) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import json
>>> with open("suse-su-2024_1139-1.json") as f:
...   json.load(f)
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/lib/python3.11/json/__init__.py", line 293, in load
    return loads(fp.read(),
                 ^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xae in position 1591: invalid start byte
>>> b=open("suse-su-2024_1139-1.json","rb")
>>> b.read()[1591]
man latin1 | grep " AE "
       256   174   AE     ®     REGISTERED SIGN

[bernhardreiter]

A call like https://pkg.go.dev/unicode/[email protected]#Valid could help.