Unable to download Microsoft CSAF

[Rafiot]

I'm not sure what the issue is, but fetching the CSAF entries from Microsoft fails in a weird way: downloader.log

{"time":"2024-11-19T15:19:47+01:00","level":"INFO","msg":"AdvisoryFileProcessor.Process: \"https://msrc.microsoft.com/csaf/changes.csv\" has an invalid time stamp in line 89: parsing time \"2024-11-12T08:00:00.0000000\" as \"2006-01-02T15:04:05Z07:00\": cannot parse \"\" as \"Z07:00\""}

The call:

csaf_downloader -d csaf_microsoft https://msrc.microsoft.com/csaf/provider-metadata.json

Do you have any idea how to fix it?

[bernhardreiter]

A line from https://msrc.microsoft.com/csaf/changes.csv:

2024/msrc_cve-2024-49060.json,2024-11-15T08:00:00.0000000

Checking the format, this seems to be missing a timezone, so it is an invalid file.

https://docs.oasis-open.org/csaf/csaf/v2.0/os/csaf-v2.0-os.html#7113-requirement-13-changescsv has

lines MUST be sorted by the current_release_date timestamp

https://docs.oasis-open.org/csaf/csaf/v2.0/os/csaf-v2.0-os.html#321122-document-property---tracking---current-release-date

(current_release_date) with value type string with format date-time

And date-time is defined in https://json-schema.org/understanding-json-schema/reference/string as

Dates and times are represented in RFC 3339, section 5.6.

"date-time": Date and time together, for example, 2018-11-13T20:20:39+00:00

https://datatracker.ietf.org/doc/html/rfc3339#section-5.6 then has an ABNF which has

time-offset = "Z" / time-numoffset
full-time = partial-time time-offset

The time-offset is missing in the file. Best would be if the provider could fix this on their end.

[Rafiot]

I'm reporting it to Microsoft, and pointing them to your details, thanks!

[Rafiot]

Update on that: the time issue is fixed, but now I get this error:

{"time":"2024-11-21T22:39:59+01:00","level":"WARN","msg":"Content type is not 'application/json'","url":"https://msrc.microsoft.com/csaf/2024/msrc_cve-2024-49028.json","content_type":"application/json; charset=utf-8"}

Their server returns application/json; charset=utf-8 when you expect application/json, but both should be valid (?).

I couldn't find it in the specs, but I'll drop them another mail if needed.

[tschmidtb51]

Their server returns application/json; charset=utf-8 when you expect application/json, but both should be valid (?).

True. IMHO, both should be valid.
Flagging @bernhardreiter to suggest on how we want to deal with the situation in the checker in general.

[s-l-teichmann]

Their server returns application/json; charset=utf-8 when you expect application/json, but both should be valid (?).

True. IMHO, both should be valid. Flagging @bernhardreiter to suggest on how we want to deal with the situation in the checker in general.

Yes, this should pass. Suggestion: If we detect an encoding other than UTF-8 we should issue a warning that the tools currently only supports UTF-8 documents. In the long run we may consider adding support for other encodings as well.

[bernhardreiter]

If we detect an encoding other than UTF-8 we should issue a warning that the tools currently only supports UTF-8 documents. In the long run we may consider adding support for other encodings as well.

As far as I see it must be an ERROR and we MUST NOT support other encodings.

The standard https://datatracker.ietf.org/doc/html/rfc8259#section-8.1 is clear on this:

JSON text exchanged between systems that are not part of a closed
ecosystem MUST be encoded using UTF-8

For the charset information it has

IANA Considerations

The media type for JSON text is application/json.

[..]

Required parameters: n/a
Optional parameters: n/a

The reported errata point https://www.rfc-editor.org/errata/eid5853 correctly argues that a charset MUST NOT be used. Also rfc6838's description of application does not have it. I follow that interpretation and thus for the checker, I suggest we keep the error if we find a charset=utf-8 parameter.

[Rafiot]

Alright, I'm talking to the Microsoft folks again. I don't really have a dog in this fight, I just don't want to implement my own downloader for fun.

[bernhardreiter]

I'm talking to the Microsoft folks again.

Thanks a lot for reporting the problem to them!

I don't really have a dog in this fight, I just don't want to implement my own downloader for fun.

Sounds like you do have an interest in a well implementable standard, though. ;-)

[Rafiot]

Absolutely! And more importantly, compatible implementations of said standard 😭

[adulau]

I'm really curious the origin of the issue has it seems the validation fails at Microsoft but also NCSC-NL too.

cve-search/vulnerability-lookup#94

But maybe adding tests with known sources would be also interesting for the downloader?

[bernhardreiter]

it seems the validation fails at [..] NCSC-NL
cve-search/vulnerability-lookup#94

The failure is correct and the error message of the downloader shows that they've "invented" an 18th month. So it is not this issue.

[Rafiot]

I just sent a reminder to Microsoft so we hopefully get it sorted.

[Rafiot]

Alright they got back to me and will give me an answer by next week or so.

@bernhardreiter I have an alternative solution, let me know what you think: the downloader could accept the presence of the encoding, check if it is UTF-8 and raise the exception only if it is something else?

[bernhardreiter]

the downloader could accept the presence of the encoding, check if it is UTF-8 and raise the exception only if it is something else?

Just preparing my talk for the CSAF community days, and thus reading https://en.wikipedia.org/wiki/Robustness_principle#Criticism . So far I think we should be strict early in the CSAF 2.0 adoption.

[Rafiot]

I get your point, and I'll keep not having a dog in this fight. Depending on Microsoft feedback I'll or won't implement a dumb downloader that just fetches the files from their repo.