Cross-SDK conformance checks

[justinabrahms]

Right now, we have some READMEs which list which large features an SDK supports. That's good for users, but not for developers working on SDKs.

I'm interested in having a per-spec-item conformance matrix. I'm thinking something like:

|  spec | python | java | rust |
|-------+--------+------+------|
| 1.1.2 | y      | y    | n    |
| 1.2.0 | n      | y    | y    |

I think this will make it much easier to get contributions and also see where the deficiencies are.

If we define a simple json data format, we can build out the matrix. I'm already doing this in the java sdk and I think that's been modified to work in other repos too.

Thoughts?

[beeme1mr]

I wonder if we could leverage Gherkin or something similar. It would be great if we could leverage existing tooling.

[dominikhaska]

I understand that each SDK has version (semver) - would it be just better to match with spec version? Of course having that in one place would be better for maintainers but if it's easy (relatively) to extract those data automatically. Still we don't have too many SDKs to follow and maybe starting with manually created table could be sufficient

[justinabrahms]

I think that would be a change that might be good for users. I'm focused on SDK authors right now though. I'm hoping that specifically for devs, I can know which spec items aren't accounted for in the SDK.

[toddbaert]

We discussed having SDK version that match the spec version, but decided against it. The main reasoning is new language features or paradigm shifts in a language itself can present the need to make breaking changes which only impact the SDK in question, and don't originate from a change in the OpenFeature spec. In fact, we're going though this exact thing in .NET right now, and releasing a 2.0. The 2.0 in .NET isn't due to any change in OpenFeature, but in the C# features we are using to implement it.