What principles should we hold and share, when considering any change to the IATI standard?

[stevieflow]

Stated in our Terms of Reference for the SSWG is

There are no overarching principles that can help inform and support assessment of any potential change.

And - as stated outcome for us is:

Propose a set of guiding principles that can underpin assessment of all future changes.

Leading to

A document outlining the guiding principles that would inform consideration of any changes to the IATI standard.

To get us started, it would be great to hear from members what principles you see as important, regardless of what type of change we might be considering.

We already touched upon some in our 17th September meeting, including:

• Having clear consideration of existing users
• Understanding - as far as possible - the impact a change might have on other parts of the IATI standard
• Giving good notice to the community in terms of expectations and timescales

So - whether you publish IATI data or use it (or both), if you were to read about a potential change to the IATI standard, what kind of things would you expect to be considered when assessing it?

Thank you for you time!

[bill-anderson]

Re: having clear consideration of existing users
This needs a more pragmatic formulation that allows the majority to benefit at the expense of a (defined) small minority.

[bill-anderson]

Stewardship of the standard has historically been dominated by supply-side actors. How do we change this balance? Demand** should be reflected in both the content and the process.

(** I recognise that the lack of demand has been a huge problem over the years and that IATI has a broader mission to stimulate demand.)

[sylvanr]

Major, Minor, Patch.

Major updates can be considered to be Incompatible changes, such as the removal of any element, or the addition of new core elements. A minor update could still be incompatible (diverging from the semver definition), by, for example, making a change to rules around existing fields, or adding child elements to existing elements. Patches could be seen as both syntactic (#6) and semantic (#5) changes to the definitions of the standard.

Major updates could be proposed by the IATI community in an updated standard update process, where a centralised discussion, such as these nice github discussions could be had. Then, once all input has been heard, the technical seat (or other technical representative to the secretariat, like ODS currently), can collect the thoughts and present these at a Members Assembly, where members vote on the ideas (alternatively to all members, only those interested or part of the technical community, in a breakout session).

Minor updates could be proposed online, possibly also in a “discussions”-like setup, by the technical team, or the technical community. At the next MA, there could be an initial creation of-, then in future MAs, a refresh of a “technical community representative group (TCRG)” , which will be able to vote online on proposed changes. The technical seat (or other technical representative to the secretariat, like ODS currently), can periodically collect these minor updates, and put those to a vote for this TCRG.

Patches should be proposed by the Secretariat with a short term voting period to determine whether this is a patch, or a minor update (possibly shorter than the proposed 45 days in #6, but that is my opinion). During this voting period the community can decide whether or not something should be a patch or a minor update. If it is deemed a patch, the Secretariat should be able to update the standard directly, while if it is deemed a minor update, the minor update flow should be followed. This process can also be applied to all updates: An update is proposed, with a proposed update type (Major, Minor, Patch), which is a short term vote, after which the actual update is then voted on.

[bill-anderson]

I think it would be worthwhile to broaden the definition of what is allowable in a patch as far as possible (again thinking pragmatically)

[sylvanr]

Friction

Friction has historically usually been present for any change to the IATI Standard. This is why the standard update process should define a “friction limit”. For example, if the voting pool is large, the “voting balance” could be 90/10 or 80/20, and if there is more than 20% against the change, the change is reconsidered. Either by altering the change, or dropping it altogether. If the voting pool is smaller, this friction limit should be higher, to prevent a small part of the IATI community preventing beneficial changes to the standard. For example, if only ten community members participate, with a vocal minority of three members refusing.

However this is highly speculative, and perhaps a metric could be developed (either by exploring previous votes, previous technical working groups etc.) to determine what a good friction limit would be. Like Bill said, a “pragmatic formulation”.

Note: I was not involved with IATI around the time of the last update, so all my information about friction is based on anecdotal reports throughout different technical meetups

[bill-anderson]

The size of the pool can be increased by another principle (agreed after v2 upgrade): upgrades should agree upon real-world use cases (involving non-technical participants) before technical drafting takes place.

[sylvanr]

Update transparency and active outreach

For every change to an existing element, specifically for changed or deprecated elements, set up automated outreach: Identify through a datastore which datasets contain reference to these elements, and use the contact information to update the publisher on this specific change.

Additionally, consider automatic updates to the datasets. In the above mentioned outreach, provide a complete updated copy of the affected dataset, and send it to the publisher along with the change definition (bonus: include snippets of the update).

Define an openly joinable “IATI Data Tools Community”, where developers or product owners of IATI related tools can freely join, and be informed about changes in IATI Secretariat-maintained tools related to any updates to the standard. This way, not only IATI Secretariat-maintained tools can be updated to reflect changes in codes, but all data tools can synchronise in these updates, if desired. (Reference to Rob’s comment on #4)

[sylvanr]

At risk of sounding too hip: We could feed all the documentation of the IATI Standard (specifically on the website), to a LLM, and ask it to provide all potential semantic uncertainty and syntactic errors.

This could be easily achieved, whilst the alternative, a person or a volunteer group going through might miss small details, especially after reading and processing the first 50 pages, while there are still 300 pages left to go.

Of course, having a team of IATI Veterans go through the rulesets would be beneficial, as they might spot semantic issues based on their knowledge of the original introduction or last change to those elements.

[akmiller01]

Hi @sylvanr , just as a quick experiment, I attached iati-activities-schema.xsd to a chat with gpt-4o-mini and gave it your exact question. A good next step I think would be to write a custom system prompt and send all of the text from the schema files straight through to an LLM, as 4o was missing too much context to give a great response. Below is the results of the chat experiment:

User: Attached is a schema file from the IATI activity standard. Provide all potential semantic uncertainty and syntactic errors you can find within the file.

gpt-4o-mini: The schema file appears to be an XML Schema Definition (XSD) for the IATI (International Aid Transparency Initiative) activities. From the initial inspection, here are some potential sources of semantic uncertainty and syntactic errors that could be present in the file:

1. Syntactic Errors:

• File inclusion issues: The schema includes two external files (iati-common.xsd and xml.xsd) using xsd:include and xsd:import. If these files are not located in the same directory or are unavailable, validation errors may occur.
• Invalid namespace references: The xsd:import includes the http://www.w3.org/XML/1998/namespace. If the schema for this namespace is incorrect or missing, this may result in validation problems.
• Version declaration: Ensure that the declared version 2.03 is supported by the system processing this schema.
• Encoding declaration: The file declares UTF-8 encoding (), but this can lead to syntactic issues if the actual encoding differs.

2. Semantic Uncertainty:

• Documentation clarity: The documentation states that the document type may be extended with additional elements and attributes from explicit XML namespaces. This could create ambiguity regarding which additional elements are allowed, and what rules apply when extending the schema.
• Compatibility with different XML parsers: Depending on the implementation, certain XML parsers may handle namespaces, imports, and includes differently, leading to uncertainty in how the schema is interpreted.
• Namespace collisions: The file references multiple namespaces. If extensions introduce new namespaces, it could lead to namespace conflicts or ambiguity.
• Element and Attribute extension rules: The comment mentions that the schema can be extended with "additional elements and attributes," but it does not specify how these extensions should be structured or whether there are any restrictions on the type of extensions allowed.
  Next Steps:
  I can perform a more in-depth analysis by checking the structure of the elements and attributes declared in the schema. Should I proceed with this?

User: Yes

gpt-4o-mini: Here are some insights from the parsed schema structure:
Elements:

• iati-activities: The root element of the schema, which seems to lack explicit type, minOccurs, or maxOccurs attributes. These attributes are important for defining how many times this element can appear, leading to potential syntactic or semantic ambiguities.
• Unnamed elements: There are elements without a name attribute, which creates ambiguity since elements need names to be referenced and understood correctly by parsers. This could result in errors during validation.
• iati-activity: Another critical element, but it also lacks explicit type, minOccurs, or maxOccurs attributes. These missing attributes might lead to confusion about how this element should be used or how frequently it can appear.
  Attributes:
• version: A required attribute of type xsd:string, likely to track the version of the schema.
• generated-datetime: An optional attribute of type xsd:dateTime, presumably to record the date and time when the data was generated.
• linked-data-default: An optional attribute of type xsd:anyURI, which may introduce semantic ambiguity regarding how linked data is handled.
• type: An optional attribute of type xsd:string, potentially leading to uncertainty about what values it accepts if not clearly documented.
  Potential Issues:
• Missing name attributes in elements: The unnamed elements present a critical issue, as the schema cannot function properly without clearly defined names for all elements.
• Lack of type, minOccurs, and maxOccurs attributes: These are essential for understanding the structure, constraints, and multiplicity of elements. Their absence can lead to improper interpretations or failures during XML instance validation.
• Optional attributes with unclear usage: Attributes like linked-data-default and type are optional but need more clear documentation to avoid misinterpretation in extensions.
  These issues should be addressed to ensure the schema functions correctly and is understood unambiguously by users and systems. Should I delve deeper into a specific part of the schema?

[sylvanr]

Already some fun results at first glance.

To optimise, we could use something like the assistants feature, or build a custom assistant thread chain, where you can provide system instructions along with the prompt, leading to possibly more concise answers. Then segmenting the prompts, for example, sending a prompt for each element, which could contain the descriptive text, relevant subset of the schema, and extra documentation, and any other official documentation that the technical team can dig up for that very small element.

Then once that process is defined, get the feedback for each field, and use that to critically review each piece of official documentation.
That way, you can allow the llm to use their limited response tokens to deep dive into each of the fields and each of the individual issues, rather than global.

[sylvanr]

Here is a response looking exclusively at the iati-identifier description. Probably, part of the response will not be useful or make 100% in the context of the review, but for example it also identifies the issue of #5, in it's 3rd point.

IATI Standard Review Assistant
Here are the potential semantic uncertainties and syntactic errors in the provided description of the IATI-identifier:

Semantic Uncertainties:
Prefix Clarification: The description states that the iati-identifier must be prefixed with either the current IATI organization identifier or a previous identifier reported in other-identifier. It may be unclear how to determine which previous identifier to use if there are multiple options available in other-identifier. Further clarification on selection criteria might be needed.

Unique String for Activity: The definition mentions that the prefix should be followed by a unique string for the activity. However, "unique" is somewhat ambiguous—does this string need to be globally unique across all reporting organizations or just unique within the context of the reporting organization?

“MUST NOT” vs. “should”: The rules differentiate between "MUST NOT" and "should". The use of "should" in rule 1.1.21 may imply that following this guideline is optional, which could lead to inconsistencies in implementation. This needs to be harmonized to either explicitly mandate or recommend starting with the organization identifier.

Syntactic Errors/Observations:
Hyphen Usage: The instruction for constructing the identifier states "separated by a hyphen “-”." However, it is unclear if this refers to using a single hyphen or if there could be additional hyphens if there are different components in the unique string (e.g., multiple parts within the activity identifier).

Redundant Phrase: The phrase "the prefix and the suffix" may be misleading as it implies there are separate prefixes and suffixes in some contexts. It could be clearer to simply refer to them as "the organization identifier" and "the activity identifier" respectively.

Inconsistent Terminology: In the definition, the term "activity identifier" is used, while in the rules, it refers to "unique string for the activity". To maintain uniformity, it would be better to consistently use the same term throughout the text.

List of Prohibitive Characters: There is a rule that states the identifier must not contain certain symbols. However, the corresponding list could be improved to have grammatical consistency (e.g., adding “or“ before the last item, "?" for clearer reading).

Incompleteness of Example: The example provided lacks clarity since it does not explicitly state that "ABC123" is considered a unique string for the activity. This could lead to misunderstanding about how to structure the identifier.

By addressing these uncertainties and syntactic issues, the clarity and usability of the IATI-identifier rules and definitions could be significantly improved.

[bill-anderson]

Comparability
The standard should ensure that data should be comparable across publishers. The primary vehicle for this is common codelists. This raises the question of how useful results data is when there are no common standards for classifying indicators.

[Melina-Loewer]

I believe this is a very important point. However, the comparability of individual data fields may not always be feasible, as the project logics can vary significantly, right? For example, IDs are comparable across all activities, but certain elements like transactions (e.g., expenditures) are not necessarily comparable across all activities. (One specific case, for instance, is the reporting for BMZ, where the activities of KfW, as the financial implementing agency, present a unique aspect.)
I think, it is probably similarly challenging to establish comparability for results data. For the comparability of results data, there would likely first need to be global agreements on how results data is collected and impacts are measured...

[markbrough]

I think there is a balance to strike between comparability and being faithful to the particular business model of each publisher. I think a pragmatic approach can allow e.g. the results elements of the Standard to be used in a consistent way across publishers, while recognising that different publishers will have different results indicators and approaches for measuring results. I think we should be looking to find commonality as much as reasonably possible, because that makes the data much easier to use, but we shouldn't be trying to fit square pegs into round holes.

I feel like some of these issues may be more a question of improving documentation and guidance (perhaps including more real-world examples, including a variety of different cases which can be represented in a similar way in the Standard)?

[bill-anderson]

Simplicity
The one advantage OECD data has is that it comes in a flat file. I’m not suggesting IATI goes flat, but we should be more sensitive to the usability of data.

[markbrough]

Fully agree. I think the implications for the usability of the data should be discussed and carefully considered when considering a proposal. There can be different ways to address usability questions (e.g. improving / adapting tools to mitigate the impact of additional burden on users; vs. not permitting additional complexity in the Standard).

[siemvaessen]

Modularity

Ensuring a development environment where clear guidelines for use of (IATI) open modules (framework, etc) is provided by IATI, allowing third parties to create their custom modules that are able to connect the standard itself. This is preferred as opposed to fixing everything for anyone in the standard itself. Modularity allows for implementation of a broad set of usecases that do not need any prior approval, and they will work as long as modules follow the rules of engagement as set in the guidelines / framework.

[Melina-Loewer]

That sounds very interesting. Perhaps more could be explained about this during the meeting today?

[davidmegginson]

A small addition: when we consider the impact on users, we have to be mindful that backwards-compatibility for data producers isn't the same as backwards-compatibility for data consumers. For example, if we add a new feature/practice Y and deprecate (but keep) X, that avoids backwards-incompatibility for publishers, but introduces it for people using IATI data.

I won't say there's no free lunch, but it's hard to find sometimes.

[robredpath]

Thanks, everyone - this is a really useful conversation. It would be useful for the outcome of this to be some concise principles that we can adopt, so I've tried to summarise below.

My first observation is that we have discussed both principles for the process of changing the standard, and principles that we should respect about the standard itself during that process. I've separated those out below. I'd welcome feedback as to whether you all feel this is an accurate representation of what we've discussed above and if there's anything we want to add/change/remove.

Principles For Change

• Pragmatism: We should use our practical understanding of how people use the Standard to guide the process that we go through to make changes, over an entirely theoretical / "on-paper" assessment of the impact of a change.
• Respect existing users of the Standard: If proposed changes affect existing users, we should consider the negative impact against the potential positive impact and make appropriate mitigations if we do decide to proceed.
• Calibrated friction: There should be the right amount of friction involved in making a change, so that the standard doesn't change too much, too quickly, or is swayed too readily by emergent trends; this should be balanced by ensuring that there isn't so much friction as to prevent any change at all
• No Surprises: changes should be communicated through appropriate channels well ahead of the change being made, with specific outreach if particular groups of users can be identified to be affected
• Clear responsibility: the roles of individual community members, the Secretariat, Board and Members (including, if we want, special categories for the Board technical seat and Secretariat technical implementation partners) should be clear, both in terms of what they are responsible for doing, and the authority that they have to make decisions.

Principles For The Standard During Changes

• Increase Simplicity: The Standard should be easy to understand, favouring straightforward forms where possible
• Increase Modularity: The Standard should be modular, such that groups interested in collaborating around a particular concept are able to do so
• Increase Precision: Ambiguity in the Standard should be minimised except where this is intentional to provide flexibility - with the understanding that this is important for comparability and interoperability of data
• Improve Integrity: The Standard should be coherent within itself
• Make a clear social contract: Everyone should understand that the Standard is an evolving artefact and that its use implies an ongoing expectation that users will update their processes every so often; this is matched by the "respect for existing users" above that means that we don't make this an excessive burden

[davidmegginson]

To follow up on @bill-anderson 's point, in the past we've always favoured making life easier for data providers by ensuring that changes are backwards-compatible for them, but in almost every case, that means that the change is backwards-incompatible for data consumers (and creates more work). As I wrote in another thread, whatever choices we make, we need at least to acknowledge that there's no silver bullet -- every change will cost someone time, effort, and disruption.

[stevieflow]

From notes from meeting 22nd Oct:

The transparency of the process should also be an explicit principle.
Broadening participation and outreach to affected groups is an important principle.
Different types of changes - use case driven, technical/maintenance driven, changes mandated by changes to other standards.

[stevieflow]

IATI Standard: Principles for change control #draft

Introduction
At the heart of the International Aid Transparency Initiative (IATI) is a data standard. This sets the framework and detail in terms of how data should be formatted, structured and ordered, and supports those that both publish and consume “IATI data”.

The data standard has been adopted, shaped and guided by a large number of organisations involved in international development and humanitarian assistance. To date, over 1,700 organisation publish data using the IATI standard, whilst a range of actors and stakeholders consume it for a wide variety of use cases.

Changing and updating the IATI Standard
Since its inception in 2010, the IATI Standard has undergone review and change, in order to become more useful. In the period 2010 to 2018, it progressed through seven cycles of change, resulting in the current version (2.03). These changes, especially early on in the initiative, helped strengthen and support the adoption of the IATI Standard, as takeup and use grew.

In the intervening six years no further updates have been rendered. In this light, the IATI Governing Board and Members commissioned a short-term Standard Stewardship Working Group (SSWG) to assess the change control process for the IATI Standard. Alongside this, the SSWG were asked to consider what principles could be put in place to further govern and guide future collaborations to update the Standard.

The purpose of principles
Sitting alongside the processes and protocols that help govern the change control process for the IATI Standard, it is envisaged that a short set of principles will set our key aspects to be observed, regardless of how big or small the update.

In discussing such principles the SSWG moved towards to sets of principles that can help guide the initiative:

• Principles For Change Process: principles that should be respected and observed during any change control process
• Principles for the Standard: principles to observe and consider when making decisions about changes to the IATI Standard

Principles For IATI Standard Change Processes

Any process to change the IATI Standard should respect these principles:

• Pragmatism: We should use our practical understanding of how people use the Standard to guide the process that we go through to make changes.
• Impact: If proposed changes affect existing users, we should consider the negative impact against the potential positive impact and make appropriate mitigations if we do decide to proceed.
• Usability: We should always consider the full range of uses for the IATI Standard, including publishing and consuming data, to evaluate any new change or overhead.
• Momentum: There should be the right amount of friction involved in making a change, so that the standard doesn't change too much, too quickly, or is swayed too readily by emergent trends; this should be balanced by ensuring that there isn't so much friction as to prevent any change at all.
• Notice: changes should be communicated through appropriate channels well ahead of the change being made, with specific outreach if particular groups of users can be identified to be affected.
• Clarity: communication about changes to the IATI Standard should always be presented in clear terms, with any jargon or other terms fully explained.
• Authority: the roles of individual community members, the Secretariat, Governing Board and Members should be clear, both in terms of what they are responsible for doing, and the authority that they have to make decisions.
• Transparency: any process should always be clear in participating, timescales and current status.

Principles For The IATI Standard During Changes

When evaluating potential changes to the IATI Standard, these principles should be considered:

• Increase Simplicity: The Standard should be easy to understand, favouring straightforward forms where possible.
• Increase Modularity: The Standard should be modular, such that groups interested in collaborating around a particular concept are able to do so.
• Increase Precision: Ambiguity in the Standard should be minimised except where this is intentional to provide flexibility - with the understanding that this is important for comparability and interoperability of data.
• Improve Integrity: The Standard should be coherent within itself.
• Integrate Change: The Standard is an evolving artefact and its ongoing use implies an expectation that users will update their processes every so often; this is matched by the "Impact" principle above that means that we don't make this an excessive burden.

[davidmegginson]

It looks good, and I appreciate the mention of data consumers.

I have one very-small proposed tweak. I think it would be good to acknowledge the ultimate intended beneficiaries of this work somewhere in the document, and to make it clear that schema changes need to serve their needs (directly or indirectly) in the end, and that they have a right to be consulted (the "nothing about us without us" principle).

[stevieflow]

Very helpful, thanks @davidmegginson

Perhaps this can be an a clear section to outline who the principles are for, as in:

• For those facilitating/coordinating any change control process
• For those in positions to make decisions about any change proposals
• For anyone observing the process (to ensure these principles are observed)

In terms of the ultimate intended beneficiaries, then agree that we can make it explicit that all the principles must align to support both existing and future users of the IATI Standard (the second cohort obviously being a little harder to pinpoint!)?

What do you think?

[davidmegginson]

That sounds good. Thanks, @stevieflow

[HermanvanLoon]

I would like to add another principle: no redundancy. With redundancy in this context I mean that a change may not cause any duplication of functionality in the standard. Duplication of functionality makes the standard more complex and the data become less comparable over publishers because the same information is expressed in different ways by different publishers. An example of a redundant change would be adding a new country codelist based on the 3-letter ISO country codes. This change should be rejected because it is functionally the same as the current 2-letter ISO country list.

The no redundancy principle would also exclude changes for data elements which can be derived from existing IATI data elements. E.g. a change for adding an activity status 'inactive' which covers both the 'suspended' and 'closed' status would be rejected.

[stevieflow]

Thanks @HermanvanLoon ! This is a very welcome suggestion. I think we can fold this into the principles around the Standard,as you suggest. More practically, this can also become a very specific question in the change proposal template, so that we have to explicitly think through the issue. What do you think? @robredpath may have also tried to capture this in his earlier drafting, so appreciate you flagging it here @HermanvanLoon

[HermanvanLoon]

For me 'Clarity' means that every change leads to a better understandable standard. So the meaning of every real-world concept modelled in the standard should be easy recognizable, understandable and open to only one interpretation. Only when this condition is met, clear communication about is change is possible.

An example i.m.o. is the definition of the Organisation Role which now leads easily to discussions on how to interpret the meaning of the different codelist values.

[stevieflow]

Again, a very welcome point @HermanvanLoon

Clarity in the principles about is currently linked to the process, so that we communicate and discuss in clear terms, etc - which I think is a principle we want to see.

But also agreed - when we assess and process ideas and proposals for change to the Standard itself, then this question of clarity could become very useful for us. Indeed, why would be want to change the Standard to make it less clear? !

Once more, we can adjust the principles as drafted - but also make sure this comes through in the templates/questions we ask about any change. Thanks again @HermanvanLoon - much appreciated