Improving NIP documentation structure and implementation reference system

[gzuuus]

NIPs have a fundamental problem: they are currently unstructured documents that fail to fulfill their primary function as a foundation for permissionless innovation. Developers have encountered this frustration before, as evidenced by commonly used clients with undocumented implementations where documentation sources are unclear. Additionally, teams of developers have experienced burnout due to political disputes over what should or shouldn't be included, ultimately bringing inconsistencies to the protocol and its ecosystem.

There have been numerous discussions about decentralizing the NIP repository, with alternative NIP repos emerging, but without any clear outcome. This situation creates confusion for newcomers trying to understand the protocol and its conventions, as well as for more experienced developers. While the NIPs repository serves as the reference, it doesn't provide a clear view of the different possible specifications and their implementations.

NIPs typically offer broad possibilities for use cases - the same NIP can be applied to diverse use cases, which is intentional and enables a range of applications from the same standard. While this approach is desirable for fostering innovation in an open and permissionless manner, it doesn't address how to document or reference these alternative or extended use cases. When someone attempts to modify a NIP to include particular details and conventions for a specific use case, it risks limiting the NIP's possibilities, which is undesirable since specifications should never be locked into specific implementations.

I would like to initiate a discussion to build consensus on how we can improve this situation and create a better structure for everyone implementing new use cases for defined NIPs. I propose the concept of NEPs (NIP Extension Proposals), which could be documents residing either in this repository or externally, but linked to their corresponding NIPs. This approach would allow NIPs to become more accurate to their nature: simple documents with a clear structure that enable permissionless innovation in an organized way, with NIPs referencing common NEPs as needed in a sort of "awesome" repository of implementations.

This approach may offer some significant advantages. By maintaining simpler and more structured NIP documents, we would likely see fewer breaking changes in the protocol. Developers working on specific use cases would be less likely to inadvertently affect the core specifications, eliminating unnecessary codebase updates for teams working on different implementations. Perhaps most importantly, this structure would empower communities, groups, and teams to reach consensus naturally among stakeholders who have skin in the game and are developing similar use cases. This creates a more organic and sustainable way to organize and maintain specifications, ultimately fostering a healthier ecosystem for all participants.

What do you think about this proposal?

[gzuuus]

@fiatjaf @pablof7z @vitorpamplona @staab @jb55

[SilberWitch]

I think making NIPs extendable is a good concept. We already do that, by having more information in our copies than in the originals.

It doesn't solve for the requirement to use GitHub, which is the root of a lot of problems.

[vitorpamplona]

You are better off proposing the structure you want to see via a PR than us debating about it.

Lots of people have debated the NIP structure and protocols. Several outside repositories have been used with mixed to negative benefits. No one ever actually proposed a better model that actually works. But I think everyone would appreciate a better way if there is one.

Question: Are you trying to improve the structure of the text in each NIP, of the file organization of all NIPs, of the NIP merging protocols to get new changes into a NIP or move away from github itself?

[SilberWitch]

I mean, we could easily move NIP patches to Ngit, but there's just no momentum, so long as everyone is on GitHub for their development projects.

The difference being that patches, unlike PRs, don't require approval from a committee.

[staab]

I basically agree with @gzuuus but as @vitorpamplona said, so far:

Several outside repositories have been used with mixed to negative benefits.

Many decent proposals have been put forward, but no one has sponsored their completion. What we need is not really more discussion, but for someone to lead the migration. This new platform should:

• Allow anyone to draft a new NIP or propose a revision of an existing one
• Allow anyone to vote on a change (in the case where text is being updated to clarify existing behavior rather than change the substance of the protocol)
• Allow NIP 89 applications to signal support for a version of a NIP (see Add feature detection for unmanaged groups #1604 for how this might work to cover migrations)
• Allow anyone without an existing web of trust to see what the current consensus is on the details of the protocol
• Have a complete revision history
• Allow durable linking to a NIP, either a current version or a revision
• Decentralize storage of NIPs, but still have a reasonably reliable index

Github actually checks almost all these boxes. But focusing on a technical solution also completely neglects the human dimension, which more difficult. Any system you build will invite politics of some kind. Switching to a politic that alienates the people who are willing to do housekeeping invites a tragedy of the commons. I'm sympathetic to the ideal of making the spec decentralized, but I worry that it's a utopian dream and will only make things worse. Leadership/ownership are useful and contribute to the stability of any venture. I'd love to see someone attempt this problem, but I'm inclined to stay with the devil we know.

[SilberWitch]

@staab that reads more like a list of git features than GitHub features.

[staab]

Vanilla git doesn't support PRs with conversations, issues, votes, or merge access. An additional platform that provides these things (whether github, ngit or something else) is necessary.

[ericfj2140]

I think making NIPs extendable is a good concept. We already do that, by having more information in our copies than in the originals.

It doesn't solve for the requirement to use GitHub, which is the root of a lot of problems.

Github as a point of failure is definitely a valid issue. However I believe this proposal is agnostic to the platform.

[amunrarara]

NEPs are an awesome thing IMO. Anyone can spin up any ol' repo to define a specialized Nostr spec that one-or-more devs agree is the protocol their system will adhere to. Agnostic to any reversion platform. If necessary, that NEP can be linked in the primary NIP repo.

I think this is all very well spelled out by @gzuuus, and the arguments / talking points posited by @vitorpamplona and @staab help to illustrate the point: there's issues in the consensus model of this NIP repo that aren't yet solved. Thus, a collective of Nostr marketplace spec devs will probably be best-suited in defining our own rules of the road, publish them in a separate repo, then spread the word and, hopefully later, merge our spec into the main NIP repo.

No, we wouldn't be solving the underlying consensus issue, which could come up to bite us later when our small group of marketplace devs becomes potentially bloated and PRs out the wazoo are comin' out of our ears, but that does give the Nostr network itself some time to solve the underlying issue, which then we can adopt in our own NEP repo, or at that time we can merge into the main NIP repo.

Purport: I say we charge forth and create a NEP repo for our Marketplace spec, open to the public, and develop it as-necessary until it is ready to be merged into the NIP repo, then worry about the consensus issues at that time, whilst we're zippin' Marketplace events between our clients with an already-established mutual protocol spec.

[ericfj2140]

You are better off proposing the structure you want to see via a PR than us debating about it.

Lots of people have debated the NIP structure and protocols. Several outside repositories have been used with mixed to negative benefits. No one ever actually proposed a better model that actually works. But I think everyone would appreciate a better way if there is one.

Question: Are you trying to improve the structure of the text in each NIP, of the file organization of all NIPs, of the NIP merging protocols to get new changes into a NIP or move away from github itself?

@gzuuus will be able to respond more thoroughly, but the main idea here is only to be able to include references to possible, more specific, extensions within the core NIPs. This would provide a reference trail from the core out to the “branches” which are more experimental or niche. These branches can develop independently as optional references with specific use cases, and elements from them could be eventually brought into the core if needed, but mostly it’s to keep innovation on the fringes and provide a reasonable mapping to that territory without inundating the core with all possible discussions. Ideas/refinements that are “good enough” get referenced.

I’m not sure if that will ever result in some obscure NIPs ever being made into NEPs as is sort of alluded to… that’s definitely a possibility for better organization. This doesn’t NEED to happen, but could be helpful eventually.

The issue is that the core NIP repo is what’s generally recognized as the current Nostr truth, and there are gatekeepers (with an important job) who can’t possibly keep up with every detail or nuance of every specific use case.

The idea here is to reduce the load of the core. The core NIPs remaining simple and flexible is highly important and we recognize the responsibility of the core maintainers here. We do need to do something like a release valve to prevent the “gatekeeper” effect on innovation that has formed on the core repo. I am sure this is not a role that the core maintainers of the NIP repo ever wanted.

So this is a lower stakes way of allowing innovation at the fringes and seeing what catches on organically before taking on the full debate of whether something happens on the core repo. If we can formalize references to NEPs, this burden can be optimized for both the core maintainers and innovators at the fringes.

[staab]

@ericfj2140 that's a helpful clarification. Currently we have a table in the readme of kinds that point to third-party specs. We could also potentially add a "third party NIPs" section to the readme (or maybe a separate file). I would imagine the requirement to get listed there would be that the spec passes a very low bar, basically it makes some kind of sense and has one implementation. Another option would be to use PRs as the third-party spec area. I know some people don't like to use github, but that area is completely open to anyone to contribute to.

[vitorpamplona]

Third-party NIP specs are quite common (Blossom, DVMs, joistr, etc ). If the goal of this is to create a marketplace spec, I would just call it that (Marketplace Implementation Possibilities). There is no need to generalize the name to account for any Nostr improvements.

[gzuuus]

I think we can see some patterns in the NIPs, where some of them are deliberately open and broad so as not to limit the use case, also there are cases where NIPs that are linked together without proper referencing each other and cause confusion or just create unnecessary complexity with possible custom implementations of something that's already defined somewhere.

I was thinking on NEPs coexisting within the same document as its parent NIP, just represented with a direct link to provide additional context and information. We could implement these changes incrementally and as needed, simply by adding sections in NIP documents that link to specific NEP repositories. Or if needed studying how to reduce the load of some NIPs and breaking them in this extension possibilities

The scope of NEPs can vary depending on their purpose. They can serve as simple clarification notes, define relationships between NIPs, or provide comprehensive documentation depending on specific requirements, all of which are extension possibilities.

I see potential for NEPs across several existing NIPs, such as lists, calendar events, long forms, wikis, labels, badges, NIP-89, DVMs, NIP-99, NIP-05, etc. Git version control would be essential to this structure, and organisations could adapt different forms to suit their needs. It's important that NEPs clearly identify their supporters and maintainers, with an easy way to contact them.

NEPs may need to meet certain criteria to be associated with their parent NIPs, but it's important to note that not every NIP needs NEPs. Some NIPs serve pure, straightforward functionalities or use cases, while others remain open to allow different implementation possibilities, or they incorporate different conventions for different use cases that might be difficult to handle or reconcile as they become more complex

I think we could streamline some NIPs to focus on core data structures, common tags used, or general usage without limiting implementation possibilities. This reduces dependencies in broader NIPs and, as Eric noted, reduces the burden on NIP maintainers.

For naming conventions, we can use something like NEP-99:Marketplaces, NEP-05:BOLT12, or `NEP-58:BADGE AUTH', as examples, where the number corresponds to the extended NIP. The ultimate aim is to achieve better defined structures with fewer dependencies, while improving the documentation of implementation practices.

[ericfj2140]

Third-party NIP specs are quite common (Blossom, DVMs, joistr, etc ). If the goal of this is to create a marketplace spec, I would just call it that (Marketplace Implementation Possibilities). There is no need to generalize the name to account for any Nostr improvements.

@vitorpamplona We’re definitely chasing a marketplace spec separately but this is bigger picture. I believe @gzuuus is working on multiple projects.

[ericfj2140]

@ericfj2140 that's a helpful clarification. Currently we have a table in the readme of kinds that point to third-party specs. We could also potentially add a "third party NIPs" section to the readme (or maybe a separate file). I would imagine the requirement to get listed there would be that the spec passes a very low bar, basically it makes some kind of sense and has one implementation. Another option would be to use PRs as the third-party spec area. I know some people don't like to use github, but that area is completely open to anyone to contribute to.

I like the idea of this being in the readme. “Third party NIPs” is less fun than NEPs imo but like tomayto tomahto ultimately.

You’re right in a technical sense about the PRs being an option but I think a lot of people feel their work dies on the vine there.

If the readme referenced what a NEP was and gave people a path for an “outer layer” of approval that could be harmlessly referenced within NIPs as @gzuuus is saying, I think this would be a really great path forward and an organizing framework that allows new entrants to follow the branches to the appropriate source… with better hopes of successfully contributing to something even if it’s not core.

[dskvr]

This looks well thought out and it brings up potential solutions to many known issues with the existing process. There are some important questions unanswered however. Namely:

1. Migration Plan; likely the most difficult. failure here can result in further fragmentation and diminish developer morale.
2. Drop-in replacements for the features provided Github which make the NIP process feasible are missing; which has already been mentioned.
3. global state on nostr is difficult if not impossible, which adds further challenges to this process. If we rely on WoT then NEPs become increasingly fragmented. There would be many situations where people can't verify a NEP exists. Or a situation where multiple identical, but incompatible, NEPs exist in parallel for no other reason than neither party is aware of the other. This could be resolved with dedicated NEP relays, but this reduces resiliancy and adds fragility to a process that demands to be anti-fragile.

Personally, I'm most interested in separating NIPs (which IMO should be Relay Implementation specific) from event schemas; laregly because the shape of an event, in most situations, has little to no bearing on the the implementation of a relay. I also believe that it would be much simpler to accomplish.

Other than the concerns I mentioned above, I'm agnostic, will defer to those with passionate opinions on the matter. Thanks for sharing!

[gzuuus]

You're right @dskvr, it will also be worthwhile to define "client-side" and "relay-side" nips. That might help to have a clearer organisation and better maintainability.

But to your unanswered questions:

1. Migration can be done incrementally, we don't need to refactor every nip right now, we can add neps as they appear, and if someone suggests a nep for a nip that is convoluted and has dependencies with others, and the community determines it we should do some prior work to break those dependencies in to neps. Neps to add nips interoperability can use a convention like NEP-89-05-51:... So it is clear that the proposal is to unite these nips in a specific use case. They also can appear referenced in the nips are using
2. We can still use git and github, but this should be platform-agnostic, the only thing I am suggesting is to identify broad use case nips and break them down into neps just to have a structured way of referencing how people are implementing these broad case nips. Neps can also be made out of PRs, so if someone is working on a PR to add x functionality and people are participating and it is being implemented and used, it can be listed as a NEP if appropriate.
3. Isn't this already happening? but in a way that we prefer to ignore rather than in a more determined and structured way? In any case, we can agree that the nips repo is the source of truth, so that neps that are referenced can be found by anyone.

This is just an exploration of how we can improve this.

[SilberWitch]

Currently we have a table in the readme of kinds that point to third-party specs. We could also potentially add a "third party NIPs" section to the readme (or maybe a separate file).

This is the only part of the repo that seems to actually help us work, as people can search the ReadMe for topics and see a link to our spec and click on it. And it keeps different projects from clashing on kind numbers. You can see the NIP repo on our homepage or on gitworkshop.dev. Don't need GitHub for that. Can also suggest-and-discuss patches or write issues there. And Alexandria will add kind 1111 comments and highlights and forks and stuff, and display wiki pages, so that will be a second possibility.

If you have a spec that doesn't contain a new kind, it's more difficult. That probably would be better in a second table, or just include those entries, as well. Listing things at the bottom of existing NIPs would lead to stagnation, I think. NIPs are really difficult to change.