Dotnix follow-up

[Ra33it0]

Project Abstract

Dotnix is a collection of Nix packages and NixOS modules designed for creating and managing Polkadot/Kusama Validator Nodes, emphasizing both security and ease of use.
This application is for a follow-up grant: 0e034e3

Grant level

• Level 1: Up to $10,000, 2 approvals
• Level 2: Up to $30,000, 3 approvals
• [ x ] Level 3: Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

• The application template has been copied and aptly renamed (project_name.md).
• [ x ] I have read the application guidelines.
• [ x ] Payment details have been provided (Polkadot AssetHub (USDC & DOT) address in the application and bank details via email, if applicable).
• [ x ] I understand that an agreed upon percentage of each milestone will be paid in vested DOT, to the Polkadot address listed in the application.
• [ x ] I am aware that, in order to receive a grant, I (and the entity I represent) have to successfully complete a KYC/KYB check.
• [ x ] The software delivered for this grant will be released under an open-source license specified in the application.
• [ x ] The initial PR contains only one commit (squash and force-push if needed).
• [ x ] The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).

[Noc2]

Thanks a lot for the application. Could you also integrate the DOT percentage? See the new template: https://github.com/w3f/Grants-Program/blob/master/applications/application-template.md#overview-1

[ajk-code]

Hey there
A couple of questions regarding this project, not sure I clearly understood while reading the application:

• In general, to me it sounds like an in-house deployment project, not something for a wider audience
• You mention as ecosystem fit 'people that want to host a validator, don't have the technical knowledge': Honestly I do not think someone without technical knowledge can deploy nixOS or nix packages on their machine. Installing docker on peoples notebooks with network bridges etc. might not be a good idea either. Copy/pasting from the polkadot wiki and using the parity repo is probably way easier, don't you think? A single line edit of the default config file is enough. If necessary in a VM for testing purposes.
• How is the system gonna be scanned for CVEs? What is this integrated scanner and which public CVE databases are used for the enrichment? How is an operator getting the result for high CVSS CVEs?
• How is usability improved by exposing the validator to polkadot.js?
• What is meant with polkadot.js in general? The libraries or the public frontend?
• What is the purpose of exposing a validators RPC in a 'validator' context? And why over VPN?
• Is the deliverable an OCI image or nix packages? Is is it podman compatible or docker only?
• Is dotnix gonna be available in the public nix repos (stable/unstable channels?)
• How is the polkadot binary built? From source, parity binaries or parity OCI/docker images?
• Is the secure validator mode supported? Referring to landlock/seccomp
• Is session key management somehow integrated apart from node key mgmt?
• How/who is gonna maintain this? What are the recurring costs?
• What about other standard security features such as selinux policies, secure boot, CIS compliance, fido2 authentication etc.?

Thank you

[PieWol]

Hey @ajk-code , thanks for participating in the review process 🙏 .

[Ra33it0]

Hey @ajk-code , thank you for your questions. I am happy to clear things up a bit.

The idea behind Dotnix is to simplify the deployment and administration of secure Polkadot validators by including various helper services for monitoring, backupping, e.g. into a single Nix flake that can be deployed through simple means.
In this grant application, e.g. we are doing the groundwork to deploy monitoring for visualizing the collected metrics on a single machine.
Later stages of Dotnix will allow deploying the same configuration on a mesh of machines that will be connected to each other using VPN.

Copying and pasting from the Polkadot wiki won't implement Linux best practices like updating the operating system, setting up the firewall, running regular backups, etc.,
All these fall into the domain of Dotnix which either already does or will in future cover it by default, aiming to simplify running a secure Polkadot validator.

The system is scanned for CVEs using Vulnix; the public database is NVD
The operator can choose to be alerted via mail or Matrix

With Polkadot.js the frontend is meant.
The VPN is primarily used to connect Dotnix machines to each other enabling monitoring and allowing backups, e.g. to machines that are otherwise unreachable to the public Internet.

The actual deliverable is a Nix flake that exposes tooling to deploy Dotnix to generate images and deploy Dotnix to arbitrary targets like Docker or bare metal.
Docker is used only since it is the preferred way of testing for W3F.

All parts that make sense to be in Nixpkgs, will be upstreamed, the domain-specific parts will remain in Dotnix.
We have chosen a similar architecture to ethereum.nix or nix-bitcoin

Polkadot is built from source using andresilva's polkadot.nix flake

Secure validator mode is supported and active by default in the current release.

Session Key Management has been integrated as a part of our deliverables within our previous Grant.

We're going to maintain this project. A the very least we would need to follow the biannual release cycle of Nixpkgs stable in order to allow automatic updates of the system.
Within this scope, Sporyon will continue to cover the maintenance costs to ensure the security of Dotnix in the forceable future.

These items are planned for subsequent grants, although there is still work to be done upstream, particularly with SELINUX and CIS compliance.

In principle, SELinux and Secure boot are possible today and are planned for subsequent grants.
Regarding CIS compliance, as of today, a default Dotnix installation fails 46% of aquasecurity's linux-bench (which implements CIS Distribution Independent Linux Benchmark version 1.1.0).
As for FIDO2, we have not researched how it can be integrated best, it's not on the roadmap, yet.
If there are strong opinions about using FIDO2, we could consider prioritizing the implementation of this feature.

Hope this clarifies things a bit
I am happy to hear your thoughts

[PieWol]

Hey @Ra33it0 ,
before we start to discuss the application any further I would like to let you know that we only support one DOT address per application. Somehow your company would need to find a way to distribute the vested DOT internally. Otherwise the application looks good and I'll share it with the committee after this issue has been resolved. Thanks!

[PieWol]

Hey @Ra33it0 ,
as much as I like your work and agree with the goals of the application to further increase the security of nodes within the Polkadot ecosystem, I think it's crucial that you provide some evidence gathered from the ecosystem that node providers actually desire this project and want to use this. Especially as we are talking about a level 3 grant and having funded the MVP already.

[keeganquigley]

Hi @Ra33it0 your previous deliveries were all very highly rated, so I guess my main question is, are any validators currently utilizing Dotnix? What are your plans to help the tool gain traction? I see there haven't been any commits made in four months or so. How about the Nix community, have they given any feedback or engagement? Do you plan to publish the NixOS modules on the official community repo?

[Ra33it0]

Thank you @PieWol for your feedback. I completely agree that concrete evidence from the ecosystem is key to demonstrating the value of this project. We’re currently facing a bit of a "chicken and egg" problem with Dotnix, as it's still in an MVP phase and not feature complete yet, which makes it difficult to offer a strong selling point for administrators to transition their running validators to it at this stage.

We chose to first build an MVP to prove that Dotnix is feasible. In doing so, we implemented tests for every component, beyond the usual testing, to ensure quality and reliability. Regarding W3F's funding of the MVP, we intentionally made it a relatively small project to establish trust between W3F and Sporyon. We felt that it was important to first demonstrate that we could do good work, proving Dotnix's viability before pursuing further development.

Our intention was always to earn our "deed" and demonstrate that Dotnix works, and then, in subsequent grants, focus on adding unique selling points such as the vulnerability scanner and state management, which we believe will be particularly valuable to validator providers and administrators.

We don’t intend to monetize Dotnix but see it as a direct contribution of a high-quality product to the Dotsama ecosystem. That said, I’d appreciate your advice on how we can generate the necessary metrics to show interest from node providers. Should we consider splitting the grant into smaller portions, perhaps multiple Level 2 grants, if the Level 3 grant doesn’t seem suitable for the current state of the project? We initially chose a Level 3 grant as we believed it would be easier for both sides, allowing us to deliver more within each milestone.

Any advice or insights you could share on how we can best approach this challenge would be greatly appreciated!

[Ra33it0]

Hi @keeganquigley thanks for your questions!

Regarding your first question, we can't definitively say how many validators are currently utilizing Dotnix since anyone can download our repo or run a nix command to execute it locally. As of now, we don't have direct visibility into usage data, but we plan to gain more insight as we continue developing and promoting the tool.

Our main strategy to help Dotnix gain traction is to present it at upcoming ecosystem conferences such as Polkadot Decoded and NixCon 2025. However, we believe it only makes sense to present it publicly when it's feature complete — that is, when we can offer concrete unique selling points apart from simply using NixOS, like vulnerability scanning, state management, and integrated monitoring. We also hope to be mentioned in the upstream documentation, though we feel it might be a bit early for that, as we want to ensure that dotnix moved out of the MVP phase.
Regarding the Nix community we got positive feedback from our local nix community in Berlin and a similar project from the Bitcoin ecosystem was presented at NixCon2024 and got good feedback as well.
We plan to make it an official Nix community repo.

In the meantime, we've written a guide detailing how to create a Polkadot validator using Dotnix on a machine that only has a rescue shell. Using kexec, you can install NixOS and then Dotnix — a helpful approach for setting up validators on cost-effective servers, which are often available at a discounted price in the Hetzner server auction. We hope this tutorial helps make it easier for more users to get started with Dotnix and Polkadot validation.

Regarding the lack of commits in the last four months, our entire team has been involved in the planning and preparations for NixCon 2024, which has required a significant portion of our time. Additionally, it took W3F about a month to evaluate our last milestone, which further impacted our development pace. Given these factors, we chose to focus on testing our validator on Westend and preparing a detailed feature roadmap rather than pushing out new commits. We see Dotnix as a direct contribution to the Dotsama ecosystem, and we don’t intend to monetize it. Rather, we see it as a fundamentally better way of validating Polkadot by leveraging Nix primitives such as reproducibility, modularity, security, and ease of use. We believe this will add to the resilience of the Polkadot network in the long term.

We belive Web 3 should be a decentralized effort not just regarding clients but also regarding infrastructure like the validators of a network , if everything even critical infrastructure is heavily reliant on docker we are introducing a single point of failure which is missing the point of how we understand web3.
Other ecosystems like Bitcoin, Ethereum have realized this and are mitigating this fact using tools like NixBitcoin or Ethereum Nix we try to achieve the same for the Polkadot ecosystem with this Dotnix.

Thanks again for your questions, and we look forward to hear your thoughts.

[keeganquigley]

Thanks for your thorough answers @Ra33it0 much appreciated. Whether you want to break up the grant or not is entirely up to you, but it's possible you may find it easier to obtain three approvals than the five needed for the current structure.

Personally I'm willing to approve it as is, considering your previous work and the fact that it will remain as common good software. I think Nix makes a lot of sense for replicating validator configs. Nix expressions could be useful too. That's great that you plan to present it at upcoming conferences; once the features are in place I hope you'll be able to effectively market this tool to the validator community.

[semuelle]

Thanks for the updates and sorry for the long wait, @Ra33it0. Would you be able to collect more information on community interest with a level 2 grant? It sounds like this is not the case, and we have had a lot of projects simply falter after a grant because no community ever built around them, so I'm also a bit hesitant here.

[Lederstrumpf]

Hi @Ra33it0,

Not a W3F Grants member, but commenting as someone personally using Nix for operating Polkadot/Kusama validators - so given the requests for community feedback from multiple W3F Grants team members, I share my 2¢ below:

At a high level: I'm glad that you want to foster easy & secure operation of Substrate-based validators using Nix. In particular, it would complement @andresilva's existing Polkadot flake & nixpkgs integration well, which I also happily use for building binaries.

Simultaneously, I reckon @ajk-code and @PieWol raise some strong arguments regarding the specifics of the proposal.

@ajk-code: In general, to me it sounds like an in-house deployment project, not something for a wider audience

This resonates with me personally: I'd probably not be using your flake directly, both for trust-minization (for instance, I wouldn't use the nix cache for polkadot.nix outputs, but use polkadot.nix to build directly on my system), but also since a lot of your other envisioned integrations are less applicable to my validator operation.
For instance, your polkadot-validator service is a component I personally reckon is useful for other Nix users to extract, but probably not use directly within your other integration:
https://github.com/sporyon/dotnix-core/blob/main/nixosModules/polkadot-validator.nix#L328-L372

@PieWol: I think it's crucial that you provide some evidence gathered from the ecosystem that node providers are actually desire this project and want to use this.

Using public telemetry as a proxy (w3f telemetry should provide more insight), less than 12% of validator use distributions other than Debian/Ubuntu for running validators.
https://telemetry.polkadot.io/#stats/0x91b171bb158e2d3848fa23a9f1c25182fb8e20313b2c1eb49219da7a70ce90c3
https://telemetry.polkadot.io/#stats/0xb0a8d493285c2df73290dfb7e61f870f17b41801197a149ca93654499ea3dafe

Only a subset of these are possibly running NixOS. Of course some other validators may also be using Nix as a package manager only, but I think this is also a small set. On the other hand, since your Dotnix deliverable would make Nix deployment easier (especially if used only via the package manager, which doesn't have the additional operational onramp of setting up NixOS), this set could grow quite a bit.

@Ra33ito: In principle, SELinux and Secure boot are possible today and are planned for subsequent grants.

I personally would encourage moving this into the current grant application since imo it's far more beneficial than some other features you're proposing like cloud deployment and polkadot.js exposure.

I've also left some inline comments in the review comments below. In particular, note that I'd strip any planned features for Hetzner.

[Lederstrumpf]

Definitely not just a personal nit:

As of '22 (as anyone who migrated remembers), Hetzner's practially banned all "cryptomining" activity, which broadly includes staking:
https://www.hetzner.com/legal/dedicated-server/
https://www.hetzner.com/legal/cloud-server/

What would be the use case for Hetzner templates then? Unless you can make a compelling argument for this, I would strip this from the deliverables.

[Lederstrumpf]

what if there are issues with the p2p-vpn layer and the node becomes inaccessible? Is the only option then KVM? Personally, I'd have concerns about this.

[Lederstrumpf]

A Polkadot/Kusama node is hardware demanding, and this requirement will increase ever more once JAM hits the floor.

For good reason, the Polkadot Wiki recommends validators to run on bare metal:
https://wiki.polkadot.network/docs/maintain-guides-secure-validator#validators

While I acknowledge that making operation easier on cloud providers has its benefits, this also adds a degree of stickiness that would make it harder to convince validators to do the right thing™.

So my personal recommendation would be to focus on making bare-metal operation easier.

[Lederstrumpf]

What state migration do you mean? Hopefully not session keys! Is this just for database snapshots & perhaps node keys?

[Ra33it0]

@semuelle Thank you for your message.
Our focus is on growing the Polkadot community and improving the validator landscape, not creating a separate Dotnix community. We’re building Dotnix because we believe Nix is an adequate tool for deploying Polkadot Validators. In regard to your question if we are able to build the project on a Lvl 2 Grant, a Level 2 grant on its own, without the possibility of subsequent grants, wouldn’t allow us to build the project sustainably. If you're suggesting splitting our grant into multiple Level 2 grants to further strengthen our track record, we're open to that idea.
We aim to simplify the deployment of secure Polkadot validators by offering an alternative method that reduces single points of failure, supports decentralization, and contributes to the overall growth of the Polkadot ecosystem.
We are under the impression our goals align perfectly with the goals of the W3F.

[PieWol]

Hey @Ra33it0,
sadly your application wasn't able to gather enough approval from the Grants Committee. This is due to all the issues and concerns raised in this thread. E.g. for me personally there wasn't enough evidence that node operators are interested in the tech. Given that we already have funded the MVP I don't see why such evidence would be impossible to gather.

You could consider asking the Polkadot treasury for funding so that you can pursue this project even without another round of financial support by the Web3 Foundation's Grants Program.

We wish you all the best for your future endeavors and hope that you keep on building in the Polkadot Ecosystem. Thanks again for your application. Feel free to reach out if you have any questions.

Best

[Ra33it0]

Hi @Ra33it0,

Not a W3F Grants member, but commenting as someone personally using Nix for operating Polkadot/Kusama validators - so given the requests for community feedback from multiple W3F Grants team members, I share my 2¢ below:

At a high level: I'm glad that you want to foster easy & secure operation of Substrate-based validators using Nix. In particular, it would complement @andresilva's existing Polkadot flake & nixpkgs integration well, which I also happily use for building binaries.

Simultaneously, I reckon @ajk-code and @PieWol raise some strong arguments regarding the specifics of the proposal.

@ajk-code: In general, to me it sounds like an in-house deployment project, not something for a wider audience

This resonates with me personally: I'd probably not be using your flake directly, both for trust-minization (for instance, I wouldn't use the nix cache for polkadot.nix outputs, but use polkadot.nix to build directly on my system), but also since a lot of your other envisioned integrations are less applicable to my validator operation. For instance, your polkadot-validator service is a component I personally reckon is useful for other Nix users to extract, but probably not use directly within your other integration: https://github.com/sporyon/dotnix-core/blob/main/nixosModules/polkadot-validator.nix#L328-L372

@PieWol: I think it's crucial that you provide some evidence gathered from the ecosystem that node providers are actually desire this project and want to use this.

Using public telemetry as a proxy (w3f telemetry should provide more insight), less than 12% of validator use distributions other than Debian/Ubuntu for running validators. https://telemetry.polkadot.io/#stats/0x91b171bb158e2d3848fa23a9f1c25182fb8e20313b2c1eb49219da7a70ce90c3 https://telemetry.polkadot.io/#stats/0xb0a8d493285c2df73290dfb7e61f870f17b41801197a149ca93654499ea3dafe

Only a subset of these are possibly running NixOS. Of course some other validators may also be using Nix as a package manager only, but I think this is also a small set. On the other hand, since your Dotnix deliverable would make Nix deployment easier (especially if used only via the package manager, which doesn't have the additional operational onramp of setting up NixOS), this set could grow quite a bit.

@Ra33ito: In principle, SELinux and Secure boot are possible today and are planned for subsequent grants.

I personally would encourage moving this into the current grant application since imo it's far more beneficial than some other features you're proposing like cloud deployment and polkadot.js exposure.

I've also left some inline comments in the review comments below. In particular, note that I'd strip any planned features for Hetzner.

Thank you, @Lederstrumpf for your valuable feedback.
We chose the same architecture as Ethereum nix to deliver a coherent package, but we welcome you just using what you need from dotnix
The idea is to lower the entry barrier to use Polkadot with nix some seasoned nix dotnix might serve as s a template for their own custom configuration.
For us, dotnix serves as base to be able to deliver a coherent, fully functional setup for integrating services.

We could prioritize SELinux and secure boot, your recommendation makes sense to us
Our intent was generic deployment template to build common ground, but we see your point with Hetzner and agree with your point that it seems like an unintentional recommendation to use Hetzner so we would to strip Hetzner from the deliverable and change it and focus only on self-hosted solutions.
The idea behind using cloud providers was to lower the barrier to validate Polkadot and onboard people to Polkadot more easily, but we wholeheartedly agree to that focusing on self-hosting Polkadot seems like the better way to continue

If the p2p VPN layer becomes inaccessible, then yes, it can't be used anymore.
This could be easily mitigated by allowing ssh over the Internet

Exactly just database snapshots and node keys