[Epic] Rust in Defra

[AndrewSisley]

A ticket to note down any notable thoughts/findings when thinking about migrating from Golang to Rust, as I've had this thought a few times and am duplicating effort by not writing stuff down and then re-researching/thinking it.

Notes:

• John has mentioned maybe implementing defra in multiple languages a couple of times
  .- I am somewhat skeptical of the the cost-benefit of this
  .- Would need to ensure very strong inter-op guarantees RE persisted data (a very good thing in itself IMO)
• cgo (Rust from Go) comes with a host of problems (nice blog here https://dave.cheney.net/2016/01/18/cgo-is-not-go)
  .- notable performance cost (use as sparingly as possible)
  .- large cross-compilation issues
  .- significantly more complex build
• Shouldn't be too difficult to preserve integration tests in Go (using cgo) in the short-medium term
  .- Will likely cause significant hassle if we expand our build system to cover more build targets though
• Go from Rust is very difficult and poorly documented
  .- unless maybe we compile rust to wasm (Golang's wasm support isnt great either, but still likely to be much easier)
• No significant rust GQL libraries, would probably have to make our own (easy for most stuff IMO, harder probably for subscriptions?)
• Not yet looked into ipfs/datastore Rust equivalents, could be a significant loss
• Notable gains
  .- Massive boost in terms of build targets/options (wasm, no-std-lib, features, etc)
  .- Rust has far better language inter-op, should make it much much easier to write defra wrappers in other langs
  .- Rust (rust-docs, safer lang, no GC, etc)
• Other notable losses/costs
  .- Dev-time to migrate
  .- Significant code complexity gain when blending langs (and to build system)
  .- Integration test risks

Update 2024-04-29:

Copied from discord thread dev-db.Rust transition:

some benchmarks to compare the original version of connor and the new rust-based. Here are the results:

3 scenarios:

Database with 100 Documents (Simple Query)

Golang: Average elapsed time of approximately 5.68 ms.
Rust via ABI: Average elapsed time of approximately 8.10 ms.

Database with 6000 Documents (Simple Query)

Golang: Average elapsed time of approximately 90.21 ms.
Rust via ABI: Average elapsed time of approximately 147.96 ms.

Database with 6000 Documents (Complex Query)

Golang: Elapsed times ranged from 86.07 ms to 105.61 ms; average approximately 95.48 ms.
Rust via ABI: Elapsed times ranged from 269.29 ms to 307.52 ms; average approximately 282.38 ms.

Update 2024-12-08:

The team is current in agreement that Rust would be a better language overall for the Defra repo and there is significant enthusiasm for us getting to that state.

However the team also believes that the opportunity cost of allocating the significant amount of development time to get enough of the codebase converted to Rust in order to have a positive impact on users is currently too large, and as such the ticket has been put on hold.

Migrating a small number of packages would have a significant detrimental effect on performance, and would introduce a significant maintenance burden to the repo - the effects of both depends largely on the number of points where data flows between the Go and Rust runtimes, so if resuming this ticket, try and minimize this.

Tasks

Preview Give feedback

1. Convert a Go package into Rust #2508
   refactor +1
   

[orpheuslummis]

Can you please clarify what you mean by "Rust from Go" and "Go from Rust"?

[fredcarle]

Go from Rust is very difficult and poorly documented

and

Rust has far better language inter-op, should make it much much easier to write defra wrappers in other langs.

seem contradictory. Am I getting this wrong?

[AndrewSisley]

Can you please clarify what you mean by "Rust from Go" and "Go from Rust"?

Calling Go from Rust, and calling Rust from Go

... seem contradictory. Am I getting this wrong?

It is only contradictory if you ignore the many other languages out there, calling Rust from most languages is far easier than calling Go (and calling out from Rust)

EDIT: To expand, I consider Golang's inter-op roughly on par with the JVM (totally crap), whereas Rust is just a bit behind C's (about as good as it gets), with most other compiled langs falling between the two (never tried calling into an interpreted lang, so I can't comment on those)

[orpheuslummis]

Any ideas for a minimal POC, a first step of Rust integration?

[AndrewSisley]

Any ideas for a minimal POC, a first step of Rust integration?

You'd really want to minimize the number of calls between the two langs, so a minimal POC could still be quite large. Would probably introduce Rust at the bottom and only have Go=>Rust calls (as that is easier and makes more sense IMO), but then it takes ages to get the build/inter-op benefits, and you'll likely hit the difficult issues RE using datastores etc.

The gql generate package might be the easiest, but would have the least gains too (dont really care about performance there, or interop atm).

Would be quite a tricky thing to do, but it is very nice that it is getting my head somewhat more focused on the interop side of things

[mahmudsudo]

can i take on this ?

[AndrewSisley]

can i take on this ?

Hey @mahmudsudo welcome to the codebase!

Sorry about the bother, this ticket has not been updated inline with the team's current thinking. I'll update the description shortly.

We decided a while back that whilst we believe Rust would be a better language to work in, we do not wish to allocate resources to converting to Rust, or maintaining a bilingual codebase at this time - the opportunity cost is too great for us to take on at the moment. If you are interested in making a serious go at converting a significant chunk of the repo I'm sure the team will be very happy to re-open this discussion.

You are super welcome to have a go at converting some packages if you like - CONTRIBUTING.md should give you a quick overview over how to work with the codebase and I'm very happy to take any questions :)

I also suggest joining our Discord if you haven't done so already (easiest way to get in touch with us rapidly).

Please do be aware that any changes that you may make towards this ticket will be subject to notable debate within the Source database development team as to whether we want them merged or not, regardless of the quality of your work, due to the previously mentioned reluctance to support a bilingual codebase.

The internal/connor package has already been converted (stale). The next best package to convert is likely internal/request/graphql/schema. Although you are of course super free to try and start somewhere else.