Allow calling from more than one language at the same time

[jhugman]

I'd like to be able to initialize and use Rust from multiple languages at the same time [in the same process].

Example use case:

From React-Native Javascript, call into Rust which then calls out to Kotlin.

The C ABI should remain the same for the foreign language, so calling in to Rust would need no modifications.

I think the only thing to need to change would be adding an index parameter to callback registration and UniffiRustFutureContinuationCallback, so that the callback knows which vtable to call into. This would require work on both foreign and Rust sides.

I don't think this would be on Mozilla's roadmap, so am quite happy to do the work.

Is this a feature you'd be willing to land?

[mhammond]

It sounds interesting and not too intrusive, so I don't see why not!

[jhugman]

Ok, so I got callbacks working. This should be just introducing a lang_index, and passing it as another value in the FfiConverterCallback, making it an FfiConverterRustBuffer.

Futures doesn't look too complicated: it looks like it should just work unchanged.

Foreign traits are looking a bit more problematic, and may need some wider changes: because they're implemented in Rust or foreign side, and you can call them like an object, they need to be able to call free and clone_pointer. Two possible approaches come to mind:

1. add a variant of the FfiType to account for the differences between a RustArcPtr and RustArcPtr with a source language.
2. modify the associated value for FfiType::RustArcPtr(_) from a seemingly unused String to a new ArcType enum with variants Raw and WithSource.
3. generate extra constructors, clonePointer and free methods that accept RustBuffers instead of Pointers.

My preference I think would be 2, but I'm wondering if this may be pulling on a ball of string to something "too intrusive". What do you think? Should I keep going? Any advice @mhammond ?

[mhammond]

Can you put up a draft PR to help make it concrete? @bendk understands all this stuff the best by far though, so I'm happy to look but his feedback will be more valuable.

[bendk]

I think a draft PR would be great. One thing we've been talking about is trying to improve the internals docs, which are really out of date at this point. Maybe you could add something there that explains the issue and how you want to solve it. Bonus points for mermaid diagrams (BTW, I'm hoping to update the lift/lower ones soon and I'm planning on stealing the diagram from your recent JS talk for that).

My initial take is that this could be another reason to dust off my handles PR. If we used handles to identify these things rather than raw pointers, then I think we could dedicate a few bits for the language index.

• When foreign language X sends Rust it's vtable, Rust stores it in an array and returns the index back to the foreign language.
• Whenever that foreign language generates a handle, it sets the language index portion of that handle to the value it got back from Rust
• When Rust wants to call a callback/trait interface method, it uses the language index to look up which vtable to use.

Seems like it could work, but I haven't thought it through that much. I don't think these arrays would need to be very big, 4 languages/2-bits seems fine to me maybe 8 languages/3 bits if we want lots of room.

[bendk]

... and now that I think of it, this feels like an extension of something I already want to do. I was hoping to have a bit that distinguished Rust handles to trait interfaces from foreign handles. That would make passing them back and forth more efficient, currently we always need to wrap the object when passing them over the FFI. So a trait interface object is passed from Rust to the foreign language and back to Rust then when Rust uses it, it has to make a call into the foreign side which then makes a call back to Rust.

Going back to this issue, maybe rather than a single bit, we could use a like 3 bits to identify the source of the handle. 0 is Rust, 1 is foreign language A, 2 is foreign language B, etc.

And now I'm wondering about one more corner case, what if foreign language A ends up with a handle to a trait interface implemented by foreign language B? One option would be to piggyback a call through Rust. Alternatively, maybe there's some where where foreign language A could get a hold of the other vtable and make the call directly.

[jhugman]

I've put up my draft PR (jhugman#2) which is a straw fox more than anything, especially given your thoughts above.

It's Kotlin only right now, and uses a RustBuffer to take the callback handle/foreign trait pointer and i32 a language index. I think using the a few bits of the handle would mean much of the plumbing disappears to the edges and definitely make it more efficient than making a new RustBuffer.

I'll try and write a self-review in the week.

[jhugman]

Self-review done. /cc @bendk