Rust full reflection #8102

candysonya · 2023-09-26T11:32:34Z

(My teammate dextero has kindly helped to review the PR and left a lot of insightful comments. Unfortunately they are not a member of the community and they cannot approve it. Currently all the comments have been resolved and it would be highly appreciated if it could be reviewed by the community because the reflection feature in Rust is a blocker for our team to use Flatbuffers!)

Compared to the C++ implementation of the reflection feature,

It includes:

Helper functions to access schema
Full reflection - getters for scalar/string/struct/table/vector types
Full reflection - setters for scalar/string types
Verification of buffer
Tests for above

It doesn't include:

Getter for VectorOfAny (Vector without element type specified)
Setter from another buffer, and converting buffer to builder

TODOs on C++ side are retained.

dextero

A bunch of small things. Feel free to ignore nits if they don't seem worthwhile.

Great work overall!

rust/flatbuffers/src/vector.rs

rust/reflection/src/lib.rs

tests/rust_reflection_test/src/main.rs

adsnaider · 2024-01-05T15:17:05Z

@candysonya @dextero We are using flatbuffers at work and I have a small patch that allows us to have zero-allocation of flatbuffer messages. I noticed that this change makes all the &'static str into String. Is this strictly necessary? Could it be Cow<'static, str>?

candysonya · 2024-01-08T15:47:45Z

@candysonya @dextero We are using flatbuffers at work and I have a small patch that allows us to have zero-allocation of flatbuffer messages. I noticed that this change makes all the &'static str into String. Is this strictly necessary? Could it be Cow<'static, str>?

Thanks for pointing it out! I've just updated the change with Cow<'static, str>.

adsnaider

Thank you for making the changes to Cow<'static, str>. This is an awesome patch set and I'm really excited to have reflection! My comments are a few cases where you used .to_string() but I believe that .into() should work instead to make the value into a Cow and not a String. I could be wrong though and in that case feel free to ignore my comments

rust/flatbuffers/src/verifier.rs

rust/flatbuffers/src/vector.rs

rust/reflection/src/lib.rs

rust/reflection/src/reflection_verifier.rs

rust/reflection/Cargo.toml

rust/reflection/src/reflection_verifier.rs

tests/rust_reflection_test/src/lib.rs

candysonya · 2024-01-17T17:48:39Z

Thank you for making the changes to Cow<'static, str>. This is an awesome patch set and I'm really excited to have reflection! My comments are a few cases where you used .to_string() but I believe that .into() should work instead to make the value into a Cow and not a String. I could be wrong though and in that case feel free to ignore my comments

Thanks so much for spotting that!

github-actions · 2024-09-24T20:34:36Z

This pull request is stale because it has been open 6 months with no activity. Please comment or label not-stale, or this will be closed in 14 days.

brt-adam-snaider · 2024-09-24T20:59:39Z

I do care about this feature. I may pick it up if @candysonya is unavailable.

brt-adam-snaider · 2024-09-24T21:00:10Z

not-stale

candysonya · 2024-09-24T21:05:16Z

Hi @brt-adam-snaider , yes please feel free to pick it up. Do you already have a plan?

candysonya · 2024-12-04T12:43:22Z

Hi @brt-adam-snaider , I'm considering picking it up. Before that, may I know if you're already working on this?

brt-adam-snaider · 2024-12-04T15:15:38Z

I haven't picked it up yet. Have at it :)

dextero

LGTM, thank you for following through on this!

We'll need an owner approval, I can't grant one unfortunately.

dbaileychess · 2025-01-10T04:18:38Z

Some CI issues with Rust. Can you take a look? It should auto run now next time you push to the PR.

dextero

I left a few nits, mostly about how some errors could be avoided by reorganizing the code a bit. LGTM overall, great work!

dextero · 2025-01-10T14:47:41Z

rust/reflection/src/safe_buffer.rs

+pub struct SafeBuffer<'a> {
+    buf: &'a [u8],
+    schema: &'a Schema<'a>,
+    buf_loc_to_obj_idx: RwLock<HashMap<usize, i32>>,


Do we really need the RwLock here? IIUC:

the hashmap is created once in SafeBuffer::new and then never modified again,

the buffer the map refers to never changes either (we only have an & reference to it)

This makes me think we could use a plain HashMap<usize, i32> here instead, and remove the RwLockPoisonError.

dextero · 2025-01-10T15:03:47Z

rust/reflection/src/safe_buffer.rs

+        unsafe { get_any_root(self.buf) }
+    }
+
+    /// Gets all the fields the [table] has. Returns error if the [table] doesn't match the buffer.


Nit: the "table must match the buffer" constraint sounds like something that would be awesome to check automatically rather than having the user ensure it.

Would it be possible to have something like SafeTable (stub name), that keeps a reference to the buffer it refers to? This could give us usage like

let buffer: SafeBuffer = SafeBuffer::new(&buf, *schema)?; let root: SafeTable = buffer.get_root(); root.get_all_fields(); // always correct because SafeBuffer only accesses the matching buffer

And there's even a chance we could get rid of InvalidTableOrStruct error!

dextero · 2025-01-10T15:11:19Z

rust/reflection/src/safe_buffer.rs

+        }
+    }
+
+    /// Returns the value of any table field as a string, regardless of what type it is. Returns empty string if the field is not set. Returns error if


Nit: if this is public API, it would be nice to point out that the value gets stringified and not reinterpreted as a string, so for example an int (1234) becomes "1234" and not "\x01\x02\x03\x04" (+/- endianness)

dextero · 2025-01-10T15:18:11Z

tests/rust_reflection_test/src/lib.rs

+    assert!(value.is_err());
+    assert_eq!(
+        value.unwrap_err().to_string(),
+        "Table or Struct doesn't belong to the buffer"


Nit: we're using thiserror so we could look for a specific variant of the error enum, to not have to worry about the string wording changing:

assert_eq!(matches!(value.unwrap_err(), FlatbufferError::InvalidTableOrStruct))

dextero · 2025-01-10T15:19:06Z

rust/reflection/src/lib.rs

+    #[error("RwLock read/write error: {0}")]
+    RwLockPoisonError(String),
+    #[error("Table or Struct doesn't belong to the buffer")]
+    InvalidTableOrStruct(),


Nit: if we don't need any data inside, we can skip the () and make it just InvalidTableOrStruct, here.

…er/float/string

It verifies buffer against schema during construction and provides all the unsafe getters in lib.rs in a safe way

github-actions bot added the rust label Sep 26, 2023

candysonya force-pushed the reflection branch 2 times, most recently from 3ea581d to 7e5b92b Compare September 27, 2023 11:43

dextero reviewed Sep 27, 2023

View reviewed changes

candysonya changed the title ~~[Draft] Rust full reflection~~ Rust full reflection Dec 20, 2023

candysonya marked this pull request as ready for review December 20, 2023 10:57

candysonya mentioned this pull request Dec 20, 2023

Feature request: Rust reflection API #8033

Open

candysonya force-pushed the reflection branch 5 times, most recently from 0ee84d1 to 6868cb3 Compare December 20, 2023 15:42

adsnaider reviewed Jan 9, 2024

View reviewed changes

rust/flatbuffers/src/verifier.rs Outdated Show resolved Hide resolved

rust/flatbuffers/src/verifier.rs Outdated Show resolved Hide resolved

rust/flatbuffers/src/verifier.rs Outdated Show resolved Hide resolved

dextero reviewed Jan 9, 2024

View reviewed changes

candysonya force-pushed the reflection branch from 73bd124 to 913d257 Compare January 17, 2024 17:44

github-actions bot added the swift label Jan 21, 2024

candysonya force-pushed the reflection branch from c9465ba to 88b2f0d Compare January 21, 2024 13:40

github-actions bot removed the swift label Jan 21, 2024

candysonya force-pushed the reflection branch from 88b2f0d to f2f7f93 Compare January 21, 2024 13:45

candysonya force-pushed the reflection branch from cc95976 to 358d6e5 Compare February 4, 2024 21:42

github-actions bot added the stale label Sep 24, 2024

github-actions bot removed the stale label Sep 25, 2024

candysonya force-pushed the reflection branch 3 times, most recently from 7d2ea1f to 6ed63ee Compare December 9, 2024 23:27

dextero reviewed Dec 10, 2024

View reviewed changes

candysonya force-pushed the reflection branch 2 times, most recently from ad5edac to d3f36c9 Compare January 6, 2025 13:10

candysonya force-pushed the reflection branch from f3b65ea to ceb2218 Compare January 10, 2025 15:06

dextero reviewed Jan 10, 2025

View reviewed changes

candysonya added 12 commits January 10, 2025 23:38

#Rust Create a crate for reflection

ea19d65

#Rust Add a crate for reflection tests and helper to access schema

054544d

#Rust Get root table of a buffer and access field with schema

c727289

#Rust Add 'Struct' struct and corresponding getter

ee33b86

#Rust Add functions of getting any table/struct field value as integ…

7892dc6

…er/float/string

#Rust Add setters for scalar fields

c951a93

#Rust Add setter for string fields

2f6bb1a

#Rust Add getter for Table/Vector fields

6c30257

#Rust Add buffer verification

6b160de

Add a 'SafeBuffer' struct which provides safe methods for reflection

f36c036

It verifies buffer against schema during construction and provides all the unsafe getters in lib.rs in a safe way

Fix Rust CI failure on using 'std' module

e2b998f

Fix Rust CI failure on using 'std' module

2986c39

candysonya force-pushed the reflection branch from 0b5d56c to 2986c39 Compare January 10, 2025 22:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rust full reflection #8102

Rust full reflection #8102

candysonya commented Sep 26, 2023 •

edited

Loading

dextero left a comment

adsnaider commented Jan 5, 2024

candysonya commented Jan 8, 2024

adsnaider left a comment

candysonya commented Jan 17, 2024

github-actions bot commented Sep 24, 2024

brt-adam-snaider commented Sep 24, 2024

brt-adam-snaider commented Sep 24, 2024

candysonya commented Sep 24, 2024

candysonya commented Dec 4, 2024

brt-adam-snaider commented Dec 4, 2024

dextero left a comment

dbaileychess commented Jan 10, 2025

dextero left a comment

dextero Jan 10, 2025

dextero Jan 10, 2025

dextero Jan 10, 2025

dextero Jan 10, 2025

dextero Jan 10, 2025

Rust full reflection #8102

Are you sure you want to change the base?

Rust full reflection #8102

Conversation

candysonya commented Sep 26, 2023 • edited Loading

dextero left a comment

Choose a reason for hiding this comment

adsnaider commented Jan 5, 2024

candysonya commented Jan 8, 2024

adsnaider left a comment

Choose a reason for hiding this comment

candysonya commented Jan 17, 2024

github-actions bot commented Sep 24, 2024

brt-adam-snaider commented Sep 24, 2024

brt-adam-snaider commented Sep 24, 2024

candysonya commented Sep 24, 2024

candysonya commented Dec 4, 2024

brt-adam-snaider commented Dec 4, 2024

dextero left a comment

Choose a reason for hiding this comment

dbaileychess commented Jan 10, 2025

dextero left a comment

Choose a reason for hiding this comment

dextero Jan 10, 2025

Choose a reason for hiding this comment

dextero Jan 10, 2025

Choose a reason for hiding this comment

dextero Jan 10, 2025

Choose a reason for hiding this comment

dextero Jan 10, 2025

Choose a reason for hiding this comment

dextero Jan 10, 2025

Choose a reason for hiding this comment

candysonya commented Sep 26, 2023 •

edited

Loading