Are we sure that removing dict_id is not removing important functionality?

[andygrove]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

I started upgrading Comet to use the latest DataFusion/Arrow and saw that an effort is being made to remove dict_id from arrow-rs Field.

It seems odd that arrow-java has dict_id as part of its field representation, but we plan to remove it from arrow-rs.

I am not 100% sure but this may be problematic for Comet since we seem to rely on the dictionary id when passing arrays between Java and Rust. I did not write this code and am not yet familiar with it, but in one example, in Rust we have:

    let mut dict_id = 0;
    let fields = arrays
        .iter()
        .enumerate()
        .map(|(i, array)| match array.data_type() {
            DataType::Dictionary(_, _) => {
                let field = Field::new_dict(
                    format!("c{}", i),
                    array.data_type().clone(),
                    true,
                    dict_id,
                    false,
                );
                dict_id += 1;
                field
            }
            _ => Field::new(format!("c{}", i), array.data_type().clone(), true),
        })

and then in Java:

DictionaryEncoding dictionaryEncoding = vector.getField().getDictionary();
Dictionary dictionary = dictionaryProvider.lookup(dictionaryEncoding.getId());

It will take me some time to get up to speed with how we are using this in Comet, and to determine whether it is needed or not.

Describe the solution you'd like

I would like more discussion on this before the field is removed.

Describe alternatives you've considered

Additional context

[andygrove]

@tustvold @alamb @brancz fyi

[tustvold]

In order to keep discussion in one place I am going to merge this with #5981