feat(rust): Allow setting custom object_store client/cloud options

[PrettyWood]

We currently use Polars in our Rust web service and need to support custom certificates in our file connector.
While this functionality already exists in object_store via the ClientOptions struct (which allows setting root certificates through the with_root_certificate method), Polars does not yet expose this struct.
This PR aims to add this capability to Polars.

Polars CloudOptions implements PartialEq, Eq and Hash, which is not the case of ClientOptions in object_store. Furthermore the defaults in polars for ClientOptions were not the same as object_store
(see get_client_options function)

So I decided to create a new type PlClientOptions with those defaults and only add what we want to set: the 3 previous fields (pure refactoring) + certificates (feature)

Some questions:

• is the overall approach good for you?
• is it ok if we ignore those certificates for PartialEq, Eq and Hash? If yes do you prefer the current approach with custom implems or the use of derivative crate with #[derivative(PartialEq="ignore")], which is more explicit and readable but adds a new dependency.
• should I simplify #[cfg(any(feature = "aws", feature = "gcp", feature = "azure", feature = "http"))] with #[cfg(feature = "cloud")]?

We can test this PR on AWS but not for GCP and Azure but since the code is the same, this test should be enough!

Thank you so much :)

[codecov]

Codecov Report

Attention: Patch coverage is 72.22222% with 15 lines in your changes missing coverage. Please review.

Project coverage is 79.88%. Comparing base (96a2d01) to head (be39376).
Report is 106 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/polars-io/src/cloud/client_options.rs	74.35%	10 Missing ⚠️
crates/polars-io/src/cloud/options.rs	61.53%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #21007      +/-   ##
==========================================
+ Coverage   79.34%   79.88%   +0.53%     
==========================================
  Files        1579     1594      +15     
  Lines      224319   227689    +3370     
  Branches     2573     2600      +27     
==========================================
+ Hits       177976   181879    +3903     
+ Misses      45755    45213     -542     
- Partials      588      597       +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[alexander-beedie]

@nameexhaustion: Any thoughts on this one? I imagine you may already have an approach in mind, or have an opinion how it might best integrate with all the (very useful!) related work you've been doing :)

[nameexhaustion]

I'm ok with having this PR. It is relatively simple, and currently it is not possible to set these options from the Rust side.

is it ok if we ignore those certificates for PartialEq, Eq and Hash

We should ensure the Hash/PartialEq take all fields into account, otherwise we may end up with caching bugs. If the underlying struct does not support this, I can recommend taking an approach similar to what we do with PlCredentialProvider, where we wrap the field in an Arc<> and take the pointer value.

[nameexhaustion]

Can you add a note above:

/// Note: This is mainly used by Rust users. Python client options go through `CloudConfig`

[PrettyWood]

@nameexhaustion Thank you so much for your valuable remarks! 🙏 Just pushed a new commit. Feedback welcome

[lukapeschke]

Just a small remark from me, thank you!

[lukapeschke]

Since you're taking ownership of pl_opts, could you skip the .clone() ?

Suggested change

	for certificate in pl_opts.root_certificates.0.iter() {
	opts = opts.with_root_certificate(certificate.clone());
	}
	for certificate in pl_opts.root_certificates.0 {
	opts = opts.with_root_certificate(certificate);
	}

[PrettyWood]

I can't do what you're suggesting because it's behind an Arc<…>, so I have to use .0.iter(). This gives me a reference to Certificate, which does not implement Copy, so I have to clone it.

[lukapeschke]

Ah all right, nevermind

[nameexhaustion]

Instead of serde(skip), can you derive Serialize/Deserialize for the ClientOptions? If the Certificate type doesn't allow for serialization, you can manually implement serde for RootCertificates:

• In Serialize, check that self.0 is empty in Serialize, otherwise panic explaining that it cannot be serialized
• In Deserialize, just return an empty vec.