(fix): extension array indexers

[ilan-gold]

Identical to kmuehlbauer#1 - probably not very helpful in terms of changes since https://github.com/kmuehlbauer/xarray/tree/any-time-resolution-2 contains most of it....

• Closes #Failure in pandas TestDataFrameToXArray.test_to_xarray_index_types #9661
• Tests added
• User visible changes (including notable bug fixes) are documented in whats-new.rst
• New functions/methods are listed in api.rst

[ilan-gold]

Great @kmuehlbauer - I want the maintainers to look at the MyPy. I could in theory fix it, but I would basically be guessing at what their wishes are for the classes' return types.

[dcherian]

is this needed? Why would someone call np.array with an extension dtype, and then expect it to get translated to a numpy dtype?

[ilan-gold]

This is for internal usage, otherwise I wouldn't have added it. I can delete the line and then see what happens, and then comment.

[ilan-gold]

@dcherian This class is basically an internal adapter so anything that asks for its data in numpy form will call this. Things like repr, subtraction, and calling .values on an xarray object are a few examples

[dcherian]

i understand that. Why doesn't the last line (return super().__array__(dtype, copy=copy)) "just" handle this?

[ilan-gold]

Hh, good point, yes, it's unnecessary. Fixed.

[dcherian]

same here. Why is this needed?

[dcherian]

this is amazing, it enables IntervalIndex indexing now.

cc @benbovy

[dcherian]

@Illviljan or @headtr1ck can you take a look at the typing failure please

[Illviljan]

pd.api.types.is_extension_array_dtype(data_old.dtype) does not imply data_old is an extension array.
You probably should use some kind of isinstance-check to be able to use .array.

I haven't used extension arrays myself that much, why can't a simple np.asarray(data_old) be used?

[ilan-gold]

Seems I was able to resolve both comments with no impact. I think pd.api.types.is_extension_array_dtype is correct as from the docs: https://pandas.pydata.org/docs/reference/api/pandas.api.types.is_extension_array_dtype.html:

This checks whether an object implements the pandas extension array interface. In pandas, this includes:

As for the to_numpy I think I just forgot that the incoming object has a __array__ implementation

[dcherian]

So get_duck_array should handle this by possibly converting to a numpy array. Is that not desired? It is ambiguous, to me at least, if we consider ExtensionArrays a "duck array"

[ilan-gold]

Right, at the moment get_duck_array on the PandasIndexingAdapter returns a numpy array, and the docstring states:

The Variable's data as an array. The underlying array type
        (e.g. dask, sparse, pint) is preserved.

which means we probably want to conserve the pandas array typing. But, maybe the implementation of get_duck_array should be updated

[ilan-gold]

@dcherian I see how deep the issue goes re: typing. I am not sure if you want the extension arrays to satisfy this constraint, but it would make the typing on NamedArray a bit more coherent (see 459f56b and then the subsequent volley of failures as a result)

[ilan-gold]

For now I've just added a few ignores. I think this issue is a bit separate since it doesn't actually affect the runtime behavior

[Illviljan]

A surprise to me is that Extension arrays doesn't implement __array_function__.
So there's no promise that np.mean(extension_array) will work. Now numpy has been clever and tries extension_array.mean before attempting it, so maybe a non-issue in practice?

[ilan-gold]

So there's no promise that np.mean(extension_array) will work

Right, nor should it for the number of extension array types there are (what's the mean of a categorical? of an interval?)

Now numpy has been clever and tries extension_array.mean before attempting it, so maybe a non-issue in practice?

We have a wrapper around extension arrays that does implement __array_function__ so that might be why they fulfill the NamedArray variable. I can look into loosening the dtype restrictions on NamedArray and that might help

[ilan-gold]

I think part of the reason this is leaking is that as_compatible_data returns a generic T_DuckArray which has only an Any bound (and Variable allows its data initialization argument to be T_DuckArray). So that's why nothing has complained before:

xarray/xarray/core/variable.py

Lines 336 to 366 in d57f05c

	def __init__(
	self,
	dims,
	data: T_DuckArray | ArrayLike,
	attrs=None,
	encoding=None,
	fastpath=False,
	):
	"""
	Parameters
	----------
	dims : str or sequence of str
	Name(s) of the the data dimension(s). Must be either a string (only
	for 1D data) or a sequence of strings with length equal to the
	number of dimensions.
	data : array_like
	Data array which supports numpy-like data access.
	attrs : dict_like or None, optional
	Attributes to assign to the new variable. If None (default), an
	empty attribute dictionary is initialized.
	(see FAQ, :ref:`approach to metadata`)
	encoding : dict_like or None, optional
	Dictionary specifying how to encode this array's data into a
	serialized format like netCDF4. Currently used keys (for netCDF)
	include '_FillValue', 'scale_factor', 'add_offset' and 'dtype'.
	Well-behaved code to serialize a Variable should ignore
	unrecognized encoding items.
	"""
	super().__init__(
	dims=dims, data=as_compatible_data(data, fastpath=fastpath), attrs=attrs
	)

I do think this is basically a non-issue since PandasIndexingAdapter can also be returned from as_compatible_array and that also doesn't fulfill the NamedArray typing scheme since it also doesn't implement imag or real on _arrayfunction (or even __array_function__!!! at least the PandasExtensionArray class does that!!!):

xarray/xarray/namedarray/_typing.py

Lines 188 to 192 in d57f05c

	@property
	def imag(self) -> _arrayfunction[_ShapeType_co, Any]: ...
	@property
	def real(self) -> _arrayfunction[_ShapeType_co, Any]: ...

IMO we should punt on the typing issue beause it existed before this PR as far as I can tell

[headtr1ck]

I'm ok with leaving the ignore, not please open a new issue about it so we can keep track.