ValueError when decoding virtual reference: Filenames containing trailing '/#\d+/' are not supported #402
Comments
|
Okay, as the error said something about filenames I tried to run the example in the icechunk docs instead (https://icechunk.io/icechunk-python/virtual/#creating-a-virtual-dataset-with-virtualizarr), which is different files with different names on S3 instead of local and got ---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/conventions.py:451](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/conventions.py#line=450), in decode_cf_variables(variables, attributes, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta)
450 try:
--> 451 new_vars[k] = decode_cf_variable(
452 k,
453 v,
454 concat_characters=_item_or_default(concat_characters, k, True),
455 mask_and_scale=_item_or_default(mask_and_scale, k, True),
456 decode_times=_item_or_default(decode_times, k, True),
457 stack_char_dim=stack_char_dim,
458 use_cftime=_item_or_default(use_cftime, k, None),
459 decode_timedelta=_item_or_default(decode_timedelta, k, None),
460 )
461 except Exception as e:
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/conventions.py:292](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/conventions.py#line=291), in decode_cf_variable(name, var, concat_characters, mask_and_scale, decode_times, decode_endianness, stack_char_dim, use_cftime, decode_timedelta)
291 if decode_times:
--> 292 var = times.CFDatetimeCoder(use_cftime=use_cftime).decode(var, name=name)
294 if decode_endianness and not var.dtype.isnative:
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/coding/times.py:1001](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/coding/times.py#line=1000), in CFDatetimeCoder.decode(self, variable, name)
1000 calendar = pop_to(attrs, encoding, "calendar")
-> 1001 dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
1002 transform = partial(
1003 decode_cf_datetime,
1004 units=units,
1005 calendar=calendar,
1006 use_cftime=self.use_cftime,
1007 )
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/coding/times.py:214](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/coding/times.py#line=213), in _decode_cf_datetime_dtype(data, units, calendar, use_cftime)
212 values = indexing.ImplicitToExplicitIndexingAdapter(indexing.as_indexable(data))
213 example_value = np.concatenate(
--> 214 [first_n_items(values, 1) or [0], last_item(values) or [0]]
215 )
217 try:
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/core/formatting.py:97](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/core/formatting.py#line=96), in first_n_items(array, n_desired)
96 array = array._data
---> 97 return np.ravel(to_duck_array(array))[:n_desired]
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/namedarray/pycompat.py:138](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/namedarray/pycompat.py#line=137), in to_duck_array(data, **kwargs)
137 else:
--> 138 return np.asarray(data)
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/core/indexing.py:578](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/core/indexing.py#line=577), in ImplicitToExplicitIndexingAdapter.__array__(self, dtype, copy)
577 if Version(np.__version__) >= Version("2.0.0"):
--> 578 return np.asarray(self.get_duck_array(), dtype=dtype, copy=copy)
579 else:
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/core/indexing.py:583](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/core/indexing.py#line=582), in ImplicitToExplicitIndexingAdapter.get_duck_array(self)
582 def get_duck_array(self):
--> 583 return self.array.get_duck_array()
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/core/indexing.py:657](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/core/indexing.py#line=656), in LazilyIndexedArray.get_duck_array(self)
654 else:
655 # If the array is not an ExplicitlyIndexedNDArrayMixin,
656 # it may wrap a BackendArray so use its __getitem__
--> 657 array = self.array[self.key]
659 # self.array[self.key] is now a numpy array when
660 # self.array is a BackendArray subclass
661 # and self.key is BasicIndexer((slice(None, None, None),))
662 # so we need the explicit check for ExplicitlyIndexed
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/zarr.py:166](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/zarr.py#line=165), in ZarrArrayWrapper.__getitem__(self, key)
165 method = self._oindex
--> 166 return indexing.explicit_indexing_adapter(
167 key, array.shape, indexing.IndexingSupport.VECTORIZED, method
168 )
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/core/indexing.py:1018](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/core/indexing.py#line=1017), in explicit_indexing_adapter(key, shape, indexing_support, raw_indexing_method)
1017 raw_key, numpy_indices = decompose_indexer(key, shape, indexing_support)
-> 1018 result = raw_indexing_method(raw_key.tuple)
1019 if numpy_indices.tuple:
1020 # index the loaded np.ndarray
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/zarr.py:156](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/zarr.py#line=155), in ZarrArrayWrapper._getitem(self, key)
155 def _getitem(self, key):
--> 156 return self._array[key]
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/array.py:1660](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/array.py#line=1659), in Array.__getitem__(self, selection)
1659 elif is_pure_orthogonal_indexing(pure_selection, self.ndim):
-> 1660 return self.get_orthogonal_selection(pure_selection, fields=fields)
1661 else:
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/_compat.py:43](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/_compat.py#line=42), in _deprecate_positional_args.<locals>._inner_deprecate_positional_args.<locals>.inner_f(*args, **kwargs)
42 if extra_args <= 0:
---> 43 return f(*args, **kwargs)
45 # extra_args > 0
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/array.py:2102](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/array.py#line=2101), in Array.get_orthogonal_selection(self, selection, out, fields, prototype)
2101 indexer = OrthogonalIndexer(selection, self.shape, self.metadata.chunk_grid)
-> 2102 return sync(
2103 self._async_array._get_selection(
2104 indexer=indexer, out=out, fields=fields, prototype=prototype
2105 )
2106 )
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/sync.py:141](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/sync.py#line=140), in sync(coro, loop, timeout)
140 if isinstance(return_result, BaseException):
--> 141 raise return_result
142 else:
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/sync.py:100](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/sync.py#line=99), in _runner(coro)
99 try:
--> 100 return await coro
101 except Exception as ex:
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/array.py:961](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/array.py#line=960), in AsyncArray._get_selection(self, indexer, prototype, out, fields)
959 if product(indexer.shape) > 0:
960 # reading chunks and decoding them
--> 961 await self.codec_pipeline.read(
962 [
963 (
964 self.store_path [/](http://localhost:8888/) self.metadata.encode_chunk_key(chunk_coords),
965 self.metadata.get_chunk_spec(chunk_coords, self.order, prototype=prototype),
966 chunk_selection,
967 out_selection,
968 )
969 for chunk_coords, chunk_selection, out_selection in indexer
970 ],
971 out_buffer,
972 drop_axes=indexer.drop_axes,
973 )
974 return out_buffer.as_ndarray_like()
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/codecs/pipeline.py:440](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/codecs/pipeline.py#line=439), in BatchedCodecPipeline.read(self, batch_info, out, drop_axes)
434 async def read(
435 self,
436 batch_info: Iterable[tuple[ByteGetter, ArraySpec, SelectorTuple, SelectorTuple]],
437 out: NDBuffer,
438 drop_axes: tuple[int, ...] = (),
439 ) -> None:
--> 440 await concurrent_map(
441 [
442 (single_batch_info, out, drop_axes)
443 for single_batch_info in batched(batch_info, self.batch_size)
444 ],
445 self.read_batch,
446 config.get("async.concurrency"),
447 )
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/common.py:64](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/common.py#line=63), in concurrent_map(items, func, limit)
62 return await func(*item)
---> 64 return await asyncio.gather(*[asyncio.ensure_future(run(item)) for item in items])
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/common.py:62](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/common.py#line=61), in concurrent_map.<locals>.run(item)
61 async with sem:
---> 62 return await func(*item)
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/codecs/pipeline.py:262](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/codecs/pipeline.py#line=261), in BatchedCodecPipeline.read_batch(self, batch_info, out, drop_axes)
261 else:
--> 262 chunk_bytes_batch = await concurrent_map(
263 [
264 (byte_getter, array_spec.prototype)
265 for byte_getter, array_spec, _, _ in batch_info
266 ],
267 lambda byte_getter, prototype: byte_getter.get(prototype),
268 config.get("async.concurrency"),
269 )
270 chunk_array_batch = await self.decode_batch(
271 [
272 (chunk_bytes, chunk_spec)
(...)
276 ],
277 )
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/common.py:64](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/common.py#line=63), in concurrent_map(items, func, limit)
62 return await func(*item)
---> 64 return await asyncio.gather(*[asyncio.ensure_future(run(item)) for item in items])
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/common.py:62](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/core/common.py#line=61), in concurrent_map.<locals>.run(item)
61 async with sem:
---> 62 return await func(*item)
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/storage/common.py:71](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/zarr/storage/common.py#line=70), in StorePath.get(self, prototype, byte_range)
70 prototype = default_buffer_prototype()
---> 71 return await self.store.get(self.path, prototype=prototype, byte_range=byte_range)
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/icechunk/__init__.py:469](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/icechunk/__init__.py#line=468), in IcechunkStore.get(self, key, prototype, byte_range)
468 try:
--> 469 result = await self._store.get(key, byte_range)
470 except KeyNotFound as _e:
471 # Zarr python expects None to be returned if the key does not exist
472 # but an IcechunkStore returns an error if the key does not exist
ValueError: store error: unsuccessful repository operation: `error when handling virtual reference error fetching virtual reference dispatch failure`
The above exception was the direct cause of the following exception:
ValueError Traceback (most recent call last)
Cell In[43], line 1
----> 1 ds = xr.open_zarr(
2 store,
3 zarr_format=3,
4 consolidated=False,
5 chunks={},
6 )
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/zarr.py:1337](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/zarr.py#line=1336), in open_zarr(store, group, synchronizer, chunks, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, consolidated, overwrite_encoded_chunks, chunk_store, storage_options, decode_timedelta, use_cftime, zarr_version, zarr_format, use_zarr_fill_value_as_mask, chunked_array_type, from_array_kwargs, **kwargs)
1322 raise TypeError(
1323 "open_zarr() got unexpected keyword arguments " + ",".join(kwargs.keys())
1324 )
1326 backend_kwargs = {
1327 "synchronizer": synchronizer,
1328 "consolidated": consolidated,
(...)
1334 "zarr_format": zarr_format,
1335 }
-> 1337 ds = open_dataset(
1338 filename_or_obj=store,
1339 group=group,
1340 decode_cf=decode_cf,
1341 mask_and_scale=mask_and_scale,
1342 decode_times=decode_times,
1343 concat_characters=concat_characters,
1344 decode_coords=decode_coords,
1345 engine="zarr",
1346 chunks=chunks,
1347 drop_variables=drop_variables,
1348 chunked_array_type=chunked_array_type,
1349 from_array_kwargs=from_array_kwargs,
1350 backend_kwargs=backend_kwargs,
1351 decode_timedelta=decode_timedelta,
1352 use_cftime=use_cftime,
1353 zarr_version=zarr_version,
1354 use_zarr_fill_value_as_mask=use_zarr_fill_value_as_mask,
1355 )
1356 return ds
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/api.py:671](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/api.py#line=670), in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, inline_array, chunked_array_type, from_array_kwargs, backend_kwargs, **kwargs)
659 decoders = _resolve_decoders_kwargs(
660 decode_cf,
661 open_backend_dataset_parameters=backend.open_dataset_parameters,
(...)
667 decode_coords=decode_coords,
668 )
670 overwrite_encoded_chunks = kwargs.pop("overwrite_encoded_chunks", None)
--> 671 backend_ds = backend.open_dataset(
672 filename_or_obj,
673 drop_variables=drop_variables,
674 **decoders,
675 **kwargs,
676 )
677 ds = _dataset_from_backend_dataset(
678 backend_ds,
679 filename_or_obj,
(...)
689 **kwargs,
690 )
691 return ds
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/zarr.py:1427](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/zarr.py#line=1426), in ZarrBackendEntrypoint.open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, group, mode, synchronizer, consolidated, chunk_store, storage_options, stacklevel, zarr_version, zarr_format, store, engine, use_zarr_fill_value_as_mask)
1425 store_entrypoint = StoreBackendEntrypoint()
1426 with close_on_error(store):
-> 1427 ds = store_entrypoint.open_dataset(
1428 store,
1429 mask_and_scale=mask_and_scale,
1430 decode_times=decode_times,
1431 concat_characters=concat_characters,
1432 decode_coords=decode_coords,
1433 drop_variables=drop_variables,
1434 use_cftime=use_cftime,
1435 decode_timedelta=decode_timedelta,
1436 )
1437 return ds
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/store.py:46](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/backends/store.py#line=45), in StoreBackendEntrypoint.open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta)
43 vars, attrs = filename_or_obj.load()
44 encoding = filename_or_obj.get_encoding()
---> 46 vars, attrs, coord_names = conventions.decode_cf_variables(
47 vars,
48 attrs,
49 mask_and_scale=mask_and_scale,
50 decode_times=decode_times,
51 concat_characters=concat_characters,
52 decode_coords=decode_coords,
53 drop_variables=drop_variables,
54 use_cftime=use_cftime,
55 decode_timedelta=decode_timedelta,
56 )
58 ds = Dataset(vars, attrs=attrs)
59 ds = ds.set_coords(coord_names.intersection(vars))
File [~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/conventions.py:462](http://localhost:8888/lab/tree/~/miniconda3/envs/coiled/lib/python3.12/site-packages/xarray/conventions.py#line=461), in decode_cf_variables(variables, attributes, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta)
451 new_vars[k] = decode_cf_variable(
452 k,
453 v,
(...)
459 decode_timedelta=_item_or_default(decode_timedelta, k, None),
460 )
461 except Exception as e:
--> 462 raise type(e)(f"Failed to decode variable {k!r}: {e}") from e
463 if decode_coords in [True, "coordinates", "all"]:
464 var_attrs = new_vars[k].attrs
ValueError: Failed to decode variable 'time': store error: unsuccessful repository operation: `error when handling virtual reference error fetching virtual reference dispatch failure`Is there something wrong with my environment / zarr version? |
|
This might be an issue with xarray supporting the latest version of zarr cc @dcherian |
|
Related? #279 |
|
Thanks @norlandrhagen ! That could well be it. I'll try to address this in VirtualiZarr then (via zarr-developers/VirtualiZarr#243). But if it actually is #279 (i.e. icechunk doesn't like non-absolute paths) then the error message from icechunk could be a lot clearer. |
|
Ahhhh yeah for now it cant handle relative paths. +1 on needing better errors #328 |
25 tasks
|
Now that I've merged zarr-developers/VirtualiZarr#243 this should be closed, as all references in VirtualiZarr are now absolute URIs. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Not sure if this is virtualizarr or icechunk's fault but the error was raised inside icechunk so I'm raising it here.
MCVE
Error
Environment
The text was updated successfully, but these errors were encountered: