fix crash in data loader caused by using stale array #765
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Helw150
approved these changes
Oct 14, 2024
| @@ -98,6 +99,8 @@ def __iter__(self): | |||
| return self.iter_from_step(None) | |||
|
|
|||
| def iter_from_step(self, start_from_batch: Optional[int] = None): | |||
| # sometimes we pass in an array for the start_from_batch, so we need to check for that | |||
| start_from_batch = int(start_from_batch) if start_from_batch is not None else None | |||
There was a problem hiding this comment.
Is there a way to account for the case the comment describes in the MyPy type hint?
There was a problem hiding this comment.
i could make it be an jnp.ndarray, but i'd rather we didn't pass it in and i'm vaguely annoyed mypy isn't raising an error already.
src/levanter/data/loader.py
Outdated
|
|
||
| gda_tree = jax.tree.unflatten(self.dl._ex_structure, gda_leaves) | ||
| yield gda_tree | ||
| # TODO: this doesn't work with named axes |
There was a problem hiding this comment.
Wouldn't anything with named axes go into the if isinstance(leaf_data, hax.NamedArray) above rather than the parent else?
|
The error in the TPU test doesn't look like a flaky one to me though! |
|
yeah it's not flaky. it's real. I might merge anyway though b/c it's a low priority test. trying to figure it out |
|
figured it out |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.