fix: throw exception in consumer thread #4583
Conversation
📝 WalkthroughWalkthroughThe updates enhance the BackgroundConsumer.run() method in both the pd and pt dataloader modules. The iteration over the data source is now enclosed in a try block, allowing any exceptions that occur to be caught and enqueued. In addition, once the iteration is complete, an explicit StopIteration signal is pushed to the queue. These changes ensure that errors are propagated properly and the completion of the data stream is clearly indicated. Changes
Sequence Diagram(s)sequenceDiagram
participant BC as BackgroundConsumer
participant DS as Data Source
participant Q as Queue
BC->>DS: Start iterating over source
alt Successful iteration
DS-->>BC: Next item
BC->>Q: Enqueue item
BC-)DS: Continue iteration
else Exception occurs
DS-->>BC: Exception raised
BC->>Q: Enqueue exception
end
BC->>Q: Enqueue StopIteration signal
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (2)
⏰ Context from checks skipped due to timeout of 90000ms (19)
🔇 Additional comments (2)
✨ Finishing Touches
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## devel #4583 +/- ##
==========================================
- Coverage 84.60% 84.59% -0.01%
==========================================
Files 680 680
Lines 64473 64479 +6
Branches 3540 3540
==========================================
+ Hits 54546 54547 +1
- Misses 8786 8790 +4
- Partials 1141 1142 +1 ☔ View full report in Codecov by Sentry. |
This pull request includes changes to improve error handling in the
dataloader.pyfiles for bothdeepmd/pd/utilsanddeepmd/pt/utils. The most important changes are the addition of a try-except block to handle exceptions during the data loading process and signaling the end of iteration properly.Error handling improvements:
deepmd/pd/utils/dataloader.py: Added a try-except block to catch exceptions during the iteration over the data source, and modified the signaling of the end of iteration to useStopIteration().deepmd/pt/utils/dataloader.py: Added a try-except block to catch exceptions during the iteration over the data source, and ensured the end of iteration is signaled withStopIteration().Summary by CodeRabbit