Skip to content

Commit

Permalink
airbyte-cdk: add comments for evict method
Browse files Browse the repository at this point in the history
  • Loading branch information
aldogonzalez8 committed Nov 27, 2024
1 parent 0d40f92 commit 466adde
Showing 1 changed file with 7 additions and 0 deletions.
7 changes: 7 additions & 0 deletions airbyte_cdk/sources/streams/http/http_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -341,6 +341,13 @@ def _send(
return response # type: ignore # will either return a valid response of type requests.Response or raise an exception

def _evict_key(self, prepared_request: requests.PreparedRequest) -> None:
"""
Addresses high memory consumption when enabling concurrency in https://github.com/airbytehq/oncall/issues/6821.
The `_request_attempt_count` attribute keeps growing as multiple requests are made using the same `http_client`.
To mitigate this issue, we evict keys for completed requests once we confirm that no further retries are needed.
This helps manage memory usage more efficiently while maintaining the necessary logic for retry attempts.
"""
if prepared_request in self._request_attempt_count:
del self._request_attempt_count[prepared_request]

Expand Down

0 comments on commit 466adde

Please sign in to comment.