Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add queries and tokens metrics #647

Merged
merged 6 commits into from
Oct 16, 2024
Merged

Add queries and tokens metrics #647

merged 6 commits into from
Oct 16, 2024

Conversation

whiterabbit1983
Copy link
Contributor

@whiterabbit1983 whiterabbit1983 commented Oct 14, 2024

Important

Add Prometheus metrics tracking to various API functions using a new increase_counter decorator.

  • Metrics:
    • Add increase_counter decorator in counters.py to increment Prometheus counters for function calls.
    • Apply increase_counter to multiple functions across various models, including create_agent, create_or_update_agent, patch_agent, update_agent, create_doc, create_entries, create_execution, create_execution_transition, create_temporal_lookup, update_execution, create_or_update_session, create_session, update_session, create_or_update_task, create_task, patch_task, update_task, create_tools, patch_tool, update_tool, create_or_update_user, create_user, patch_user, update_user.
  • Dependencies:
    • Add prometheus-client to pyproject.toml for metrics tracking.
  • Misc:
    • Add total_tokens_per_user counter in metrics.py to track token usage per user in chat.py.

This description was created by Ellipsis for fe315d7. It will automatically update as commits are pushed.

Copy link
Contributor

@ellipsis-dev ellipsis-dev bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ Changes requested. Reviewed everything up to 464d538 in 33 seconds

More details
  • Looked at 573 lines of code in 28 files
  • Skipped 1 files when reviewing.
  • Skipped posting 0 drafted comments based on config settings.

Workflow ID: wflow_dn4TpIUz4ZyuOW3O


Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

agents-api/agents_api/metrics/counters.py Outdated Show resolved Hide resolved
Copy link
Contributor

@ellipsis-dev ellipsis-dev bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ Changes requested. Incremental review on 3c77808 in 25 seconds

More details
  • Looked at 49 lines of code in 3 files
  • Skipped 0 files when reviewing.
  • Skipped posting 1 drafted comments based on config settings.
1. agents-api/agents_api/metrics/counters.py:20
  • Draft comment:
    Consider adding a check to ensure id_field_name exists in kwargs before using it. This will prevent potential issues if the expected keyword argument is not provided.
            metric.labels(kwargs.get(id_field_name, "not_set")).inc()
  • Reason this comment was not posted:
    Confidence changes required: 50%
    The increase_counter decorator in counters.py is designed to increment a counter for each function call, but it doesn't handle the case where the function might not receive the expected keyword argument. This could lead to incorrect labeling in Prometheus metrics.

Workflow ID: wflow_JwUkLwJv7YyrBYfm


Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

agents-api/agents_api/routers/sessions/metrics.py Outdated Show resolved Hide resolved
Copy link
Contributor

@ellipsis-dev ellipsis-dev bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Looks good to me! Incremental review on 557950c in 14 seconds

More details
  • Looked at 25 lines of code in 2 files
  • Skipped 0 files when reviewing.
  • Skipped posting 1 drafted comments based on config settings.
1. agents-api/agents_api/routers/sessions/metrics.py:6
  • Draft comment:
    The change from labels to labelnames is correct and necessary for the Prometheus Counter initialization. Ensure this is consistent across all similar initializations.
  • Reason this comment was not posted:
    Confidence changes required: 0%
    The change from 'labels' to 'labelnames' in the Counter initialization is correct and aligns with the Prometheus client library's API.

Workflow ID: wflow_TkPgz28zea4IK6wV


You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

Copy link
Contributor

@ellipsis-dev ellipsis-dev bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Looks good to me! Incremental review on fe315d7 in 16 seconds

More details
  • Looked at 12 lines of code in 1 files
  • Skipped 0 files when reviewing.
  • Skipped posting 1 drafted comments based on config settings.
1. agents-api/tests/test_docs_queries.py:73
  • Draft comment:
    Avoid using @skip without a clear reason or timeline for fixing the test. Consider adding a TODO comment with details on why it's skipped and when it should be revisited. This applies to other instances of @skip in the file as well.
  • Reason this comment was not posted:
    Confidence changes required: 50%
    The use of the @skip decorator without a clear reason or timeline for fixing the test is not ideal. It can lead to tests being forgotten and not addressed.

Workflow ID: wflow_fZRR7KYalbgPMCqY


You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

Copy link
Contributor

@ellipsis-dev ellipsis-dev bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skipped PR review on facb454 because no changed files had a supported extension. If you think this was in error, please contact us and we'll fix it right away.


Generated with ❤️ by ellipsis.dev

@whiterabbit1983 whiterabbit1983 merged commit 9019b57 into dev Oct 16, 2024
7 of 11 checks passed
@whiterabbit1983 whiterabbit1983 deleted the f/queries-metrics branch October 16, 2024 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants