Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(instrumentation): Add event-based tracking implementation across providers #2541

Open
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

onyedikachi-david
Copy link

@onyedikachi-david onyedikachi-david commented Jan 22, 2025

/claim #2456
Fixes: #2456

  • I have added tests that cover my changes.
  • If adding a new instrumentation or changing an existing one, I've added screenshots from some observability platform showing the change.
  • PR name follows conventional commits format: feat(instrumentation): ... or fix(instrumentation): ....
  • (If applicable) I have updated the documentation accordingly.

Description

This PR adds event-based tracking implementation across multiple providers while maintaining backward compatibility with the legacy attribute-based approach.

Completed Implementations:

  • ✅ Anthropic
  • ✅ Bedrock
  • ✅ Cohere
  • ✅ Google GenerativeAI
  • ✅ Groq
  • ✅ Langchain
  • ✅ LlamaIndex
  • ✅ MistralAI
  • ✅ Ollama
  • ✅ Replicate

Pending Implementations:

  • 🔄 SageMaker
  • 🔄 Together
  • 🔄 Transformers
  • 🔄 VertexAI
  • 🔄 WatsonX

Key Changes:

  1. Added event-based configuration with use_legacy_attributes support
  2. Implemented event utilities for:
    • Prompts
    • Completions
    • Embeddings (where applicable)
    • Tool/Function calls (where applicable)
  3. Added comprehensive test coverage for event emission
  4. Added support for both sync and async clients
  5. Added streaming support where applicable
  6. Maintained backward compatibility with legacy mode

Each provider implementation includes:

  • Event utilities (events.py)
  • Configuration updates (config.py)
  • Core instrumentation updates (__init__.py)
  • Comprehensive test coverage (test_events.py)

…raceloop#2456)

- Add event-based tracking support while maintaining legacy behavior
- Implement use_legacy_attributes config parameter
- Add event logger integration with proper trace context
- Add streaming support with event buffering
- Include comprehensive test coverage
- Add sample app integration

Part of the larger effort to support event-based tracking across all packages.

Signed-off-by: David Anyatonwu <[email protected]>
@dosubot dosubot bot added the size:XXL This PR changes 1000+ lines, ignoring generated files. label Jan 22, 2025
@CLAassistant
Copy link

CLAassistant commented Jan 22, 2025

CLA assistant check
All committers have signed the CLA.

@onyedikachi-david
Copy link
Author

@nirga Could i get a review, to confirm I am on the right track. Thanks.

@nirga
Copy link
Member

nirga commented Jan 22, 2025

@onyedikachi-david can you rebase so we can run the CI?

@onyedikachi-david
Copy link
Author

@onyedikachi-david can you rebase so we can run the CI?

alright

@onyedikachi-david
Copy link
Author

Done @nirga

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🙋 Bounty claim new instrumentation size:XXL This PR changes 1000+ lines, ignoring generated files. testing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

🚀 Feature: support emitting events for prompts in addition to current behavior
3 participants