Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stats with Opentelemetry #141

Merged
merged 186 commits into from
Sep 5, 2023
Merged

Stats with Opentelemetry #141

merged 186 commits into from
Sep 5, 2023

Conversation

ruthenian8
Copy link
Member

@ruthenian8 ruthenian8 commented Jun 2, 2023

Description

Collect and export telemetry information using Opentelemetry Python sdk, Clickhouse, and Apache Superset.

Checklist

  • I have covered the code with tests
  • I have added comments to my code to help others understand it
  • I have updated the documentation to reflect the changes
  • I have performed a self-review of the changes

kudep and others added 30 commits October 24, 2022 18:15
Also move superset_dashboard in /dff/config/
Since superset_dashboard was moved to dff/config need to update its location in sourcefiles
Also add superset_dashboard to MANIFEST.in
Pipeline's interface was changed between versions v1 and v2. This commit changes imported names in stats examples to reflect that.

Note: Also mass replace showed that documentation for pipeline is outdated.
Since documentation building requires all files in examples dir to be named differently, rename _utils.
- Add dot_path_to_addon to find examples location
- Explicitly list examples

Signed-off-by: Roman Zlobin <[email protected]>
- Add .dockerignore from stats
- Add volumes to existing services
- Add [.env_file] to new services
- Add CLICKHOUSE config to .env_file
- Add df_stats/dockerfile as dockerfile_stats

Signed-off-by: Roman Zlobin <[email protected]>
- Add sql_dependencies (a lot of dbs use sqlalchemy so I decided to move it to a separate list)
- Add asyncpg as postgres dependency (stats used it instead of psycopg)
- Add an entry point for dff.stats

Signed-off-by: Roman Zlobin <[email protected]>
This commit does not rename table names from df_stats to dff.stats
This change moves pings to dbs from tests/db_connector/test_dbs.py to tests/db_list.py
.gitattributes Outdated Show resolved Hide resolved
Comment on lines +113 to +119
"""
Type of object, that is passed to components in runtime.
Contains current component info (`name`, `path`, `timeout`, `asynchronous`).
Also contains `execution_state` - a dictionary,
containing execution states of other components mapped to their paths.
"""

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So maybe we should move it to another one?

dff/stats/cli.py Outdated Show resolved Hide resolved
dff/stats/cli.py Outdated Show resolved Hide resolved
dff/stats/cli.py Outdated Show resolved Hide resolved
dff/stats/instrumentor.py Show resolved Hide resolved
tests/stats/__init__.py Outdated Show resolved Hide resolved
tests/stats/test_main.py Outdated Show resolved Hide resolved
tutorials/stats/1_extractor_functions.py Show resolved Hide resolved
tutorials/stats/2_pipeline_integration.py Outdated Show resolved Hide resolved
Copy link
Member

@RLKRo RLKRo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.github/workflows/update_dashboard.yml Outdated Show resolved Hide resolved
docs/source/user_guides/superset_guide.rst Outdated Show resolved Hide resolved
dff/stats/__main__.py Show resolved Hide resolved
docs/source/user_guides/superset_guide.rst Outdated Show resolved Hide resolved
dff/stats/__main__.py Show resolved Hide resolved
docs/source/conf.py Outdated Show resolved Hide resolved
docs/source/user_guides/superset_guide.rst Outdated Show resolved Hide resolved
docs/source/user_guides/superset_guide.rst Outdated Show resolved Hide resolved
dff/config/README.md Outdated Show resolved Hide resolved
Copy link
Member

@RLKRo RLKRo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now superset user guide shows a plot that was removed (service_stats.png).

You could use this image instead:
image

dff/stats/cli.py Show resolved Hide resolved
dff/stats/cli.py Show resolved Hide resolved
@RLKRo RLKRo mentioned this pull request Aug 29, 2023
5 tasks
@kudep kudep merged commit f194ffe into dev Sep 5, 2023
@kudep kudep deleted the feat/otlponly branch September 5, 2023 12:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants