Fix/dashboard #227

ruthenian8 · 2023-09-14T12:59:46Z

Description

Create data provider, add new charts to dashboard; rework existing sections

Checklist

I have covered the code with tests
I have added comments to my code to help others understand it
I have updated the documentation to reflect the changes
I have performed a self-review of the changes

…and responses; update test

RLKRo

Some of my suggested changes are to yaml configuration files.
I'm not sure if they can be applied directly. I think it's better to apply corresponding changes via superset web app and export new config files.

tutorials/stats/3_sample_data_provider.py

docs/source/user_guides/superset_guide.rst

tutorials/stats/3_sample_data_provider.py

dff/config/superset_dashboard/charts/Terminal_labels_15.yaml

dff/config/superset_dashboard/dashboards/DFF_statistics_dashboard_1.yaml

dff/config/superset_dashboard/charts/Current_topic_slot_bar_chart_4.yaml

tutorials/stats/3_sample_data_provider.py

dff/utils/testing/toy_script.py

Co-authored-by: Roman Zlobin <[email protected]>

…to Superset documentation

docs/source/user_guides/superset_guide.rst

tutorials/stats/3_sample_data_provider.py

dff/utils/testing/toy_script.py

tutorials/stats/3_sample_data_provider.py

dff/config/superset_dashboard/charts/Node_Visits_7.yaml

dff/config/superset_dashboard/charts/Terminal_labels_15.yaml

dff/config/superset_dashboard/dashboards/DFF_statistics_dashboard_1.yaml

…sample_data_provider

dff/stats/__main__.py

RLKRo · 2023-10-05T14:39:06Z

dff/stats/__main__.py

https://github.com/deeppavlov/dialog_flow_framework/blob/eddc688912c1f3d07bf5f092f7d259e4dd7c60d3/dff/stats/__main__.py#L80-L99

I think it's better to not use actions for that.
Because the arguments are required, users are forced to specify them in the command (dff.stats config.yaml -U superset -P -dP).

I think it is better to make them not required and then process them inside the main function instead of processing with PasswordAction.

I think the current way is better, because it allows for clearer code; the face that users have to pass -U and -P does not sound like a concern, since that is the way it functions in MySQL, Postgresql and all kinds of other systems

RLKRo · 2023-10-05T14:42:55Z

dff/stats/cli.py

https://github.com/deeppavlov/dialog_flow_framework/blob/eddc688912c1f3d07bf5f092f7d259e4dd7c60d3/dff/stats/cli.py#L184-L187

Create a temporary file instead.

Otherwise it will delete temp.zip file in the current directory if it exists.

Replaced with uuid random name

tutorials/stats/3_sample_data_provider.py

dff/config/superset_dashboard/datasets/dff_database/dff_node_stats.yaml

dff/config/superset_dashboard/charts/Terminal_labels_15.yaml

docs/source/user_guides/superset_guide.rst

docker-compose.yml

docs/source/user_guides/superset_guide.rst

dff/config/superset_dashboard/datasets/dff_database/dff_node_stats.yaml

dff/utils/docker/superset_config_docker.py

…; make a variable for superset metadata db; adjust dockerfile to use configuration overrides

dff/config/superset_dashboard/charts/Flow_visit_ratio_monitor_13.yaml

dff/stats/cli.py

dff/config/superset_dashboard/datasets/dff_database/dff_stats.yaml

.github/workflows/update_dashboard.yml

docker-compose.yml

@RLKRo

Apply suggestions from @RLKRo Co-authored-by: Roman Zlobin <[email protected]>

Apply suggestions to label_lag/flow_lag Co-authored-by: Roman Zlobin <[email protected]>

…om charts; update docker-compose healthcheck; apply formatting

MANIFEST.in

dff/stats/cli.py

RLKRo · 2023-10-26T00:48:37Z

tests/stats/test_main.py

I think that in the light of the bugs I've discovered we should really test the charts.

Bugs such as request_ids being sorted as strings are difficult to catch by simply looking at the charts which makes them all the more dangerous as they still present wildly incorrect data.

I think we should prioritize finding a way to test the data in these charts.

dff/config/superset_dashboard/charts/Terminal_labels_15.yaml

RLKRo · 2023-10-26T00:55:19Z

docker-compose.yml

@@ -66,13 +87,21 @@ services:
      - '9000:9000'
    volumes:
      - ch-data:/var/lib/clickhouse/


It's weird that only clickhouse has its own volume.
Why not give a volume to all db containers?

Restarting all containers still resets all the metadata stored in dashboard-metadata.

I think that adding volumes to all containers lies outside the scope of this PR, so for now I will only add volumes for clickhouse and dashboard-metadata

RLKRo · 2023-10-26T00:58:09Z

dff/utils/docker/entrypoint_stats.sh

@@ -2,6 +2,7 @@
 export SERVER_THREADS_AMOUNT=8
 set -m
 nohup /bin/bash /usr/bin/run-server.sh &
+sleep 5


Maybe instead of a simple sleep we could use something like what we have in make wait_db?

I cannot figure out the exact event that we need to be waiting for; it would be feasible, if we knew what it was

RLKRo · 2023-10-26T01:00:44Z

docker-compose.yml

@@ -50,10 +50,31 @@ services:
      context: ./dff/utils/docker
      dockerfile: dockerfile_stats
    image: ghcr.io/deeppavlov/superset_df_dashboard:latest


We should build a local image in tests instead of using a remote image.
(as mentioned here)

This can be addressed by passing the --build flag to the docker-compose up command

Apparently, it was already building instead of pulling an image in previous commits:

I don't know why tests didn't fail due to authentication errors.
Maybe because context storages were also launched it gave enough time to whatever dashboard awaits for.

@RLKRo

Apply suggestions by @RLKRo Co-authored-by: Roman Zlobin <[email protected]>

…k expression; create volume for dashboard metadata

…node_stats

# Release notes ## dff.stats - Fix tests for statistics module (#272) - Dashboard config now persists after restart (#227) - Fix several bugs with SQL statements for datasets (#227) - Update guide with information about custom charts (#227) - Add text descriptions for charts in dashboard (#227) - Various chart fixes (#227) ## Other - Update PR template (#258) - Small doc fixesRelease v0.6.2

ruthenian8 added 12 commits September 5, 2023 17:02

update dashboard workflow

7c8e5ca

run on workflow_dispatch

bbd2b3d

trigger on PR to dev for debugging

6712898

update build push action

941a581

update context && file

5d7c753

remove trigger on PR

eaf772a

add sample data provider

27170ea

Update dashboard, finalize data provider

0531c49

Add extractors for requests and responses as default instrumentors

97b67c3

add 'get_last_request', 'get_last_response'; add charts for requests …

c180182

…and responses; update test

Update sections in superset_guide; update extractor functions tutor

92318cd

doc build fix: Change functions to func names in __all__

e45c298

RLKRo requested changes Sep 19, 2023

View reviewed changes

ruthenian8 and others added 9 commits September 20, 2023 10:31

Apply suggestions by RLKRO from code review

ac00230

Co-authored-by: Roman Zlobin <[email protected]>

intermediary update; update data provider tutor; update charts;

1d5a92a

Apply suggestions from code review by RLKRO

ce2f1d7

Co-authored-by: Roman Zlobin <[email protected]>

Upload annotations.png via GUI

ee86703

Update requests & responses charts; update superset_guide

721593e

expand the Superset guide with chart creation instructions;Add links …

fd05a11

…to Superset documentation

update graph plot; update dataset to get flow lag; update data provider

675e71b

update guide; move sample data collector to /utils

6af9a9f

update links to avoid sphinx warnings

532597e

RLKRo requested changes Sep 25, 2023

View reviewed changes

ruthenian8 added 7 commits September 26, 2023 13:41

merge dev

ee61909

update dashboard description; update tutorials

6ace191

update superset guide

c6f1259

Merge branch 'dev' into fix/dashboard

6646577

update label extractor; add infile parameter to main; remove link to …

17225e1

…sample_data_provider

update tests

d355290

update format

eddc688

minor doc improvements

3c15511

RLKRo requested changes Oct 5, 2023

View reviewed changes

ruthenian8 added 6 commits October 6, 2023 16:02

update dashboard datasource config

457a50c

add dff_stats dataset; link charts to dff_stats dataset

a470da3

update tests and documentation to reflect config changes

49d2e8a

update formatting

ed2db47

add the test configuration file

d9265ca

merge remote dev; resolve conflicts

0809a48

RLKRo requested changes Oct 19, 2023

View reviewed changes

RLKRo and others added 4 commits October 19, 2023 17:52

move get_current_label to after_handler in sample_data_provider.py

af92f36

update sql with if statements

9ae731e

specify healthcheck for clickhouse and wait for healthcheck with otlp…

cd0fc8e

…; make a variable for superset metadata db; adjust dockerfile to use configuration overrides

add dashboard-metadata docker image; adjust to run on port 5433

c53ab31

RLKRo requested changes Oct 24, 2023

View reviewed changes

ruthenian8 and others added 4 commits October 24, 2023 11:17

Update .github/workflows/update_dashboard.yml

32ce2ec

Apply suggestions from @RLKRo Co-authored-by: Roman Zlobin <[email protected]>

Update dff/stats/cli.py

9bfaeac

Apply suggestions to label_lag/flow_lag Co-authored-by: Roman Zlobin <[email protected]>

remove query statements from dashboard files; remove filter values fr…

d73b8d4

…om charts; update docker-compose healthcheck; apply formatting

wait for pg database; sleep to avoid process conflict

e8e725d

RLKRo requested changes Oct 26, 2023

View reviewed changes

ruthenian8 and others added 5 commits October 26, 2023 12:00

Apply suggestions by @RLKRo

ec7cf06

Apply suggestions by @RLKRo Co-authored-by: Roman Zlobin <[email protected]>

Update config files; Update sql expressions in CLI; update healthchec…

fb3cf72

…k expression; create volume for dashboard metadata

convert request_id to int in final_nodes

fa5d550

remove sql in final_nodes

3fad9d6

update history filter to numerical range type & change datasource to …

464a2bd

…node_stats

RLKRo approved these changes Oct 26, 2023

View reviewed changes

remove utils include from manifest

c329025

ruthenian8 merged commit 9b2d45f into dev Oct 27, 2023
16 checks passed

RLKRo mentioned this pull request Nov 1, 2023

Test charts #262

Draft

4 tasks

RLKRo mentioned this pull request Nov 9, 2023

Release v0.6.2 #273

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix/dashboard #227

Fix/dashboard #227

ruthenian8 commented Sep 14, 2023

RLKRo left a comment

RLKRo Oct 5, 2023

ruthenian8 Oct 9, 2023

RLKRo Oct 5, 2023

ruthenian8 Oct 9, 2023

RLKRo Oct 26, 2023

RLKRo Oct 26, 2023

ruthenian8 Oct 26, 2023

RLKRo Oct 26, 2023

ruthenian8 Oct 26, 2023

RLKRo Oct 26, 2023

ruthenian8 Oct 26, 2023

RLKRo Oct 26, 2023

Fix/dashboard #227

Fix/dashboard #227

Conversation

ruthenian8 commented Sep 14, 2023

Description

Checklist

RLKRo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment