Update generic metrics for Average Miles for each mode (selected by u… #113

iantei · 2024-01-18T20:23:23Z

…sers) chart, and update generic metrics sensed notebook for Average miles per transport mode selected (primary_mode).

iantei · 2024-01-18T20:55:31Z

Tested with the following settings:

Execution Details:

``` Dataset: `fc_2022-05-09` and `openpath-prod-usaid-laos` > show dbs Stage_database 4.428GB admin 0.000GB config 0.000GB local 0.000GB openpath_prod_usaid_laos_ev 0.760GB ```

Executed both the following notebooks: Generic Metrics & Generic Metrics Sensed notebooks

Generic Metrics notebook:

(emission) root@a969248e7123:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/update_mappings.py mapping_dictionaries.ipynb
(emission) root@a969248e7123:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/usaid-laos-ev.nrel-op.json
Successfully downloaded config with version 1 for USAID-NREL Support for Electric Vehicle Readiness and data collection URL https://USAID-laos-EV-openpath.nrel.gov/api/
Dynamic labels download was successful for nrel-openpath-deploy-configs: usaid-laos-ev
Running at 2024-01-18T20:12:49.735168+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (<Arrow [2023-05-01T00:00:00+00:00]>, <Arrow [2024-01-01T00:00:00+00:00]>)
Running at 2024-01-18T20:12:49.780057+00:00 with params [Parameter('program', str, value='default'), Parameter('study_type', str, value='study'), Parameter('include_test_users', bool, value=True), Parameter('dynamic_labels', dict, value={'MODE': [{'value': 'walk', 'baseMode': 'WALKING', 'met_equivalent': 'WALKING', 'kgCo2PerKm': 0}, {'value': 'e-auto_rickshaw', 'baseMode': 'MOPED', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.085416859}, {'value': 'auto_rickshaw', 'baseMode': 'MOPED', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.231943784}, {'value': 'motorcycle', 'baseMode': 'MOPED', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.113143309}, {'value': 'e-bike', 'baseMode': 'E_BIKE', 'met': {'ALL': {'range': [0, -1], 'mets': 4.9}}, 'kgCo2PerKm': 0.00728}, {'value': 'bike', 'baseMode': 'BICYCLING', 'met_equivalent': 'BICYCLING', 'kgCo2PerKm': 0}, {'value': 'drove_alone', 'baseMode': 'CAR', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.22031}, {'value': 'shared_ride', 'baseMode': 'CAR', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.11015}, {'value': 'e_car_drove_alone', 'baseMode': 'E_CAR', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.08216}, {'value': 'e_car_shared_ride', 'baseMode': 'E_CAR', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.04108}, {'value': 'taxi', 'baseMode': 'TAXI', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.30741}, {'value': 'bus', 'baseMode': 'BUS', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.20727}, {'value': 'train', 'baseMode': 'TRAIN', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.12256}, {'value': 'free_shuttle', 'baseMode': 'BUS', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.20727}, {'value': 'air', 'baseMode': 'AIR', 'met_equivalent': 'IN_VEHICLE', 'kgCo2PerKm': 0.09975}, {'value': 'not_a_trip', 'baseMode': 'UNKNOWN', 'met_equivalent': 'UNKNOWN', 'kgCo2PerKm': 0}, {'value': 'other', 'baseMode': 'OTHER', 'met_equivalent': 'UNKNOWN', 'kgCo2PerKm': 0}], 'PURPOSE': [{'value': 'home'}, {'value': 'work'}, {'value': 'at_work'}, {'value': 'school'}, {'value': 'transit_transfer'}, {'value': 'shopping'}, {'value': 'meal'}, {'value': 'pick_drop_person'}, {'value': 'pick_drop_item'}, {'value': 'personal_med'}, {'value': 'access_recreation'}, {'value': 'exercise'}, {'value': 'entertainment'}, {'value': 'religious'}, {'value': 'other'}], 'translations': {'en': {'walk': 'Walk', 'e-auto_rickshaw': 'E-tuk tuk', 'auto_rickshaw': 'Tuk Tuk', 'motorcycle': 'Motorcycle', 'e-bike': 'E-bike', 'bike': 'Bicycle', 'drove_alone': 'Car Drove Alone', 'shared_ride': 'Car Shared Ride', 'e_car_drove_alone': 'E-Car Drove Alone', 'e_car_shared_ride': 'E-Car Shared Ride', 'taxi': 'Taxi/Loca/inDrive', 'bus': 'Bus', 'train': 'Train', 'free_shuttle': 'Free Shuttle', 'air': 'Airplane', 'not_a_trip': 'Not a trip', 'home': 'Home', 'work': 'To Work', 'at_work': 'At Work', 'school': 'School', 'transit_transfer': 'Transit transfer', 'shopping': 'Shopping', 'meal': 'Meal', 'pick_drop_person': 'Pick-up/ Drop off Person', 'pick_drop_item': 'Pick-up/ Drop off Item', 'personal_med': 'Personal/ Medical', 'access_recreation': 'Access Recreation', 'exercise': 'Recreation/ Exercise', 'entertainment': 'Entertainment/ Social', 'religious': 'Religious', 'other': 'Other'}, 'lo': {'walk': 'ດ້ວຍການຍ່າງ', 'e-auto_rickshaw': 'ລົດ 3 ລໍ້ໄຟຟ້າ ຫລື ຕຸກຕຸກໄຟຟ້າ', 'auto_rickshaw': 'ເດີນທາດ້ວຍ ລົດຕຸກຕຸກ ຫລື ລົດສາມລໍ້', 'motorcycle': 'ລົດຈັກ', 'e-bike': 'ວຍລົດຈັກໄຟຟ້າ', 'bike': 'ລົດຖີບ', 'drove_alone': 'ເດີນທາງ ດ້ວຍລົດໃຫ່ຍ ເຊີ່ງເປັນລົດທີ່ຂັບເອງ', 'shared_ride': 'ເດີນທາງດ້ວຍລົດໃຫ່ຍ ຮ່ວມກັບລົດຄົນອຶ່ນ', 'e_car_drove_alone': 'ດ້ວຍການຂັບລົດໄຟຟ້າໄປເອງ', 'e_car_shared_ride': 'ດ້ວຍການຈ້າງລົດໄຟຟ້າໄປ', 'taxi': 'ແທັກຊີ', 'bus': 'ລົດເມ', 'train': 'ລົດໄຟ', 'free_shuttle': 'ລົດຮັບສົ່ງຟຣີ', 'air': 'ຍົນ', 'not_a_trip': 'ບໍ່ແມ່ນການເດີນທາງ', 'home': 'ບ້ານ', 'work': 'ໄປເຮັດວຽກ', 'at_work': 'ຢູ່ບ່ອນເຮັດວຽກ', 'school': 'ໄປໂຮງຮຽນ', 'transit_transfer': 'ການຖ່າຍໂອນການເດີນທາງ', 'shopping': 'ຊອບປິ້ງ', 'meal': 'ອາຫານ', 'pick_drop_person': 'ໄປຮັບ ຫລື ສົນ ຄົນ', 'pick_drop_item': 'ໄປຮັບ ຫລື ສົ່ງສິນຄ້າ', 'personal_med': 'ໄປຫາໝໍ', 'access_recreation': 'ເຂົ້າເຖິງການພັກຜ່ອນ', 'exercise': 'ພັກຜ່ອນ/ອອກກຳລັງກາຍ', 'entertainment': 'ບັນເທີງ/ສັງຄົມ', 'religious': 'ຈຸດປະສົງທາງສາດສະໜາ', 'other': 'ອື່ນໆ'}}})]

Generic Metrics Sensed notebook:

(emission) root@a969248e7123:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics_sensed.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/usaid-laos-ev.nrel-op.json
Successfully downloaded config with version 1 for USAID-NREL Support for Electric Vehicle Readiness and data collection URL https://USAID-laos-EV-openpath.nrel.gov/api/
Dynamic labels download was successful for nrel-openpath-deploy-configs: usaid-laos-ev
Running at 2024-01-18T20:14:28.655883+00:00 with args Namespace(plot_notebook='generic_metrics_sensed.ipynb', program='default', date=None) for range (<Arrow [2023-05-01T00:00:00+00:00]>, <Arrow [2024-01-01T00:00:00+00:00]>)
Running at 2024-01-18T20:14:28.696234+00:00 with params [Parameter('year', int), Parameter('month', int), Parameter('program', str, value='default'), Parameter('study_type', str, value='study'), Parameter('include_test_users', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')]

Results:

Charts:

Now, both the charts "Average Trips Length (selected by users)" and "Average Trip Length (sensed metrics)" are both available.

…, made the timestamp more visible.

iantei · 2024-01-23T18:38:23Z

Slight modification to the above changes to incorporate the following:

The timestamp is more visible, as it's pushed below the X-axis.
The font of label Count of the right Y-axis is identical to the label of Average (miles) on the left Y-axis.

Overall changes

Average Trip Length (sensed)

Average Trip Length (selected by users)

Summary of overall changes:

Incorporation of new function in Generic Metrics and Generic Metrics (sensed) notebook for Average Trip Length (selected by users) chart and Average Trip Length (sensed) chart respectively.
i. For Average Trip Length (selected by users), reverted changes as mentioned here:
Charts are not generated for Average Trips Length (selected by users) and Average Trip Length (sensed) metrics #107 (comment)
ii. For Average Trip Length (sensed), made identical changes with respect to the Average Trip Length in Generic Metrics.

shankari · 2024-01-26T04:50:22Z

I am not sure that this is the correct solution. Basically, if you have a metric that is showing up as blank, you have two options:

create the metric in the notebook OR
remove it from the dropdowns

This implements (1). However, we already have an average trip length metric (the one with > 3 entries per mode). I don't see the need for a second one. For the record, we had the unfiltered metric implemented before, but ran into issues in which we had one scootershare trip that was 10 miles (potentially mislabeled) and it threw off the entire y axis.

…art.

iantei · 2024-02-02T15:09:50Z

Summary of additional changes:

Dropped the metric - "Average Trip Length (selected by users)" from metrics_program.html, metrics_study.html, metrics_program_withoutEnergyMetrics and metrics_study_withoutEnergyMetrics.
Reverted changes in both plots.py and generic_metrics.ipynb for above Average Trip Length (selected by users).
Refactored notebook changes for Average Trip Length (sensed). Updated suffix in correspondence with other charts for the chart title.

Execution:

(emission) root@f8de8827c3f3:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics_sensed.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/usaid-laos-ev.nrel-op.json
Successfully downloaded config with version 1 for USAID-NREL Support for Electric Vehicle Readiness and data collection URL https://USAID-laos-EV-openpath.nrel.gov/api/
Dynamic labels download was successful for nrel-openpath-deploy-configs: usaid-laos-ev
Running at 2024-02-02T15:09:03.263404+00:00 with args Namespace(plot_notebook='generic_metrics_sensed.ipynb', program='default', date=None) for range (<Arrow [2023-05-01T00:00:00+00:00]>, <Arrow [2024-02-01T00:00:00+00:00]>)
Running at 2024-02-02T15:09:03.302315+00:00 with params [Parameter('year', int), Parameter('month', int), Parameter('program', str, value='default'), Parameter('study_type', str, value='study'), Parameter('include_test_users', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')]

Result:

Final changes

shankari · 2024-02-03T00:32:09Z

viz_scripts/plots.py

+def barplot_sensed_mode(data,x,y,plot_title, labels, file_name):
+    colours = dict(zip(labels, plt.cm.tab20.colors[:len(labels)]))


Why do we need this function? This is identical to barplot_mode. Why not just use barplot_mode?

Yes, I did realize the function is identical to barplot_mode. But I decided to keep an exclusively different function for the sensed chart generation, despite being identical because:

Following the existing pattern of function definition: The other existing function definition for pie_chart_mode is also identical to the pie_chart_sensed_mode function. This led me to think, if we need to make changes exclusively impacting the sensed charts, and not the non-sensed charts; having two distinct functions would be good decision.

This function barplot_sensed_mode can be modifiable for changes related with sensed charts, without having any impact for the non-sensed charts generated in other notebooks.

I understand having two identical functions barplot_sensed_mode & barplot_mode brings up code-duplication. But I see a trade-off for the approach of having two functions will help us leverage easy modification in future.

Please let me know what would be the right approach in this case.
I think, if there is a probability of having different ways in which sensed chart and non-sensed charts would be generated, it seems like a good trade-off to keep two identical functions. But if these two charts generation would always be identical, I think using a single function is a good idea.

Based on our discussion, I feel it's ideal to remove code-duplication and not engineer for future.
I have removed the redundant barplot_sensed_mode(), and replaced the call with barplot_mode().

Documenting the discussion here for the future.

But I see a trade-off for the approach of having two functions will help us leverage easy modification in future.
But if these two charts generation would always be identical, I think using a single function is a good idea.

I do not see how this will leverage easy modification in the future. If we do need to modify in the future, we will still need to test the modified implementation, whether we generate the new implementation by copying the existing implementation at that time or now. Once we know what the potential changes are, we may also choose to implement the changes by refactoring the current implementation instead of copying, it depends on the changes required. This is pretty much the textbook definition of DRY and over-engineering, and should be removed now.

iantei · 2024-02-08T07:36:21Z

Test scenario:

Dataset used: openpath_prod_usaid_laos_ev

Executed the below script to run generic_metrics_sensed notebook:

(emission) root@6820b88a5553:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics_sensed.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/usaid-laos-ev.nrel-op.json
Successfully downloaded config with version 1 for USAID-NREL Support for Electric Vehicle Readiness and data collection URL https://USAID-laos-EV-openpath.nrel.gov/api/
Dynamic labels download was successful for nrel-openpath-deploy-configs: usaid-laos-ev
Running at 2024-02-08T07:23:36.057174+00:00 with args Namespace(plot_notebook='generic_metrics_sensed.ipynb', program='default', date=None) for range (<Arrow [2023-05-01T00:00:00+00:00]>, <Arrow [2024-02-01T00:00:00+00:00]>)
Running at 2024-02-08T07:23:36.102561+00:00 with params [Parameter('year', int), Parameter('month', int), Parameter('program', str, value='default'), Parameter('study_type', str, value='study'), Parameter('include_test_users', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')]
Running at 2024-02-08T07:23:43.010974+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=5), Parameter('program', str, value='default'), Parameter('study_type', str, value='study'), Parameter('include_test_users', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')]
Running at 2024-02-08T07:23:48.694208+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=6), Parameter('program', str, value='default'), Parameter('study_type', str, value='study'), Parameter('include_test_users', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')]
Running at 2024-02-08T07:23:54.575169+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=7), Parameter('program', str, value='default'), Parameter('study_type', str, value='study'), Parameter('include_test_users', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')]
Running at 2024-02-08T07:23:59.816921+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=8), Parameter('program', str, value='default'), Parameter('study_type', str, value='study'), Parameter('include_test_users', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')]
Running at 2024-02-08T07:24:04.881566+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=9), Parameter('program', str, value='default'), Parameter('study_type', str, value='study'), Parameter('include_test_users', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')]
Running at 2024-02-08T07:24:10.340430+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=10), Parameter('program', str, value='default'), Parameter('study_type', str, value='study'), Parameter('include_test_users', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')]
Running at 2024-02-08T07:24:15.355431+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=11), Parameter('program', str, value='default'), Parameter('study_type', str, value='study'), Parameter('include_test_users', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')]
Running at 2024-02-08T07:24:20.807435+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=12), Parameter('program', str, value='default'), Parameter('study_type', str, value='study'), Parameter('include_test_users', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')]
Running at 2024-02-08T07:24:26.403548+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=1), Parameter('program', str, value='default'), Parameter('study_type', str, value='study'), Parameter('include_test_users', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')]
Running at 2024-02-08T07:24:32.047919+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=2), Parameter('program', str, value='default'), Parameter('study_type', str, value='study'), Parameter('include_test_users', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned')]

Result:

Result: Average trip length (sensed)

The above result shows the chart being generated for Average trip length (sensed) while also de-listing Average trip length (selected by users) from the list of Metrics in the public dashboard.

shankari · 2024-02-09T21:17:11Z

viz_scripts/plots.py

Extraneous changes. This entire file does not have any real changes, and should not appear in the PR. It's commit history should not be changed.

I have reverted all the extraneous changes in this file. The file doesn't show up under "Files Changed.".
I tried to execute git checkout <commit_before_this_PR_change> -- viz_scripts/plots.py, but this didn't show any staging file.
I am not sure if the commit history would be affected with this, though.

shankari · 2024-02-10T04:21:08Z

Squash-merging to avoid messing up the commit history

Update generic metrics for Average Miles for each mode (selected by u…

0b34e1b

…sers) chart, and update generic metrics sensed notebook for Average miles per transport mode selected (primary_mode).

Updated plt.txt for barplot_mode2() to be identical to barplot_mode()…

f07056f

…, made the timestamp more visible.

iantei marked this pull request as ready for review January 23, 2024 18:41

iantei added 3 commits February 2, 2024 07:42

Revert changes for Average Miles for each mode (selected by users) ch…

0502995

…art.

Remove the metric Average Miles for each mode (selected by users).

c26ebcd

Refactor changes for Average Trip Length (sensed).

6a6b85b

shankari requested changes Feb 3, 2024

View reviewed changes

iantei added 2 commits February 8, 2024 00:16

Removed code-duplication of barplot_sensed_mode().

6442e7b

Replace call to barplot_sensed_mode() with barplot_mode().

f9cad42

shankari requested changes Feb 9, 2024

View reviewed changes

iantei added 4 commits February 9, 2024 17:13

Reverting extraneous changes.

6629abb

Revert extraneous changes to viz_scripts/plots.py

6a373aa

Revert viz_scripts/plots.py to original state.

b8d2f10

Revert viz_scripts/plots.py to original state. Forgot a space last time.

afa03aa

Abby-Wheelis mentioned this pull request Feb 10, 2024

Distinguish Imperial vs Metric Units based on the config #114

Merged

shankari approved these changes Feb 10, 2024

View reviewed changes

shankari merged commit 824ffeb into e-mission:main Feb 10, 2024

shankari mentioned this pull request Feb 11, 2024

Unify color scheme for labels #117

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update generic metrics for Average Miles for each mode (selected by u… #113

Update generic metrics for Average Miles for each mode (selected by u… #113

iantei commented Jan 18, 2024

iantei commented Jan 18, 2024

iantei commented Jan 23, 2024 •

edited

Loading

shankari commented Jan 26, 2024

iantei commented Feb 2, 2024

shankari Feb 3, 2024

iantei Feb 3, 2024

iantei Feb 8, 2024

shankari Feb 9, 2024

iantei commented Feb 8, 2024

shankari Feb 9, 2024

iantei Feb 10, 2024 •

edited

Loading

shankari commented Feb 10, 2024

		def barplot_sensed_mode(data,x,y,plot_title, labels, file_name):
		colours = dict(zip(labels, plt.cm.tab20.colors[:len(labels)]))

Update generic metrics for Average Miles for each mode (selected by u… #113

Update generic metrics for Average Miles for each mode (selected by u… #113

Conversation

iantei commented Jan 18, 2024

iantei commented Jan 18, 2024

iantei commented Jan 23, 2024 • edited Loading

shankari commented Jan 26, 2024

iantei commented Feb 2, 2024

shankari Feb 3, 2024

Choose a reason for hiding this comment

iantei Feb 3, 2024

Choose a reason for hiding this comment

iantei Feb 8, 2024

Choose a reason for hiding this comment

shankari Feb 9, 2024

Choose a reason for hiding this comment

iantei commented Feb 8, 2024

shankari Feb 9, 2024

Choose a reason for hiding this comment

iantei Feb 10, 2024 • edited Loading

Choose a reason for hiding this comment

shankari commented Feb 10, 2024

iantei commented Jan 23, 2024 •

edited

Loading

iantei Feb 10, 2024 •

edited

Loading