Purge + Restore user timeseries data with long-term storage by MukuFlash03 · Pull Request #952 · e-mission/e-mission-server

Commits on Jan 31, 2023

feat: Add timeseries purge script
paultranvan
committed

Commits on Mar 28, 2023

Added csv export feature to bin/purge_user_timeseries
TTalex
committed

Commits on Jan 8, 2024

Replaced print() with logging.debug()

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Merge remote-tracking branch 'ttalex/purge-timeseries' into purge-restore-timeseries

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Storing data as JSON + Restore code added

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed

Commits on Jan 9, 2024

Current working code for JSON based purge/restore of data

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Added CSV export as an option

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed

Commits on Jan 11, 2024

Added more tests for purge and restore operations

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Updated test file path

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Updated default directory path

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed

Commits on Jan 19, 2024

Added import options

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Added conditional checks before purging

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Remove print

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed

Commits on Jul 9, 2024

Testing changes to get TestExportModule to work

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed

Commits on Jul 10, 2024

Testing adding database argument to export script

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed

Commits on Jul 11, 2024

Trying to run export using export pipeline + Added logging statements

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Executing the purge process via the export pipeline now. Added Test file as well. Some pointers: had to comment out logging.config.dictConfig in export_stage as it was giving a logging has no modul…

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Trying to make the entire flow work : export -> purge -> restore

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Cleaned up code before pushing commits.

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Added TODO comment to Test file

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Fixed datetime module + Cleaned up redundant changes from other PRs

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed

Commits on Jul 31, 2024

Removing changes made to original export PR scripts.

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Removed newline change from .gitignore

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Added purge + restore pipeline implementation + Corrected EXPORT_DATA pipeline state to use unique value.

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Added purge + restore pipeline implementation + Corrected EXPORT_DATA pipeline state to use unique value.

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Removed import_timeseries import - no longer using this file.

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Skipping export, purge, restore stages in pipeline test.

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed

Commits on Aug 1, 2024

Removed purge/restore standalone scripts; can reuse existing extract/load scripts.

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Added flag to existing extract script to allow purging

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Removed continue_on_error parameter to load timeseries function

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Deleting unused Test file

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed

Commits on Aug 29, 2024

Draft commit - Testing 1 hour incremental export - some entries missing

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Implemented Full + Incremental export + purge + restore

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed

Commits on Aug 30, 2024

Revert "Added flag to existing extract script to allow purging"

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Added a new import_timeseries that replicates load_multi_timeline_for_range

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed

Commits on Aug 31, 2024

Reduced number of files by merging data using batch_size

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Draft commit -> Fix full export batch size limit + Trying to fix incremental

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Removed batch size limit

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Shortened core logic + Added tests to check file contents

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Draft commit - added print statements to Test; pending restore multiple times test

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Revert "Shortened core logic + Added tests to check file contents"

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed
Tests for assserting few entries after export

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed

Commits on Sep 1, 2024

Added more tests for comparing entries from db and export files

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed

Commits on Sep 16, 2024

Cleaned up duplicate code, log statements + Refactored export.py

Mahadik, Mukul Chandrakant
authored and
Mahadik, Mukul Chandrakant
committed

Commits

Commits on Jan 31, 2023

Commits on Mar 28, 2023

Commits on Jan 8, 2024

Commits on Jan 9, 2024

Commits on Jan 11, 2024

Commits on Jan 19, 2024

Commits on Jul 9, 2024

Commits on Jul 10, 2024

Commits on Jul 11, 2024

Commits on Jul 31, 2024

Commits on Aug 1, 2024

Commits on Aug 2, 2024

Commits on Aug 9, 2024

Commits on Aug 27, 2024

Commits on Aug 28, 2024

Commits on Aug 29, 2024

Commits on Aug 30, 2024

Commits on Aug 31, 2024

Commits on Sep 1, 2024

Commits on Sep 16, 2024