Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update the measurements creation to use parquet/dask rather than arrow/vaex files #800

Merged
merged 66 commits into from
Jan 29, 2025

Conversation

ddobie
Copy link
Contributor

@ddobie ddobie commented Jan 23, 2025

To do:

  • Update the docs to remove all references to vaex and arrow files
  • Merge dependency upgrades

Notes:
The log files aren't displaying properly in the web interface. The interface can find the files, but not display them, e.g.
image

I realised that the measurement pairs parquet files don't need to be generated (in fact doing so breaks everything because they already exist). So there's a few changes that need to be made to the code (mostly done) and the nomenclature ("Generate parquet files" -> "Generate measurements parquet")

Screenshots that need updating:

  • docs/using/genparquet.md parquet-files-available.png
  • docs/using/genparquet.md generate-parquet-button.png
  • docs/using/genparquet.md generate-parquet-modal.png
  • docs/using/genparquet.md generate-parquet-notification.png
  • docs/using/genparquet.md generate-parquet-files-log

Other things to update

  • Update run config

@ddobie ddobie mentioned this pull request Jan 24, 2025
@ddobie ddobie marked this pull request as ready for review January 28, 2025 22:49
@ddobie ddobie requested a review from mauch January 28, 2025 22:49
Copy link
Contributor

@mauch mauch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Just some minor typos and a suggestion to make a separate function to delete files or directories.

docs/using/genparquet.md Outdated Show resolved Hide resolved
docs/using/runconfig.md Outdated Show resolved Hide resolved
vast_pipeline/management/commands/createmeasparquet.py Outdated Show resolved Hide resolved
vast_pipeline/management/commands/runpipeline.py Outdated Show resolved Hide resolved
vast_pipeline/management/commands/runpipeline.py Outdated Show resolved Hide resolved
@ddobie
Copy link
Contributor Author

ddobie commented Jan 29, 2025

I've implemented the suggested changes. Also fixed a few minor bugs and warnings that I hadn't picked up the first time I tested it.

@ddobie ddobie requested a review from mauch January 29, 2025 01:37
mauch
mauch approved these changes Jan 29, 2025
@ddobie ddobie removed the do not merge Do not merge this PR label Jan 29, 2025
@ddobie ddobie merged commit c8c25e1 into v2.0 Jan 29, 2025
4 checks passed
@ddobie ddobie deleted the v2-measurements-creation branch January 29, 2025 02:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants