Use rove for QC in the ingestor #42

intarga · 2024-12-11T15:35:54Z

Schema changes to support flags
Plumbing
qc_data implementation
identify time_resolution from type_id
load and use real pipeline
Make sure mock pipeline is consistent with the real one
Merge confident.flags into public.data

This PR integrates rove to perform QC in the ingestor with the current limitations:

Only fresh pipelines are supported (not periodic or consistency)
We only have 1 actual pipeline (hourly air temperature), and even that has 2 checks disabled as we don't have access to model data
No caching of prior ingested observations

I considered verifying the QC output in the integration tests, but decided this was best left for when we implement an API that actually uses them (frost backend?) for now I manually queried the DB after running the integration tests to confirm qc_usable was as expected.

I restructured the generic Datum and DataChunk to match their obsinn equivalents. The original idea behind the different structure was that I thought they should be neutral towards different data sources that might structure messages differently, but I realised that the Obsinn structure of clustering by station and timestamp is what makes sense for any source of fresh data, so we should optimise for that.

There is a smaller PR of associated fixes in ROVE that should be merged before this one

Upon further consideration, I think matching obsinn's format like this makes sense. Since we are ingesting live data from weather stations, it's reasonable to expect a station will send observations fromall its params at once with the same timestamp, so we should optimise for this use case

Cargo.toml

in the future we probably want to have directories like qc_pipelines/periodic and qc_pipelines/consistency, but for now we don't handle that so we just load qc_pipelines/fresh directly

db/flags.sql

db/public.sql

ingestion/src/kldata.rs

ingestion/src/lib.rs

integration_tests/tests/end_to_end.rs

util/src/bin/generate_partition_queries.rs

integration_tests/src/main.rs

Manuel noted that some data in the migrations is older than 1850. This change is necessary to ensure we can ingest it

This reverts commit 0d0d363.

db/partitions_generated.sql

db/flags.sql

db/public.sql

ingestion/src/kldata.rs

ingestion/src/lib.rs

intarga added the enhancement New feature or request label Dec 12, 2024

intarga force-pushed the rove_integration branch 2 times, most recently from edf95f6 to 389f56a Compare December 13, 2024 17:05

intarga added 6 commits January 9, 2025 10:51

ingestion: plumbing for QC and stub qc_data implementation

f3aa8ad

db: flag tables for confident flags

bb39130

ingestion: implement qc_data

8684b5b

ingestion: get time_resolution for qc from type_id

6556b35

use real QC pipelines

bd9be7e

intarga force-pushed the rove_integration branch from 389f56a to bd9be7e Compare January 9, 2025 09:53

intarga added 10 commits January 9, 2025 13:14

remove obsolete test case

72b0b97

fix typo in index on flags.confident_provenance table

280309e

fix ambiguous column identifier in qc_data query

9b133bb

show the message from ingestor when the success assertion fails

d72e95b

add partitions for flag tables

776408a

fix integration tests

5ef23c0

fix rustfmt complaint about empty file

299fc7e

fix clippy lint

1fe4773

fix schema ordering in e2e test setup

8caae2b

merge confident.flags into public.data

1762bdf

intarga commented Jan 17, 2025

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

intarga added 2 commits January 17, 2025 15:32

modify the one current qc pipeline to match the new format

12ce22c

fix path to load qc pipelines from

150c653

in the future we probably want to have directories like qc_pipelines/periodic and qc_pipelines/consistency, but for now we don't handle that so we just load qc_pipelines/fresh directly

intarga mentioned this pull request Jan 17, 2025

Fixes associated with the lard integration metno/rove#94

Merged

intarga marked this pull request as ready for review January 17, 2025 15:16

intarga requested review from Lun4m and jo-asplin-met-no January 17, 2025 15:16

intarga linked an issue Jan 17, 2025 that may be closed by this pull request

Rove integration #16

Closed

Lun4m reviewed Jan 20, 2025

View reviewed changes

fix typo in provenance table constraint names

ace1a7b

intarga added 6 commits January 21, 2025 20:55

disable spike checks in fresh pipelines

352e5a8

use rove connector to fetch context for a datum not currently in the db

c10c2bb

expand todo about optimising permit checks

a8f8e25

rename qc_data to qc_fresh data, add comments to reflect its specificity

ec1191e

extend first data partition back to 1700

1380280

Manuel noted that some data in the migrations is older than 1850. This change is necessary to ensure we can ingest it

switch back to main branch of rove after merging fixes

27a7f6b

Lun4m approved these changes Jan 23, 2025

View reviewed changes

intarga added 4 commits January 23, 2025 21:38

switch qc_usable to default to true instead of NULL

c6987f9

migrations: support qc_usable column

0d0d363

Revert "migrations: support qc_usable column"

5996440

This reverts commit 0d0d363.

db: add DEFAULT TRUE to public.data.qc_usable

228fd32

jo-asplin-met-no reviewed Jan 28, 2025

View reviewed changes

db/partitions_generated.sql Show resolved Hide resolved

jo-asplin-met-no reviewed Jan 28, 2025

View reviewed changes

db/flags.sql Show resolved Hide resolved

jo-asplin-met-no reviewed Jan 28, 2025

View reviewed changes

db/public.sql Outdated Show resolved Hide resolved

jo-asplin-met-no reviewed Jan 28, 2025

View reviewed changes

ingestion/src/kldata.rs Show resolved Hide resolved

fix typo in comment

9faa557