Code

Use rove for QC in the ingestor#42

Code

Merged

intarga merged 30 commits intotrunkmetno/lard:trunkfrom rove_integrationmetno/lard:rove_integrationCopy head branch name to clipboardJan 29, 2025

+1,000-326

Commits

Commits on Jan 9, 2025

• ingestion: plumbing for QC and stub qc_data implementation

  intarga

  committed
  f3aa8ad

  Copy full SHA for f3aa8ad

• db: flag tables for confident flags

  intarga

  committed
  bb39130

  Copy full SHA for bb39130

• ingestion: implement qc_data

  intarga

  committed
  8684b5b

  Copy full SHA for 8684b5b

• ingestion: refactor Data to DataChunk, pulling up shared fields in Datum

  Show description for 66ea3d7
  intarga

  committed
  66ea3d7

  Copy full SHA for 66ea3d7

• ingestion: get time_resolution for qc from type_id

  intarga

  committed
  6556b35

  Copy full SHA for 6556b35

• use real QC pipelines

  intarga

  committed
  bd9be7e

  Copy full SHA for bd9be7e

• remove obsolete test case

  intarga

  committed
  72b0b97

  Copy full SHA for 72b0b97

Commits on Jan 13, 2025

• fix typo in index on flags.confident_provenance table

  intarga

  committed
  280309e

  Copy full SHA for 280309e

• fix ambiguous column identifier in qc_data query

  intarga

  committed
  9b133bb

  Copy full SHA for 9b133bb

• show the message from ingestor when the success assertion fails

  intarga

  committed
  d72e95b

  Copy full SHA for d72e95b

Commits on Jan 14, 2025

• add partitions for flag tables

  intarga

  committed
  776408a

  Copy full SHA for 776408a

• fix integration tests

  intarga

  committed
  5ef23c0

  Copy full SHA for 5ef23c0

• fix rustfmt complaint about empty file

  intarga

  committed
  299fc7e

  Copy full SHA for 299fc7e

• fix clippy lint

  intarga

  committed
  1fe4773

  Copy full SHA for 1fe4773

Commits on Jan 15, 2025

• fix schema ordering in e2e test setup

  intarga

  committed
  8caae2b

  Copy full SHA for 8caae2b

Commits on Jan 16, 2025

• merge confident.flags into public.data

  intarga

  committed
  1762bdf

  Copy full SHA for 1762bdf

Commits on Jan 17, 2025

• modify the one current qc pipeline to match the new format

  intarga

  committed
  12ce22c

  Copy full SHA for 12ce22c

• fix path to load qc pipelines from

  Show description for 150c653
  intarga

  committed
  150c653

  Copy full SHA for 150c653

Commits on Jan 21, 2025

• fix typo in provenance table constraint names

  intarga

  committed
  ace1a7b

  Copy full SHA for ace1a7b

• disable spike checks in fresh pipelines

  intarga

  committed
  352e5a8

  Copy full SHA for 352e5a8

• use rove connector to fetch context for a datum not currently in the db

  intarga

  committed
  c10c2bb

  Copy full SHA for c10c2bb

Commits on Jan 22, 2025

• expand todo about optimising permit checks

  intarga

  committed
  a8f8e25

  Copy full SHA for a8f8e25

• rename qc_data to qc_fresh data, add comments to reflect its specificity

  intarga

  committed
  ec1191e

  Copy full SHA for ec1191e

• extend first data partition back to 1700

  Show description for 1380280
  intarga

  committed
  1380280

  Copy full SHA for 1380280

Commits on Jan 23, 2025

• switch back to main branch of rove after merging fixes

  intarga

  committed
  27a7f6b

  Copy full SHA for 27a7f6b

• switch qc_usable to default to true instead of NULL

  intarga

  committed
  c6987f9

  Copy full SHA for c6987f9

• migrations: support qc_usable column

  intarga

  committed
  0d0d363

  Copy full SHA for 0d0d363

Commits on Jan 27, 2025

• Revert "migrations: support qc_usable column"

  Show description for 5996440
  intarga

  committed
  5996440

  Copy full SHA for 5996440

• db: add DEFAULT TRUE to public.data.qc_usable

  intarga

  committed
  228fd32

  Copy full SHA for 228fd32

Commits on Jan 28, 2025

• fix typo in comment

  intarga

  committed
  9faa557

  Copy full SHA for 9faa557