Data Quality Monitor

[vanderburgt]

User story

As a journalist using Bron,
I want to access a Data Quality Monitor,
so that I can assess the completeness of data from various government organizations over time and identify potential gaps.

Description
Bron aggregates documents from multiple sources like ORI, PoliFLW, and open.overheid.nl, but data completeness is inconsistent across organizations and time periods. For example, certain municipalities may have missing data for specific years or lack documents entirely. To ensure I am aware of these gaps before conducting my research, I need a tool that provides a clear, interactive overview of the data quality.

Acceptance criteria

1. Overview by organization:

• The monitor should display a list of all government organizations (e.g., municipalities, water boards, ministries) with document coverage information.
• Organizations missing from the data entirely should be flagged.

2. Time-based analysis:

• The monitor should present data coverage broken down by week, month, and year for each organization.
• Time periods with missing or sparse documents should be visually highlighted (e.g., using color codes or patterns).

3. Filters:

• Users can filter the overview by organization type (e.g., municipalities only) or by source (ORI, PoliFLW, etc.).- Users can filter by a specific time range to focus on relevant periods.

Maybe:

5. Interactive visualization:

• Users should be able to interact with charts or tables to zoom in on specific organizations or time periods.
• Hovering or clicking on a gap should provide additional metadata (e.g., exact missing years, potential source issues).

6. Data quality score

• Data quality scoring per search/feed to indicate data completeness users can expect in a feed or search

[vanderburgt]

@Coretteket let's discuss this story and generate some ideas for the implementation.