Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pageserver: make heatmap generation additive #10597

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

Conversation

VladLazar
Copy link
Contributor

Problem

Previously, when cutting over to cold secondary locations,
we would clobber the previous, good, heatmap with a cold one.
This is because heatmap generation used to include only resident layers.

Once this merges, we can add an endpoint which triggers full heatmap
hydration on attached locations to heal cold migrations.

Summary of changes

With this patch, heatmap generation becomes additive. If we have a
heatmap from when this location was secondary, the new uploaded heatmap
will be the result of a reconciliation between the old one and the on
disk resident layers.

More concretely, when we have the previous heatmap:

  1. Filter the previous heatmap and keep layers that are (a) present
    in the current layer map, (b) visible, (c) not resident. Call this set
    of layers visible_non_resident.
  2. From the layer map, select all layers that are resident and visible.
    Call this set of layers resident.
  3. The new heatmap is the result of merging the two disjoint sets.

Related #10541

While in secondary mode tenants persist the heatmap on disk.
This patch loads it up when attaching the tenant and threads
it down to the `Timeline` struct.

Follow-up patches will reconcile with the old heatmap before
when uploading a new one.
Previously, when cutting over to cold secondary locations,
we would clobber the previous, good, heatmap with a cold one.
This is because heatmap generation used to include only resident layers.

With this patch, heatmap generation becomes additive. If we have a
heatmap from when this location was secondary, the new uploaded heatmap
will be the result of a reconciliation between the old one and the on
disk resident layers.

More concretely, when we have the previous heatmap:
1. Filter the previous heatmap and keep layers that are (a) present
in the current layer map, (b) visible, (c) not resident. Call this set
of layers `visible_non_resident`.
2. From the layer map, select all layers that are resident and visible.
Call this set of layers `resident`.
3. The new heatmap is the result of merging the two disjoint sets.
Copy link

6750 tests run: 6434 passed, 0 failed, 316 skipped (full report)


Flaky tests (5)

Postgres 17

Postgres 16

Test coverage report is not available

The comment gets automatically updated with the latest test results
82b550a at 2025-01-30T21:18:32.791Z :recycle:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant