Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DNM] sql: add fast checks for memo staleness #137217

Closed
wants to merge 3 commits into from
Closed

[DNM] sql: add fast checks for memo staleness #137217

wants to merge 3 commits into from

Conversation

fqazi
Copy link
Collaborator

@fqazi fqazi commented Dec 10, 2024

This patch adds the following:

  1. Updates the lease manager to expose a generation ID which allows you to know if any descriptors changed
  2. Updates the table stats cache to add a generation ID to determine if any new stats were added
  3. Updates the IsStale check to confirm if either the generation IDs have changed, search_path has changed or current database has changed. If everything was the same in the last execution, the full CheckDependencies can be skipped
  4. Updates the schema changer logic to lease new descriptors if the schema is already leased. This allows invalidation to work within a search path if new objects are made.

@cockroach-teamcity
Copy link
Member

This change is Reviewable

@fqazi fqazi force-pushed the memoIsStaleOptimize branch 4 times, most recently from 4da69a1 to 4b4b8f3 Compare December 12, 2024 21:32
fqazi added 3 commits December 13, 2024 12:04
@fqazi
catalog/lease: add a fast way to detect descriptor changes
57bd5af 
Previously, we had no way of know if any new descriptor versions exist
without asking to lease them out again. This could make memo staleness
checks for queries expensive, since they would need to re-resolve every
single descriptor. To address this, this patch adds the idea of a lease
generation counter that is incremented every time a new descriptor
version is available. Also to make it easier to detect if new objects
exist within a search path, this patch forces object creators to wait
for the lease manager to acknowledge the existence of new descriptors
(specifically ones that are created in an already resolved schema).

Release note: None
@fqazi
sql/stats: add a fast way to detect statistics changes
93ecd09 
Previously, the only way to know if any of the statistics changed was to
query the statistics cache for a specific table. To make memo staleness
checks cheaper, this patch adds a generation count to the statistics
cache, which will be modified on any statistics update.

Release note: None
@fqazi
sql: add and adopt fast memo staleness checks
05b2fca 
Previously, determining if a memo was stale involved re-resolving
descriptor references to ensure that the versions and descriptor IDs
were the same. This could be fairly expensive and hit various points of
contention. To address this, this patch takes advantage of the
lease manager and stats cache generation to determine if descriptors
have and statistics have to be re-resolved. It does so by introducing a
fingerprint that will contain this generation information, current
database, system zone config and search_path. If none of these things
change, then we know that a memo can be re-used without any extra checks.

Release note: None
Fixes: cockroachdb#105867
@fqazi fqazi force-pushed the memoIsStaleOptimize branch from 4b4b8f3 to 05b2fca Compare December 13, 2024 17:16
@fqazi fqazi closed this Dec 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@fqazi @cockroach-teamcity