Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HIVE-28773: CBO fails when a materialized view is dropped but its pre-compiled plan remains in the registry. #5665

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

kasakrisz
Copy link
Contributor

@kasakrisz kasakrisz commented Feb 28, 2025

What changes were proposed in this pull request?

  • The materialized view registry is initialized by loading all MVs from HMS that are enabled for query rewrite and precompiled. Later, a scheduled thread refreshes the registry periodically in the same way. This patch adds logic to remove MVs from the registry that no longer exist in HMS or have been disabled for query rewrite since the last refresh.
  • Rename MaterializedViewCache to MaterializedViewMap because it is not a cache.
  • Modify the way MVs are stored in the map: remove the database grouping and add the database name as a key prefix to the MV name. Since all MVs are queried and added to the planner during query rewrite, this approach aligns better with the most common function of the registry/map.

Why are the changes needed?

Dropped MVs may casue CBO failure. See jira HIVE-28773 for details

Does this PR introduce any user-facing change?

Yes. CBO failure is intermittent in clusters with multiple HS2s and depending on the query the whole compilation process could fail. This patch fixes these intermittent issues.

Is the change a dependency upgrade?

No.

How was this patch tested?

mvn test -Dtest=TestQueryRewrite,TestHiveMaterializedViewRegistry -pl itests/hive-unit -Pitests
mvn test -Dtest=TestMaterializedViewMap -pl ql

@kasakrisz kasakrisz force-pushed the HIVE-28773-master-mv-reg2 branch from 4f27409 to 3ce1cc6 Compare March 3, 2025 08:25
@kasakrisz kasakrisz marked this pull request as ready for review March 3, 2025 08:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants