Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix for Data Quality perf and issue other tweaks. #11711

Merged
merged 15 commits into from
Nov 29, 2024
Merged

Conversation

jdunkerley
Copy link
Member

@jdunkerley jdunkerley commented Nov 29, 2024

Pull Request Description

  • Fix bug where DB_Table data quality indicators broke deserialization in the table viz.
  • Memorization of the untrimmed data quality indicator and move to it being an operation and column function.
  • If more than 10,000 rows then use a sample for untrimmed.
  • ALIASes for blank functions.
  • Fix for Snowflake drill down.
  • Bug fix for Long and Double columns with Nothings at end.

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

  • The documentation has been updated, if necessary.
  • Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.
  • All code follows the
    Scala,
    Java,
    TypeScript,
    and
    Rust
    style guides. In case you are using a language not listed above, follow the Rust style guide.
  • Unit tests have been written where possible.
  • If meaningful changes were made to logic or tests affecting Enso Cloud integration in the libraries,
    or the Snowflake database integration, a run of the Extra Tests has been scheduled.
    • If applicable, it is suggested to paste a link to a successful run of the Extra Tests.

@jdunkerley jdunkerley marked this pull request as ready for review November 29, 2024 12:00
Copy link
Member

@AdRiley AdRiley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to add tests for any of the new functionality?

@jdunkerley
Copy link
Member Author

Is it possible to add tests for any of the new functionality?

Tests added for the table viz have been updated to cover the DB Table and sampled data quality indicator.
Will add a test for the column slice.

@jdunkerley jdunkerley added the CI: No changelog needed Do not require a changelog entry for this PR. label Nov 29, 2024
Copy link
Member

@radeusgd radeusgd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good now, mostly some small nitpicks that I think may be worth adding but not necessary.

Only thing I'd like to fix before merging is the computation of countUntrimmed if the cached future was cancelled. Currently it runs on background thread but tries to fetch the context - that is bound to fail. We should just compute it on main thread if the background one was cancelled and it is being requested right now - deferring to background blocks the main thread until background thread is free to execute a new job, which may theoretically be a pretty long time.

@jdunkerley jdunkerley added the CI: Ready to merge This PR is eligible for automatic merge label Nov 29, 2024
@mergify mergify bot merged commit 85c8f76 into develop Nov 29, 2024
37 of 38 checks passed
@mergify mergify bot deleted the wip/jd/table-viz-perf branch November 29, 2024 17:54
jdunkerley added a commit that referenced this pull request Nov 29, 2024
- Fix bug where `DB_Table` data quality indicators broke deserialization in the table viz.
- Memorization of the untrimmed data quality indicator and move to it being an operation and column function.
- If more than 10,000 rows then use a sample for untrimmed.
- ALIASes for blank functions.
- Fix for Snowflake drill down.
- Bug fix for Long and Double columns with Nothings at end.

(cherry picked from commit 85c8f76)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI: No changelog needed Do not require a changelog entry for this PR. CI: Ready to merge This PR is eligible for automatic merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants