Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store sticky pages separately #12198

Merged
merged 10 commits into from
Dec 4, 2024
Merged

Store sticky pages separately #12198

merged 10 commits into from
Dec 4, 2024

Conversation

kunga
Copy link
Member

@kunga kunga commented Dec 2, 2024

Changelog entry

...

Changelog category

  • Not for changelog (changelog entry is not required)

Additional information

...

@kunga kunga changed the title wip storing sticky pages separately Move sticky pages logic out of private cache Dec 2, 2024
Copy link

github-actions bot commented Dec 2, 2024

2024-12-02 08:30:00 UTC Pre-commit check linux-x86_64-release-asan for cea5cad has started.
2024-12-02 08:30:11 UTC Artifacts will be uploaded here
2024-12-02 08:33:11 UTC ya make is running...
🟡 2024-12-02 09:41:33 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13554 13459 0 41 7 47

🟢 2024-12-02 09:42:39 UTC Build successful.
🟢 2024-12-02 09:43:06 UTC ydbd size 4.9 GiB changed* by +56.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 9bef86a merge: cea5cad diff diff %
ydbd size 5 290 521 624 Bytes 5 290 579 728 Bytes +56.7 KiB +0.001%
ydbd stripped size 1 364 839 376 Bytes 1 364 851 152 Bytes +11.5 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Dec 2, 2024

2024-12-02 08:30:28 UTC Pre-commit check linux-x86_64-relwithdebinfo for cea5cad has started.
2024-12-02 08:30:38 UTC Artifacts will be uploaded here
2024-12-02 08:33:35 UTC ya make is running...
🟡 2024-12-02 09:27:01 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
21251 19716 0 23 1397 115

2024-12-02 09:28:45 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-12-02 09:40:30 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
804 (only retried tests) 676 0 20 0 108

2024-12-02 09:40:43 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-12-02 09:52:14 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
704 (only retried tests) 576 0 20 0 108

🟢 2024-12-02 09:52:22 UTC Build successful.
🟢 2024-12-02 09:52:44 UTC ydbd size 2.5 GiB changed* by +27.1 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 9bef86a merge: cea5cad diff diff %
ydbd size 2 691 439 344 Bytes 2 691 467 088 Bytes +27.1 KiB +0.001%
ydbd stripped size 483 344 080 Bytes 483 350 096 Bytes +5.9 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Dec 2, 2024

2024-12-02 15:47:00 UTC Pre-commit check linux-x86_64-relwithdebinfo for 72b7dab has started.
2024-12-02 15:47:12 UTC Artifacts will be uploaded here
2024-12-02 15:50:10 UTC ya make is running...
2024-12-02 16:07:30 UTC Check cancelled

Copy link

github-actions bot commented Dec 2, 2024

2024-12-02 15:49:59 UTC Pre-commit check linux-x86_64-release-asan for 72b7dab has started.
2024-12-02 15:50:11 UTC Artifacts will be uploaded here
2024-12-02 15:53:14 UTC ya make is running...
2024-12-02 16:07:16 UTC Check cancelled

Copy link

github-actions bot commented Dec 2, 2024

2024-12-02 16:11:53 UTC Pre-commit check linux-x86_64-relwithdebinfo for accb152 has started.
2024-12-02 16:12:30 UTC Artifacts will be uploaded here
2024-12-02 16:16:04 UTC ya make is running...
🟡 2024-12-02 17:10:06 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
21282 19773 0 3 1395 111

2024-12-02 17:11:50 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-12-02 17:30:09 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
682 (only retried tests) 573 0 2 1 106

2024-12-02 17:30:20 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-12-02 17:41:38 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
635 (only retried tests) 528 0 2 0 105

🟢 2024-12-02 17:41:47 UTC Build successful.
🟢 2024-12-02 17:42:10 UTC ydbd size 2.5 GiB changed* by +77.1 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: a1f5cb2 merge: accb152 diff diff %
ydbd size 2 692 239 088 Bytes 2 692 318 056 Bytes +77.1 KiB +0.003%
ydbd stripped size 483 446 416 Bytes 483 468 112 Bytes +21.2 KiB +0.004%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Dec 2, 2024

2024-12-02 16:12:08 UTC Pre-commit check linux-x86_64-release-asan for accb152 has started.
2024-12-02 16:12:19 UTC Artifacts will be uploaded here
2024-12-02 16:15:14 UTC ya make is running...
🟡 2024-12-02 17:30:15 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13562 13496 0 12 9 45

🟢 2024-12-02 17:31:27 UTC Build successful.
🟡 2024-12-02 17:31:53 UTC ydbd size 4.9 GiB changed* by +121.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: a1f5cb2 merge: accb152 diff diff %
ydbd size 5 292 094 392 Bytes 5 292 219 072 Bytes +121.8 KiB +0.002%
ydbd stripped size 1 365 094 800 Bytes 1 365 119 440 Bytes +24.1 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Dec 3, 2024

2024-12-03 08:00:54 UTC Pre-commit check linux-x86_64-release-asan for 291b753 has started.
2024-12-03 08:01:05 UTC Artifacts will be uploaded here
2024-12-03 08:04:00 UTC ya make is running...
2024-12-03 08:05:02 UTC Check cancelled

Copy link

github-actions bot commented Dec 3, 2024

2024-12-03 08:00:54 UTC Pre-commit check linux-x86_64-relwithdebinfo for 291b753 has started.
2024-12-03 08:01:05 UTC Artifacts will be uploaded here
2024-12-03 08:04:05 UTC ya make is running...
2024-12-03 08:05:02 UTC Check cancelled

Copy link

github-actions bot commented Dec 3, 2024

2024-12-03 08:08:54 UTC Pre-commit check linux-x86_64-relwithdebinfo for 097790a has started.
2024-12-03 08:09:05 UTC Artifacts will be uploaded here
2024-12-03 08:12:00 UTC ya make is running...
2024-12-03 08:17:26 UTC Check cancelled

Copy link

github-actions bot commented Dec 3, 2024

2024-12-03 08:09:36 UTC Pre-commit check linux-x86_64-release-asan for 097790a has started.
2024-12-03 08:09:47 UTC Artifacts will be uploaded here
2024-12-03 08:12:53 UTC ya make is running...
2024-12-03 08:17:23 UTC Check cancelled

Copy link

github-actions bot commented Dec 3, 2024

2024-12-03 08:21:10 UTC Pre-commit check linux-x86_64-release-asan for 6a00d1d has started.
2024-12-03 08:21:55 UTC Artifacts will be uploaded here
2024-12-03 08:25:48 UTC ya make is running...
🟡 2024-12-03 09:33:23 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13562 13488 0 18 7 49

🟢 2024-12-03 09:34:31 UTC Build successful.
🟢 2024-12-03 09:35:03 UTC ydbd size 4.9 GiB changed* by +27.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: d939c0f merge: 6a00d1d diff diff %
ydbd size 5 293 109 408 Bytes 5 293 137 272 Bytes +27.2 KiB +0.001%
ydbd stripped size 1 365 363 600 Bytes 1 365 366 864 Bytes +3.2 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Dec 3, 2024

2024-12-03 08:21:45 UTC Pre-commit check linux-x86_64-relwithdebinfo for 6a00d1d has started.
2024-12-03 08:21:58 UTC Artifacts will be uploaded here
2024-12-03 08:25:08 UTC ya make is running...
🟡 2024-12-03 09:19:45 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
21282 19764 0 4 1397 117

2024-12-03 09:21:30 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-12-03 09:33:24 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
838 (only retried tests) 729 0 1 0 108

2024-12-03 09:33:36 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-12-03 09:45:19 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
674 (only retried tests) 566 0 1 0 107

🟢 2024-12-03 09:45:31 UTC Build successful.
🟢 2024-12-03 09:45:51 UTC ydbd size 2.5 GiB changed* by +15.3 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: d939c0f merge: 6a00d1d diff diff %
ydbd size 2 692 775 336 Bytes 2 692 791 000 Bytes +15.3 KiB +0.001%
ydbd stripped size 483 543 184 Bytes 483 547 024 Bytes +3.8 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@kunga kunga changed the title Move sticky pages logic out of private cache Keep sticky pages separately Dec 3, 2024
@kunga kunga self-assigned this Dec 3, 2024
Copy link

github-actions bot commented Dec 3, 2024

2024-12-03 10:49:39 UTC Pre-commit check linux-x86_64-relwithdebinfo for 7cad624 has started.
2024-12-03 10:49:51 UTC Artifacts will be uploaded here
2024-12-03 10:52:57 UTC ya make is running...
🟡 2024-12-03 12:02:45 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
21287 19767 0 11 1396 113

2024-12-03 12:04:31 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-12-03 12:16:01 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
813 (only retried tests) 706 0 0 0 107

🟢 2024-12-03 12:16:11 UTC Build successful.
🟢 2024-12-03 12:16:33 UTC ydbd size 2.5 GiB changed* by +57.6 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: e7cb027 merge: 7cad624 diff diff %
ydbd size 2 692 792 424 Bytes 2 692 851 448 Bytes +57.6 KiB +0.002%
ydbd stripped size 483 545 648 Bytes 483 561 680 Bytes +15.7 KiB +0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Dec 3, 2024

2024-12-03 10:50:04 UTC Pre-commit check linux-x86_64-release-asan for 7cad624 has started.
2024-12-03 10:50:16 UTC Artifacts will be uploaded here
2024-12-03 10:53:20 UTC ya make is running...
🟡 2024-12-03 12:20:08 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13567 13495 0 18 5 49

🟢 2024-12-03 12:21:19 UTC Build successful.
🟢 2024-12-03 12:21:49 UTC ydbd size 4.9 GiB changed* by +84.4 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: e7cb027 merge: 7cad624 diff diff %
ydbd size 5 293 166 872 Bytes 5 293 253 344 Bytes +84.4 KiB +0.002%
ydbd stripped size 1 365 377 232 Bytes 1 365 401 872 Bytes +24.1 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Dec 3, 2024

2024-12-03 14:18:32 UTC Pre-commit check linux-x86_64-relwithdebinfo for f079e9e has started.
2024-12-03 14:18:37 UTC Artifacts will be uploaded here
2024-12-03 14:21:38 UTC ya make is running...
🟡 2024-12-03 15:24:14 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
21317 19805 0 2 1397 113

2024-12-03 15:26:00 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-12-03 15:37:37 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
720 (only retried tests) 614 0 0 0 106

🟢 2024-12-03 15:37:47 UTC Build successful.
🟢 2024-12-03 15:38:08 UTC ydbd size 2.5 GiB changed* by +24.7 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 7e246c6 merge: f079e9e diff diff %
ydbd size 2 692 960 600 Bytes 2 692 985 856 Bytes +24.7 KiB +0.001%
ydbd stripped size 483 576 560 Bytes 483 582 448 Bytes +5.8 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Dec 3, 2024

2024-12-03 14:19:39 UTC Pre-commit check linux-x86_64-release-asan for f079e9e has started.
2024-12-03 14:21:04 UTC Artifacts will be uploaded here
2024-12-03 14:24:33 UTC ya make is running...
🟡 2024-12-03 15:37:37 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13596 13523 0 19 5 49

🟢 2024-12-03 15:38:46 UTC Build successful.
🟢 2024-12-03 15:39:15 UTC ydbd size 4.9 GiB changed* by +52.0 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 7e246c6 merge: f079e9e diff diff %
ydbd size 5 293 660 400 Bytes 5 293 713 688 Bytes +52.0 KiB +0.001%
ydbd stripped size 1 365 534 032 Bytes 1 365 542 800 Bytes +8.6 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@@ -299,50 +286,11 @@ const TSharedData* TPrivatePageCache::Lookup(TPageId pageId, TInfo *info) {

if (page->Empty()) {
ToLoad.PushBack(page);

// Note: we mark flat index pages sticky before we load them
if (!page->Sticky && info->GetPageType(page->Id) == EPage::FlatIndex) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Решил что можно уже этим пожертвовать и помечать плоский индекс Sticky сразу при загрузке парта

Иначе на каждую загрузку страниц придётся их тип проверять, и потом ещё когда они загрузились тоже помещать их в Sticky

Пока же вышло, что все Sticky страницы появляются сразу при загрузки парта и потом не меняются (только разве что на alter с ECache::Ever

Но в будущем как будто можно все страницы считать в потребление не зависимо от их Sticky признака

@kunga kunga changed the title Keep sticky pages separately Store sticky pages separately Dec 3, 2024
Copy link

github-actions bot commented Dec 4, 2024

2024-12-04 08:13:12 UTC Pre-commit check linux-x86_64-release-asan for ad10367 has started.
2024-12-04 08:13:22 UTC Artifacts will be uploaded here
2024-12-04 08:16:21 UTC ya make is running...
🟡 2024-12-04 10:30:06 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13589 13104 0 243 185 57

🟢 2024-12-04 10:31:19 UTC Build successful.
🟢 2024-12-04 10:32:02 UTC ydbd size 4.9 GiB changed* by +52.3 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: d6a279e merge: ad10367 diff diff %
ydbd size 5 307 983 192 Bytes 5 308 036 752 Bytes +52.3 KiB +0.001%
ydbd stripped size 1 365 523 472 Bytes 1 365 532 240 Bytes +8.6 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Dec 4, 2024

2024-12-04 08:13:19 UTC Pre-commit check linux-x86_64-relwithdebinfo for ad10367 has started.
2024-12-04 08:13:30 UTC Artifacts will be uploaded here
2024-12-04 08:16:42 UTC ya make is running...
🟡 2024-12-04 09:21:49 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
21318 19764 0 4 1426 124

2024-12-04 09:23:34 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-12-04 09:35:07 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
775 (only retried tests) 669 0 0 0 106

🟢 2024-12-04 09:35:17 UTC Build successful.
🟢 2024-12-04 09:35:38 UTC ydbd size 2.5 GiB changed* by +24.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: d6a279e merge: ad10367 diff diff %
ydbd size 2 701 651 032 Bytes 2 701 676 400 Bytes +24.8 KiB +0.001%
ydbd stripped size 483 580 528 Bytes 483 586 480 Bytes +5.8 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@kunga kunga marked this pull request as ready for review December 4, 2024 08:14
@kunga kunga requested a review from snaury December 4, 2024 08:14
@kunga kunga merged commit fa3d243 into ydb-platform:main Dec 4, 2024
10 checks passed
@kunga kunga mentioned this pull request Dec 4, 2024
24 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants