pageserver: fix race cleaning up timeline files when shut down during bootstrap

[jcsp]

Problem

Timeline bootstrap starts a flush loop, but doesn't reliably shut down the timeline (incl. waiting for flush loop to exit) before destroying UninitializedTimeline, and that destructor tries to clean up local storage. If local storage is still being written to, then this is unsound.

Currently the symptom is that we see a "Directory not empty" error log, e.g. https://neon-github-public-dev.s3.amazonaws.com/reports/main/12966756686/index.html#testresult/5523f7d15f46f7f7/retries

Summary of changes

• Move fallible IO part of bootstrap into a function (notably, this is fallible in the case of the tenant being shut down while creation is happening)
• When that function returns an error, call shutdown() on the timeline

[github-actions]

7414 tests run: 7063 passed, 0 failed, 351 skipped (full report)

---

Flaky tests (7)

Postgres 17

• test_pgdata_import_smoke[8-1024-RelBlockSize.MULTIPLE_RELATION_SEGMENTS]: debug-x86-64-without-lfc, release-arm64-without-lfc, release-arm64-with-lfc
• test_pgdata_import_smoke[None-1024-RelBlockSize.MULTIPLE_RELATION_SEGMENTS]: debug-x86-64-without-lfc, release-arm64-without-lfc, release-arm64-with-lfc

Postgres 14

• test_location_conf_churn[3]: release-arm64-with-lfc

Code coverage* (full report)

• functions: 33.3% (8496 of 25492 functions)
• lines: 49.1% (71432 of 145523 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
65b60cb at 2025-01-29T22:18:53.197Z :recycle:}

[skyzh]

the test_import_from_vanilla test failure doesn't seem transient?

[jcsp]

the test_import_from_vanilla test failure doesn't seem transient?

Yeah, the issue was more pervasive than I realised, so once I made the drop() strict about checking for shutdown, other things started failing.

Reworked this to more reliably enforce rules about shutdown, by wrapping everything in an UninitializedTimeline.write() helper that spawns flush loop but also ensures it gets shut down on failures.