Vacuum the sqlite rpmdb if necessary #3452
Conversation
tests/rpmdb.at
Outdated
| ], | ||
| [0], | ||
| [D: VACUUM: 0 | ||
| D: Rpmdb Sqlite backend VACUUM maxfree: 1024, free: 397312 -> 0 |
There was a problem hiding this comment.
/usr/lib/sysimage/rpm is a "live" rpmdb, so these values can and will change over time, breaking the test.
There was a problem hiding this comment.
Yeah, that's kinda on purpose. I wanted a real database for testing. But may be the test suite can do without.
lib/backend/sqlite.cc
Outdated
| int max_size = rpmExpandNumeric("%{?_sqlite_vacuum}"); | ||
| if (max_size <= 0) | ||
| max_size = 20*1024*1024; | ||
| long free_space = sqlite_free_space(sdb); |
There was a problem hiding this comment.
On 32bit architectures, long is a mere 32bit integer. Better be safe and use explicit int64_t for the size since that's you're getting it from sqlite to begin with.
There was a problem hiding this comment.
Yeah, I have been pondering about int vs long vs int64_t here.
rpmExpandNumeric also only returns an int. I wonder if we should at least move the C++ variant to 64 bits. Not that we need it here urgently but adding a 32 bit API seems kinda wrong.
|
Minor nits in what's not even my review 😆 aside, this is certainly a good thing to have 👍 I remember initially skipping this because I was worried about reliability and stuff for such a major live surgery, clearly not yet having quite the unwavering trust towards sqlite that we have now. I've been intending to look at this but I guess I got a bit burned out on the sqlite front back then and just haven't found the will to dig into it. |
|
Nit picks addressed. Yeah, that's why I did this in C. This looks a lot like something we want to have - even in older releases. |
lib/backend/sqlite.cc
Outdated
| int max_size = rpmExpandNumeric("%{?_sqlite_vacuum_kb}"); | ||
| if (max_size <= 0) | ||
| max_size = 20*1024; | ||
| int64_t free_space = sqlite_free_space(sdb) >> 10; |
There was a problem hiding this comment.
int64_t here doesn't help when it's long in sqlite_free_space(). Which should return the same kind of data as the macro really so this calculation belongs to that function, more like. Doing *1024 but then >> 10 looks super confusing, lets just stick to basic maths for doing math, because to me a bitshift suggests there's something more special going on.
lib/backend/sqlite.cc
Outdated
|
|
||
| if (free_space > max_size) { | ||
| sqlexec(sdb, "VACUUM"); | ||
| rpmlog(RPMLOG_DEBUG, "rpmdb sqlite backend VACUUM maxfree: %ikB, free: %" PRIu64 "kB -> %" PRIu64 "kB\n", max_size, free_space, sqlite_free_space(sdb)); |
There was a problem hiding this comment.
Oh and just spotted this now: please keep lines < 80 chacters to keep things readable in 80x terminal.
Check if there is 20MB in free pages and then execute a VACUUM command. The threshold can be controlled by the _sqlite_vacuum macro. Don't add this to the macros file on purpose as we don't want people to get involved with such details. Here it is mainly used for testing. Using a 20 MB threshold should prevent the vacuuming to happend too often while still triggering after large transactions. As we install new headers first and then remove the old ones transactions leave behind large amounts of free pages. We do not use PRAGMA auto_vacuum here as it does not defrag the database and only frees empty pages. So it still requires running VACUUM from time to time. Freeing the empty pages would get rid of the condition we use here for running VACUUM. Using pure C for the macro expansion on purpopse in case we want to back port this. Resolves: rpm-software-management#3309
Check if there is 20MB in free pages and then execute a VACUUM command. The threshold can be controlled by the _sqlite_vacuum macro. Don't add this to the macros file on purpose as we don't want people to get involved with such details. Here it is mainly used for testing.
Using a 20 MB threshold should prevent the vacuuming to happend too often while still triggering after large transactions. As we install new headers first and then remove the old ones transactions leave behind large amounts of free pages.
We do not use PRAGMA auto_vacuum here as it does not defrag the database and only frees empty pages. So it still requires running VACUUM from time to time. Freeing the empty pages would get rid of the condition we use here for running VACUUM.
Using pure C for the macro expansion on purpopse in case we want to back port this.
Resolves: #3309