[CRASH] KeyDB 6.3.4 crash on replication db.cpp:2944 'm_fTrackingChanges >= 0' is not true #874
Comments
sbaier1
changed the title
|
Hi @sbaier1 Just on a side note, the patch above will only get KeyDB to startup with redisearch module with FLASH. However, once started up, the indexes will not be created so it will not actually work. You will need additional work on startup to:
I am unfamiliar with redisjson and timeseries, but I suspect they might also need some work like redisearch to with FLASH. If you disabled FLASH, will replication start to work? |
i think that was mostly a red hering, i just included all the info for complete-ness. it crashes in the same way regardless of
at this point i could imagine it's simply due to the master i'm replicating being an upstream redis 7.2.0 vs keydb being based on 6.x, so maybe it's just a format error causing a crash (which is still cosmetically not great though) |
|
I am facing the same issue as the OP. I am trying to migrate to KeyBD from a Redis 7.4.1. I tried to create a KeyDB instance as a replica of the Redis server and got the same errors as above. In fact I can replicate it locally if I use the following services:
redis:
image: redis:7.4.1
keydb:
image: eqalpha/keydb
depends_on:
- redis
command: keydb-server /etc/keydb/keydb.conf --replicaof redis 6379And then run Which gives me the following output before the KeyDB container exits: |
|
hi @dt-bernd Just briefly looking at the code, I can see a possible code flow which can cause the crash. Can you try this to see if the crash still happens:
If there is no crash, keydb is likely calling beforeSleep() when doing replication at startup, before it can fully initialise. |
|
Thanks for having a look @keithchew - I've tried that with the same result: |
|
hi @dt-bernd Did the crash happen as soon as you entered the cli command? Note that you need to remove the replicaof command from your docker-compose.yml so keydb starts up as master. |
|
@keithchew - Yes. I updated the docker compose command option, so that it was in standalone mode. I made sure everything was working. Then manually issued the "REPLICAOF" command. Then get the following output in the logs: followed by the error above. |
|
I see it! When it encounters "Can't handle RDB format version 12", the method did not "goto eoferr" to resume tracking changes. And looking further, keydb's supported RDB version is 9, so your redis is quite far ahead. Note sure the best workaround, the only thing I can think of is to first migrate your redis data to version 9, then move to keydb. Alternatively, if you are sure you don't have version specific data, then perhaps you can update the keydb's code to accept higher versions. Since you are just evaluating, the latter might be a quick way just to see if keydb can get the data. |
|
@keithchew Thanks, I had also tried the diskless sync option, hoping that might avoid the RDB version issue - but got similar results. Another option was using riot replication to move the data from the Redis 7 format to KeyDB. I was able to replicate the data using the And was able to get most of the data across. Unfortunately one of our use cases is streams, and while it moved the actual stream data across, it didn't move the consumer metadata across too. For anyone else, this might be an option in doing a "live" migration. |
|
The issue is your data, not the replication method. Because of the version mismatch, keydb does not even attempt to replicate, and its incorrect error handling causes the crash. Your attempt with riot struct method seems painful, if you decide to try my suggestion above, it is only a 1-liner change in keydb's source code. |
|
Looking into this a bit more, I think the one-liner has a good chance of working, as streams are from version 9: Don't know what is in version 10, 11, 12 so maybe you can double check this before proceeding. |
|
Thanks, will see what I can do with the code change and test. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Crash report
looks similar to #541 but i'm running 6.3.4 (also tried current HEAD@main FWIW)
Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.
Aditional information
Context: I am evaluating KeyDB as an alternative for a current Redis (7.2.0 FWIW) setup. I was trying to use
replicaofto run KeyDB as a read-replica for an existing redis replica. The replica contains various time-series and JSON objects.Out of curiosity, I've also tried running the same image on amd64 and with
server-threads 1, neither of which resolved the issue, so it seems to be thread-safety (for the modules) + architecture independent.The crash happens during replication, while no clients are connected yet.
The configuration file is unremarkable. the minimal config that reproduces the issue is just loadmodule + replicaof to connect to the replica.
I'm using a container build based on ubuntu 22.04 including a patch from another issue here to get redisearch to work with KeyDB.
Dockerfile FWIW
patch
The text was updated successfully, but these errors were encountered: