Best practices (if any) to run multiple instances of CDL in parallel? #423
-
|
I sometimes get bored by waiting for CDL to download a particularly large Coomer profile which takes literally days. (I normally use very conservative settings to only allow one scrape or download in parallel from any site.) So, I sometimes try to launch another instance of CDL in parallel, for another URL. What's interesting is that results seem to vary greatly from time to time. Sometimes it seems to "just work", with two instances coexisting peacefully with no observed issues. And sometimes I start getting constant "database locked" errors from both instances. I know it's probably finicky and not how things are supposed to work. (I have no idea what database CDL uses and whether it allows concurrent connections.) But I'm puzzled by the fact that it sometimes works, and I would like to understand better how I could make it work more often, if not always. thougths? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
It's not supported. CDL uses SQLite for the database which only allows one write operation at a time. For it to work properly you have to use a different |
Beta Was this translation helpful? Give feedback.
It's not supported. CDL uses SQLite for the database which only allows one write operation at a time.
For it to work properly you have to use a different
--appdata-folderfor each instance. That will force each instance to use a different database. But you will have to create a method for how to merge the 2 databases back into 1