delay csvdb load tasks by a random offset to avoid database lock issues

[kevinrue]

This resolved my issues when processing a lot of FASTQ files in parallel.
(There may be better ways to address the issue, though)

[sebastian-luna-valero]

Thanks for the input, Kevin. It looks like a good workaround.

Please bear in mind that SQLite is not a proper Relational Database Management System, and actually its performance is dependent on the underlying filesystem.

I tried to fix SQLite operational errors with the following changes but more thought might be required:

• Pipeline changes: Code changes for improved portability of pipelines #380
• scripts changes: Allow multi-threaded sqlite3 operations cgat#377

Those changes were added to the master in the v0.3.2 release, could you please let me know what version are you running with the following commands:

pip list | grep -i cgat

Best regards,
Sebastian

[kevinrue]

Hi Sebastian,
I think I should be up to date. I pulled from the master branch this morning.

$ pip list | grep -i cgat
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
CGAT (0.3.2, /gfs/devel/kralbrecht/cgat)
CGATPipelines (0.3.2, /gfs/devel/kralbrecht/CGATPipelines)
CGATReport (0.7.6.1)

Let me know if there's any command I should run to update the CGAT installation, or something.

[kevinrue]

For the record, I was systematically getting the 'database locked' error message for pipeline_readqc.py with 384 FASTQ files. I don't know exactly where this falls in the range of input files that you usually process in a single run.
As I said, the random delay of 1-30 seconds worked in my case, but might need to be increased or even parameterised if the error pops out again for larger amounts of input files.
Best
Kevin

[sebastian-luna-valero]

Thanks, Kevin.

We try to solve this by asking ruffus to limit the number of concurrent jobs to 1`:

  @jobs_limit(PARAMS.get("jobs_limit_db", 1), "db") 

However, it does not seem to work properly.

We are about to perform a major refactoring on the code, and we'll try to solve this problem then. The idea you propose in this PR is a potential solution. I will leave this open for our reference and I will decide what to do during the code refactoring.

Many thanks!
Sebastian

[kevinrue]

Hi @sebastian-luna-valero
Yes I saw the jobs_limit instruction in the code, and it did puzzle me as to why it does not seem to work. However, I was short on time and instead of investigating the issue, implemented the quick fix offered in this PR. Obviously, the ideal fix here would be to somehow have this jobs_limit instruction work as expected.

I 100% agree that this PR is not the ideal implementation, and I would not take it personally if it is closed without being merged. Happy to see it used as a reference.
As a user, I'll say that I'm happy enough to have this random delay even increased to a minute or two if needed, which would still represent a fraction of the total pipeline run, in compensation for reducing the chance of database block virtually to nil.

Thanks for your work on the refactoring, I look forward to the result!