Job delayed too long, service full error when bulk uploading

[GordianDziwis]

As you can see, I am putting your plugin through the paces ;) When I submit all the datasets at once, some get this error message:

When I retry, it works. So I would guess there is some timeout set too low.

[jqnatividad]

Thanks @BonaBeavis for pushing it ;)

Will investigate further. Can you share more info about the number of jobs enqueued? Were they local files? URLs? How big were the files?

[GordianDziwis]

About 60 datasets, both local and remote. The datasets are not big, about a few dozen entries and a dozen headers.

I am running ckan datapusher submit --yes do you have some command I could run to get a debug output?

[jqnatividad]

Yes @BonaBeavis , you can turn on logging by tweaking some values in your .env file (https://github.com/dathere/datapusher-plus/blob/master/datapusher/dot-env.template)

You can also have qsv create its own logs by setting these env_vars:

export QSV_LOG_LEVEL=debug
export QSV_LOG_DIR=/tmp

The log file should be named qsvdp_rCURRENT.log in /tmp.

[jqnatividad]

This will be fixed with DP+ 1.x, where we convert DP+ to be an extension, rather than a separate web service.

cc @tino097 @TomeCirun

[GordianDziwis]

Workaround until then:

for id in $(ckan dataset list | grep -oP '^\w{8}-\w{4}-\w{4}-\w{4}-\w{12}')
do
  ckan datapusher submit --yes $id
  sleep 1
done