Configurable job parallelism

[smashwilson]

I'm introducing a runner configuration option, concurrentJobs, to allow multiple jobs to be scheduled in parallel.

[smashwilson]

I realized after I put this together that this is too simplistic; it assumes that, at most, the most recent n-1 data directories (possibly) belong to concurrent builds. If you have one long build and many short ones that happen before it completes, you can have many more data directories still in use.

I'll have to introduce a better way to track which data directories are in use and which are not.

[knownasilya]

Good start

[niallo]

Good to see some work on this. The data dirs could be keyed by job ID or similar. Reaping them becomes a little tricky. Some folks may like to re-use build directories in some circumstances too (even though that's not always going to be safe). Just something to think about.

[knownasilya]

I feel like concurrent jobs for the same project in the same branch don't make sense at the moment. Maybe that should be the limitation.

[smashwilson]

I feel like concurrent jobs for the same project in the same branch don't make sense at the moment. Maybe that should be the limitation.

Hmm! That's going to be tricky to manage - an async.queue won't cut it any more, because more commits can arrive on a branch corresponding to an existing job even while the queue is unsaturated.

Concurrent builds on a single branch will be an issue if they trigger deployments, though.

The data dirs could be keyed by job ID or similar. Reaping them becomes a little tricky.

I had been thinking about maintaining an in-memory structure of data directories that are in-use and excluding them from the culling somehow. I think the runner can get away with something in memory because simple-runner jobs don't persist across service restarts anyway - otherwise I'd need to either parse the job ID from the data directory name or use some kind of flag file to identify active data directories.

Some folks may like to re-use build directories in some circumstances too (even though that's not always going to be safe). Just something to think about.

Re-using a data directory from one job to another? I'd thought that was what the cache was intended for. With the job ID in the data directory name, how does that work - or am I misunderstanding?

[knownasilya]

So there are two concepts, the cache and the data dir (as far as I know) and the cache is used by certain plugins but the data dir is where the project is cloned into and built. Personally I always copy the contents from the data dir to /var/www/<project name> when using strider locally.

[smashwilson]

Okay, I've revisited this to:

• Use a custom JobQueue. It's an async queue that prevents the concurrent scheduling of tasks that share a key, namely a (project, branch) tuple. Otherwise, it runs parallel jobs up to a defined concurrency limit.
• Prune recent data directories specific to each project and branch rather than globally. (This should also be helpful for For jobs data directory keeper is not branch aware and count is not configurable on ui #28.) Because jobs on the same branch don't run concurrently, this'll prevent Strider from deleting a build directory that's in use by a concurrent job.

Here's what the data directory structure looks like:

~/code/strider-simple-runner (parallel-jobs=) 
$ find /Users/ashl6947/.strider/data -type d -maxdepth 2
/Users/ashl6947/.strider/data
/Users/ashl6947/.strider/data/deconst-deconst-control-master
/Users/ashl6947/.strider/data/deconst-deconst-control-master/job-56f29d31a141bc8723e5bf9a
/Users/ashl6947/.strider/data/deconst-deconst-docs-control-master
/Users/ashl6947/.strider/data/deconst-deconst-docs-control-master/job-56f29d30a141bc8723e5bf98
/Users/ashl6947/.strider/data/deconst-deconst-docs-master
/Users/ashl6947/.strider/data/deconst-deconst-docs-master/job-56f29d2fa141bc8723e5bf97
/Users/ashl6947/.strider/data/deconst-deploy-master
/Users/ashl6947/.strider/data/deconst-deploy-master/job-56f29d30a141bc8723e5bf99

I've also tried to be careful in extracting branch names from the job in a sane way because Job's ref field can hold arbitrary things. I'm testing for a few common keys (branch, fetch) but falling back to an ugly-but-safe "URI-encoded stable JSON representation of the whole thing".

[knownasilya]

Do you think this is ready for merge? Did you have any plans to write some basic tests for this JobQueue?

[knownasilya]

Nvm, you have tests.

[smashwilson]

I was just giving it a last look-over to see if there was anything obvious I'd missed. I don't think so 😄

[knownasilya]

So it's probably a good idea to expose the configuration in Strider UI.

[smashwilson]

Yeah, agreed. The number of recent directories to keep should be simple enough to add. The level of concurrency is less clear to me.

Is there a place for installation-wide (rather than build-specific) configuration for runner plugins?

[knownasilya]

Hum, no but Strider-CD/strider#681 would help when I get more time to finish it.

[knownasilya]

How is it configured now, an env var? Need to document in the mean time.

[smashwilson]

Yeah, CONCURRENT_JOBS.

[smashwilson]

It's a little generic, but I wasn't sure how far to go down the STRIDER_SIMPLE_RUNNER_... prefixing road. I can patch it if there's a pattern I wasn't following.

[knownasilya]

No, this is good. I documented it in the strider repo and updated the version of simple-runner.