Connection timeout uploading multiple files

[dleadbetter]

Occasionally when uploading multiple files from a client application, the IIIF Cloud server will timeout.

Error R14 (Memory quota exceeded)

We should up the size of the dyno used on Heroku to see if it resolves the issue, and do some load testing to find the upper limit of the number of requests that can be sent concurrently.

We should also consider moving any high-memory consumption processes to jobs (i.e. converting images to ptif).

[dleadbetter]

@NickLaiacona, @camdendotlol, @jamiefolsom - I've done some digging into this. I was able to resolve the memory consumption issue by off loading the image processing to an asynchronous job.

However the issue with the request taking longer than Heroku allows still persists. The bottle neck is really the Rails application uploading the image the S3. It's a blocking process, so the consuming application (UDCSL, NBU, etc) will always need to wait for this to finish. And the larger the file, the longer this will take.

Heroku recommends using something like direct uploads to avoid the timeout issue, which is something we could definitely implement, but will take some time.

Another solution could be to limit the number of files, or total size of the files that can be uploaded at once?

I'm not totally sure how much effort we want to sink into this right now or if we'll have clients that will want to upload hundreds of files at once, so open to suggestions on what you all think the appropriate solution would be for both the long term and short term.

[NickLaiacona]

If we gave the client an account on the IIIF server and had them create a manifest there, would that be easier?

[dleadbetter]

If we gave the client an account on the IIIF server and had them create a manifest there, would that be easier?

I think the timeout issue when uploading dozens of files would still exist. We're using a strategy where files are uploaded to the Rails application (IIIF Cloud), processed, then uploaded to S3. So the more files that are uploaded at once and the larger the files, the longer the request is going to take. Eventually it will hit the timeout limit imposed by Heroku.

Another option for getting around this may be updating NBU to not use the /upload end point, but instead make individual POST requests for resources to the client. So essentially if a user is uploading 50 images, we'd make 50 requests to NBU. I can test this out today with UDCSL to see if there is any performance drawbacks.

[camdendotlol]

I think the individual uploads would be a good solution. The loading spinner could also indicate how far along it is, e.g. "Uploading files: 12 out of 30"

[NickLaiacona]

Sounds good.