It may make more sense to upload the logs first in case the graph wasn't built. If the graph wasn't built, the script might error out on a file not found error perhaps.

Will that actually happen? I suppose I can make the change regardless, but I don't think the execution of any line is dependent on the previous line, is it?

I'm increasingly hesitant about the design of this method. I think this is going to limit us moving forward to be writing bash in java strings. Something I would really like to see is more detailed reporting and logging of the progress of building and loading a graph and automatically retrying if something fails that can be retried. We could attempt to do this with bash, but it would be ugly. If we're trying to ship this quick, then fine I can live with it, but I think we should be making a separate startup script specifically for this where we can do more detailed things. Furthermore, right now this is basically using Amazon s3 as a database of sorts for storing various information when we already have a mongo and postgres instance. Maybe it would make sense for the startup script to directly edit the mongo db or maybe make queries to datatools-server. Either way, we should be thinking about this feature long-term and about what the easiest way to create maintainable code will be.

Thanks for the comments. Perhaps there is a better way to handle what gets run in user data, but at some level it is going to require running some bash (unless we opt to include some kind of startup script on the AMI or write a script to a file that gets fetched by the instance). Because of this, I think our best option for now is to keep everything contained in a single place -- this bash script.

Regarding using S3 as a database, can you describe what you mean more fully? We're just storing static files that were generated as a result of the deploy process. We also store GTFS zip files in S3 as well. I'm really not too sure of what the concern there is. The alternative you're proposing as I understand it (the OTP instance connecting to the mongo instance and handing off files/messages) sounds far more complex.

That's what I'm saying. I'm recommending creating a script in some kind of programming language - be it java, python or JavaScript. Then this startup script would communicate directly with either mongo db or datatools-server via HTTP requests perhpas. As it is right now, all of this bash is in one place, but we have to construct it using strings in java. Therefore, we can't view it nicely formatted in an IDE, we can't lint or do static typing analysis, it's much harder to import 3rd party libraries and it is impossible to write isolated tests.

Regarding s3 I'm fine with most of what's going on. Most of the data makes sense to store in a raw format (logs, GTFS files, OSM pbf). The thing in particular that I'm concerned about is constantly checking for bundle download and graph build completion and who knows what else we may add in the future by checking for the existence of an s3 file. I'm not recommending that OTP connect to the mongo instance, but I am recommending that a replacement of the bash script do this and handle the checks of bundle download, graph build, and graph load. These intermediate files that are waited upon seem like the more complex option to me when the bash replacement script could just be writing to the mongo database directly or making an HTTP request to datatools-server, or maybe even using some other messaging framework.

Furthermore, using s3 as a database like this will cost more money since s3 has per-request pricing so the more we scale the more this will cost us. The cost seems lowish at $0.005 per 1k requests, but let's say new graphs are built on a nightly basis. And let's say graph building takes 1 hour for a largeish region with elevation data. So, it'll make an s3 request every 5 seconds for an hour or 3,600 requests per day and 1.3 million a year. If we have 100 regions doing this, the cost of just these s3 requests would be $657 a year compared to $0 for using a database or sending over an HTTP request to the datatools-server instance.

I like this addition as well.