Separate downloading and filtering in NPM commands?

[gibranparvez]

Since downloading and filtering are separate steps, perhaps there should be a separate npm command for filtering? Though in many use cases it might be redundant, it is a tad unclear what npm run download actually does.

[orangejulius]

I like this idea, since, as I understand it, right now there's no way to updated the downloaded whosonfirst sqlite file besides manually deleting it. Exposing more clearly defined "download new data" and "extract using existing data" seems nice.

[missinglink]

I would support this, the new SQLite code is designed in a way which can fairly easy accommodate this.

We would not be able to change the existing behaviour for npm run download because code is relying on that, but 👍 for adding additional commands.

[missinglink]

If we do this, I'd also like to move this code somewhere else: https://github.com/pelias/whosonfirst/blob/master/utils/download_data.js#L17-L50

It got kind of messy and I regret it being piled in that file when it could be in its own file.

[orangejulius]

This would be increasingly helpful now, as the WOF team has now automated pushing updates to the sqlite files. They therefore change quite regularly, and downloading many GB of database file might not always be desired.

[missinglink]

The existing code checks the timestamp on disk vs the timestamp on the server and doesn't re-download data if it's not required.

[orangejulius]

@missinglink that's exactly the point. Sometimes a user might be in a low/no-internet situation, or not want to bother re-downloading, but still want to extract a new set of data from the existing sqlite DB.