Heroku "Free app running time quota exhausted"

[danielweck]

https://readium2.herokuapp.com

Application error
An error occurred in the application and your page could not be served. If you are the application owner, check your logs for details.

[danielweck]

It looks like some traffic is caused by http://readium-2.surge.sh (there may be other HTTP referers too, I only looked at the tail of the Heroku logs using the CLI heroku logs --app readium2 -t)

[danielweck]

Email message from Heroku:

You have used all of your 550 Free Dyno hours for April 2018.
If you verify your account with a credit card, you can receive 450 more Free Dyno hours a month.
One or more of your Free Dyno apps are becoming more popular. Upgrade your app to Hobby dynos for $7 per dyno per month so it never sleeps and users can access it 24/7. A monitoring service could unintentionally be keeping your app awake. Run heroku logs -t to view your app's logs to see if this is the case and remove Pingdom, New Relic or other availability monitoring services that are not required for a free app Your application may not have a web dyno at all. Once worker-only apps are started, they run 24/7. You can avoid this by upgrading to a hobby dyno, or by manually scaling down your application
Dyno sleeping lets us provide you and millions of other users with a free tier for experimenting, hacking on personal projects and running low traffic apps.

[danielweck]

It looks like a Marketing bot is creating requests too:
http://ahrefs.com/robot/

To address this particular source of traffic, let's add /robots.txt http://www.robotstxt.org/robotstxt.html

[danielweck]

Other bots:
http://www.google.com/bot.html
http://www.bing.com/bingbot.htm
http://www.semrush.com/bot.html

[danielweck]

The server CLI now automatically prevents Robots, see commit:
f0e8446

...so this should fix the r2-streamer-js Heroku deployment immediately ( see https://github.com/edrlab/r2-streamer-js/blob/develop/Procfile ), but will only take effect when a fresh free Dyno allowance is restored (next month / May?).

[danielweck]

For future reference, here are useful Heroku CLIs to find Robot agents in the app logs:
heroku logs -n 1500 --app readium2 --source app | grep -i bot
heroku logs -n 1500 --app readium2 --ps router | grep -i robots.txt

[danielweck]

Note that the r2-streamer-js deployment at Now.sh (which is used as an alternative to Heroku, albeit without continuous integration / deploy automation) is live, including the recent addition of the anti-Robots Express route, see: https://readium2.now.sh/robots.txt

[danielweck]

Related: #22