First commit ...

aws.md

@@ -0,0 +1,162 @@
+# AWS
+
+- Install cli tools for happiness
+  - `pip install awscli`
+  - `pip install aws-shell`
+  - `apt-get install jq`
+- https://github.com/open-guides/og-aws
+- https://medium.com/@kief/https-medium-com-kief-using-pipelines-to-manage-environments-with-infrastructure-as-code-b37285a1cbf5
+- http://creately.com/blog/diagrams/aws-templates-for-architecture-diagrams/
+- https://www.slideshare.net/AmazonWebServices/masterclass-advanced-security-best-practices
+- https://www.nvteh.com/news/problems-with-public-ebs-snapshots
+
+
+## EC2
+- The bread and butter of AWS
+- Default into public subnet
+- You can run a command across all instances if you have their agent installed
+- `UserData` config allows you to provide some instance config or provisioning, which is run under sudo
+  - `/var/lib/cloud/instance/scripts/part-001`
+  - `ls /var/log/cloud-init-output.log`
+
+### Security Groups
+- http://harish11g.blogspot.com/2015/06/best-practices-tips-on-amazon-web-services-security-groups-aws-security-managed-services.html
+- http://www.stratoscale.com/blog/compute/aws-security-groups-5-best-practices/
+
+
+## Elastic Beanstalk
+- Think of a Heroku setup, with server setup abstracted away
+- Has a cli tool just like Heroku
+- You choose your platform, eg PHP. python, node.js etc and go from there
+- Orchestrates setting up EC2 instances with load balancing etc
+- Sets everything up using dynamic on-the-flow CloudFormation templates
+
+**Links**
+- http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/ebextensions.html
+- https://github.com/lapygithub/eb_config_examples
+- https://medium.com/trisfera/getting-to-know-and-love-aws-elastic-beanstalk-configuration-files-ebextensions-9a4502a26e3c
+- http://blog.rudylee.com/2014/05/22/configuring-elastic-beanstalk-environment-with-ebextensions/
+- https://tech.pic-collage.com/two-must-have-configurations-when-using-aws-elastic-beanstalk-with-docker-720ce7c5ec91
+- https://github.com/awslabs/eb-demo-php-simple-app
+- http://blog.flux7.com/blogs/docker/10-steps-deploying-docker-containers-on-elastic-beanstalk
+- https://www.slideshare.net/AmazonWebServices/running-microservices-and-docker-on-aws-elastic-beanstalk-august-2016-monthly-webinar-series
+- `.elasticbeanstalk/config.yaml`
+- `.ebextensions/options.config`
+- `.ebextensions/options.config`
+
+
+## VPC
+- Plan out your subnets or plan to fail!
+- Is software defined networking
+
+**Links**
+- https://www.slideshare.net/gsilverm/aws-vpc-in
+- https://medium.com/aws-activate-startup-blog/practical-vpc-design-8412e1a18dcc
+- https://charity.wtf/2016/03/23/aws-networking-environments-and-you/
+- http://blog.flux7.com/blogs/aws/vpc-best-configuration-practices
+- http://cloudacademy.com/blog/aws-vpc-configuration-five-kick-yourself-mistakes/
+- https://www.pluralsight.com/blog/it-ops/simplify-routing-how-to-organize-your-network-into-smaller-subnets
+
+
+#### Subnets
+- A VPC has multiple subnets
+#### Route Tables
+- Routes you associate with your subnet
+
+#### NACLS
+- Return/response traffic can come in through a range of ports, make sure to leave that range open
+http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_NACLs.html
+
+
+## S3
+- Buckets of storage yo
+
+
+## CloudWatch
+- Detailed monitoring vs regular is more granular interval of recording
+- Has alarms and alerting
+- https://github.com/jorgebastida/awslogs
+- http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html
+- http://blog.brianbeach.com/2014/09/cloudwatch-logs-push.html
+- http://zoltanaltfatter.com/2017/01/13/centralized-logging-with-cloudwatch-logs/
+- https://cloudacademy.com/blog/centralized-log-management-with-aws-cloudwatch-part-1-of-3/
+- https://alestic.com/2010/12/ec2-user-data-output/
+- https://aws.amazon.com/blogs/devops/send-ecs-container-logs-to-cloudwatch-logs-for-centralized-monitoring/
+
+## Route 53
+- DNS management
+- Can configure health checks to request a specific domain or ip
+- Allows you to create complex routing configurations with traffic policies etc
+
+## CloudFormation
+Infrastructure as code! You can create yaml|json defintions of your infrastructure stack which creates your 
+stack.
+- You can get a cost estimate from a CloudFormation template
+http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-paying.html
+- You can reference other CloudFormation templates for organization sake
+- Use `AWS::CloudFormation::Init` for initial setup
+- `aws cloudformation validate-template --template-body file:////home//local//test//sampletemplate.json`
+
+**Links**
+- https://github.com/toddm92/aws/wiki/CloudFormation-Best-Practices
+- http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-paying.html
+- https://www.slideshare.net/AmazonWebServices/dvo304-aws-cloudformation-best-practices
+- https://www.quora.com/Why-are-there-no-AWS-CloudFormation-success-stories
+- https://paperairoplane.net/?p=680
+- http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-init.html
+- https://blog.engelke.com/2012/12/28/provisioning-a-server-with-cloudformation/
+
+## IAM
+### Groups
+- Have policies, up to 10 policies per group http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-limits.html
+- A user can belong to many groups
+
+### Users
+- Can have api only access or console access or both
+- Can belong to one or many groups
+- Can be assigned specific policies vs policies inherited from group membership
+
+### Policy
+- Dictates what you can or can't do
+- Inline policies are for one-off special snowflakes and should be avoided generally 
+
+### Roles
+- You can assign a role to a service, eg an EC2 instance can have a role of PublicWebServer which has its own policies
+- Think of a role as an identity, not a group
+- Leverage for cross account access
+
+### KMS
+Storing encryption keys in the cloud
+
+- https://blog.fugue.co/2015-04-21-aws-kms-secrets.html
+- https://github.com/fugue/credstash
+- https://medium.com/@mda590/simple-secrets-management-via-aws-ec2-parameter-store-737477e19450
+- https://github.com/mozilla/sops
+
+## AWS Config
+- Run a set of checks against resources to ensure adherance to certain standards
+- $2 a rule
+
+## CloudTrail
+Think of it as the auditd of linux but for AWS
+
+## CodeCommit
+- Private git repos which can have triggers!
+- http://www.paul-kearney.com/2015/09/migrating-from-github-to-aws-codecommit.html
+- If you have an instance role that has access to a repos you need to use the credential helper
+    http://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-https-windows.html#setting-up-https-windows-credential-helper
+    ```shell
+    git config --global credential.helper '!aws codecommit credential-helper $@'
+    git config --global credential.UseHttpPath true
+    ```
+
+
+## Other
+- Use Resource GRoups to get a birds eye view of everything, across AZs
+
+### AWS Certificate Mananger
+- http://serverfault.com/questions/9708/what-is-a-pem-file-and-how-does-it-differ-from-other-openssl-generated-key-file
+- https://docs.aws.amazon.com/acm/latest/userguide/acm-services.html
+- http://docs.aws.amazon.com/acm/latest/userguide/import-certificate.html
+- https://jamielinux.com/docs/openssl-certificate-authority/certificate-revocation-lists.html
+- https://www.schneier.com/academic/paperfiles/paper-pki-ft.txt

influxdb.md

@@ -0,0 +1,36 @@
+# Notes - Influxdb
+
+
+- measurements ~ table
+- fields are non-indexed columns
+- If you are using a field & tag that are the same, use a different name
+  eg fields: {class: 1, sn: 23434, cpu 0.232}, tags: {class:1} vs tags:{pclass:1}
+  makes querying easier
+- tags are indexed columns and should be used on columns that don't have lots of unique values make filtering faster (~10x)
+- tune precision (seconds, minutes, etc)
+- values default to int
+- types are bool, int, float, str and after the first record the field to type mapping is set
+- https://cobe.io/blog/posts/memory-adventures-with-influxdb/
+- https://vimeo.com/169742452
+- https://vimeo.com/170035101
+- https://vimeo.com/195058724
+- https://vimeo.com/198723778
+- https://vimeo.com/200898048
+- https://grisha.org/blog/2015/03/20/influxdb-data/
+- http://www.ryandaigle.com/a/time-series-db-design-with-influx
+- http://roobert.github.io/2015/10/10/Columned-Graphite-Data-in-InfluxDB/
+- https://maxchadwick.xyz/blog/monitoring-magento-cron
+- http://techblog.shutl.com/2016/06/moving-from-graphite-to-influxdb/
+- http://ryanfrantz.com/posts/solving-monitoring/
+- https://blog.karmawifi.com/the-software-behind-karma-metrics-and-alerting-57cb1d0ddaf#.1k9rpu78w
+- http://blog.aiven.io/2016/04/monitoring-influxdb-telegraf-grafana.html
+- https://www.digitalocean.com/community/tutorials/how-to-analyze-system-metrics-with-influxdb-on-centos-7
+- http://www.vishalbiyani.com/graphing-performance-with-collectd-influxdb-grafana/
+- https://milinda.svbtle.com/cluster-and-service-monitoring-using-grafana-influxdb-and-collecd
+- https://streamsets.com/blog/streamsets-monitoring-grafana-influxdb-jmxtrans/
+- http://rmoff.net/2016/05/12/monitoring-logstash-ingest-rates-with-influxdb-and-grafana/
+- http://jansipke.nl/monitoring-hosts-with-collectd-influxdb-and-grafana/
+- https://markri.nl/custom-monitoring-stack/
+- http://tech.aabouzaid.com/2016/08/monitoring-processes-with-telegraf-influxdb-kapacitor-python.html
+- https://github.com/mark-rushakoff/awesome-influxdb
+- https://influxdata.com/tldr-influxdb-tech-tips-february-16-2017/

metrics.md

@@ -0,0 +1,29 @@
+# Metrics
+
+- https://kartar.net/2015/08/monitoring-survey-2015---metrics/
+- http://dieter.plaetinck.be/post/a_few_common_graphite_problems_and_how_they_are_already_solved/
+- https://matt.aimonetti.net/posts/2013/06/26/practical-guide-to-graphite-monitoring/
+- https://artofmonitoring.com/
+- http://le-huy.blogspot.com/2013/08/some-useful-tips-for-graphite.html
+- http://stackoverflow.com/questions/18108047/statsd-graphite-naming-conventions-for-metrics
+- http://obfuscurity.com/2012/05/Organizing-Your-Graphite-Metrics
+- http://florin.myip.org/blog/massive-scalability-when-querying-metrics-stored-graphite
+- https://prometheus.io/docs/practices/naming/
+- http://metrics20.org/
+- http://www.slideshare.net/Dieterbe/metrics-stack-20
+- https://docs.newrelic.com/docs/plugins/plugin-developer-resources/developer-reference/metric-naming-reference
+- https://support.signalfx.com/hc/en-us/articles/201213425-Metrics-dimensions-and-metadata
+- https://support.signalfx.com/hc/en-us/articles/201213445
+- http://www.codinginstinct.com/2013/03/metrics-and-graphite.html
+- https://codeascraft.com/2011/02/15/measure-anything-measure-everything/
+- https://codeascraft.com/category/infrastructure/
+- http://www.xkyle.com/graph-everything-with-graphite/
+- http://shop.oreilly.com/product/0636920035794.do
+- http://www.programblings.com/2013/11/14/sensu-checks-to-report-metrics/
+- https://github.com/ryandotsmith/l2met
+- http://www.ryandaigle.com/a/time-series-db-design-with-influx
+- https://github.com/lukevenediger/statsd.net/blob/master/statsd.net/Documentation/guidance/metric-anti-patterns.md
+- https://influxdata.com/blog/getting-started-with-sending-statsd-metrics-to-telegraf-influxdb/
+- https://blog.raintank.io/logs-and-metrics-and-graphs-oh-my/
+- http://www.robustperception.io/monitoring-not-just-for-outages/
+- http://www.robustperception.io/rate-then-sum-never-sum-then-rate/

nodejs.md

@@ -0,0 +1,10 @@
+# NodeJS
+
+## Debugging node.js
+- http://techblog.netflix.com/2014/11/nodejs-in-flames.html
+- https://nodejs.org/en/docs/guides/simple-profiling/
+- https://blog.heroku.com/node-habits-2016
+- http://www.slideshare.net/yunongx/debugging-node-in-prod
+- https://github.com/node-inspector/v8-profiler
+- https://blog.risingstack.com/finding-a-memory-leak-in-node-js/
+- http://www.brendangregg.com/blog/2014-09-17/node-flame-graphs-on-linux.html

python.md

@@ -0,0 +1,23 @@
+# Notes - Python
+
+
+## Python Resources
+- http://jtushman.github.io/blog/2014/01/14/python-|-multiprocessing-and-interrupts/
+- http://www.programcreek.com/python/example/57787/multiprocessing.JoinableQueue
+- https://noswap.com/blog/python-multiprocessing-keyboardinterrupt
+- https://zhimingwang.org/blog/2015-05-05-graceful-handling-of-sigint-when-using-pythons-multiprocessingprocess.html
+- http://stackoverflow.com/questions/2342397/pythons-time-sleep-never-waking-up
+- https://blog.ionelmc.ro/2013/06/05/python-debugging-tools/
+- http://jessenoller.com/blog/2009/02/01/python-threads-and-the-global-interpreter-lock
+- https://wiki.python.org/moin/DebuggingWithGdb
+- http://stackoverflow.com/questions/11854519/python-multiprocessing-some-functions-do-not-return-when-they-are-complete-que
+- https://www.troyfawkes.com/learn-python-multithreading-queues-basics/
+- https://mkaz.tech/python-string-format.html
+- https://zapier.com/engineering/debugging-python-boss/
+- https://fangpenlin.com/posts/2012/08/26/good-logging-practice-in-python/
+- http://kronosapiens.github.io/blog/2014/07/28/understanding-package-imports-in-python.html
+- http://docs.quantifiedcode.com/python-anti-patterns/correctness/method_could_be_a_function.html
+- https://www.codementor.io/sheena/how-to-write-python-custom-exceptions-du107ufv9
+- http://tech.marksblogg.com/faster-django-testing.html
+- http://alexmic.net/flask-sqlalchemy-pytest/
+- https://gist.github.com/Spindel/1d07533ef94a4589d348

systemd.md

@@ -0,0 +1,48 @@
+# Notes - SystemD
+
+Important note, the `Environment=PATH=...` does not interopolate other environment variables
+- https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files
+- http://hokstadconsulting.com/devops/writing-systemd-units
+- http://sysadvent.blogspot.com/2015/12/day-17-grokking-systemd-for-fun-and.html
+- https://asylum.madhouse-project.org/blog/2015/09/09/systemd-job-monitoring/
+- https://fedoramagazine.org/systemd-template-unit-files/
+- http://www.tecmint.com/manage-services-using-systemd-and-systemctl-in-linux/
+- https://www.dynacont.net/documentation/linux/Useful_SystemD_commands/
+- `/usr/lib/systemd/system`
+
+```shell
+# After installing config
+sudo systemctl daemon-reload
+
+# Show available units
+sudo systemctl
+
+sudo systemctl status tsdb-recorder.service
+sudo systemctl restart tsdb-recorder.service
+sudo systemctl show tsdb-recorder.service
+
+# Logging
+journalctl -u  tsdb-recorder.service
+
+# To test systemd restarting
+# sudo kill -KILL $(ps aux | grep record | head -n1 | tr -s ' ' | cut -d ' ' -f 2)
+
+
+# Utilizations
+systemd-cgtop -m
+
+# Running services
+systemctl -t service | grep -v systemd
+
+# CSV format of unit,load,active,sub
+systemctl -t service | tr -s ' ' | tr -d '●' | awk '$1 ~ /^[a-z]/ {out=""; for(i=1;i<5;i++){out=out" "$i", "}; print out}'
+
+# If you have a failed service you removed, it will stay until you run this
+systemctl reset-failed
+
+# Remove stopped 1 off jobs
+sudo systemctl stop [email protected]
+
+# Show the config for the service
+sudo systemctl cat tsdb-recorder.service
+```