From 35e4517d41890334ddd3929ec902d79da9a3ffab Mon Sep 17 00:00:00 2001 From: Andres Hermosilla Date: Tue, 18 Jul 2017 10:24:44 -0700 Subject: [PATCH] First commit ... --- aws.md | 162 ++++++++++++++++++++++++++++++++++++++++++++++++++++ influxdb.md | 36 ++++++++++++ metrics.md | 29 ++++++++++ mongodb.md | 0 nodejs.md | 10 ++++ python.md | 23 ++++++++ systemd.md | 48 ++++++++++++++++ 7 files changed, 308 insertions(+) create mode 100644 aws.md create mode 100644 influxdb.md create mode 100644 metrics.md create mode 100644 mongodb.md create mode 100644 nodejs.md create mode 100644 python.md create mode 100644 systemd.md diff --git a/aws.md b/aws.md new file mode 100644 index 0000000..5278c6b --- /dev/null +++ b/aws.md @@ -0,0 +1,162 @@ +# AWS + +- Install cli tools for happiness + - `pip install awscli` + - `pip install aws-shell` + - `apt-get install jq` +- https://github.com/open-guides/og-aws +- https://medium.com/@kief/https-medium-com-kief-using-pipelines-to-manage-environments-with-infrastructure-as-code-b37285a1cbf5 +- http://creately.com/blog/diagrams/aws-templates-for-architecture-diagrams/ +- https://www.slideshare.net/AmazonWebServices/masterclass-advanced-security-best-practices +- https://www.nvteh.com/news/problems-with-public-ebs-snapshots + + +## EC2 +- The bread and butter of AWS +- Default into public subnet +- You can run a command across all instances if you have their agent installed +- `UserData` config allows you to provide some instance config or provisioning, which is run under sudo + - `/var/lib/cloud/instance/scripts/part-001` + - `ls /var/log/cloud-init-output.log` + +### Security Groups +- http://harish11g.blogspot.com/2015/06/best-practices-tips-on-amazon-web-services-security-groups-aws-security-managed-services.html +- http://www.stratoscale.com/blog/compute/aws-security-groups-5-best-practices/ + + +## Elastic Beanstalk +- Think of a Heroku setup, with server setup abstracted away +- Has a cli tool just like Heroku +- You choose your platform, eg PHP. python, node.js etc and go from there +- Orchestrates setting up EC2 instances with load balancing etc +- Sets everything up using dynamic on-the-flow CloudFormation templates + +**Links** +- http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/ebextensions.html +- https://github.com/lapygithub/eb_config_examples +- https://medium.com/trisfera/getting-to-know-and-love-aws-elastic-beanstalk-configuration-files-ebextensions-9a4502a26e3c +- http://blog.rudylee.com/2014/05/22/configuring-elastic-beanstalk-environment-with-ebextensions/ +- https://tech.pic-collage.com/two-must-have-configurations-when-using-aws-elastic-beanstalk-with-docker-720ce7c5ec91 +- https://github.com/awslabs/eb-demo-php-simple-app +- http://blog.flux7.com/blogs/docker/10-steps-deploying-docker-containers-on-elastic-beanstalk +- https://www.slideshare.net/AmazonWebServices/running-microservices-and-docker-on-aws-elastic-beanstalk-august-2016-monthly-webinar-series +- `.elasticbeanstalk/config.yaml` +- `.ebextensions/options.config` +- `.ebextensions/options.config` + + +## VPC +- Plan out your subnets or plan to fail! +- Is software defined networking + +**Links** +- https://www.slideshare.net/gsilverm/aws-vpc-in +- https://medium.com/aws-activate-startup-blog/practical-vpc-design-8412e1a18dcc +- https://charity.wtf/2016/03/23/aws-networking-environments-and-you/ +- http://blog.flux7.com/blogs/aws/vpc-best-configuration-practices +- http://cloudacademy.com/blog/aws-vpc-configuration-five-kick-yourself-mistakes/ +- https://www.pluralsight.com/blog/it-ops/simplify-routing-how-to-organize-your-network-into-smaller-subnets + + +#### Subnets +- A VPC has multiple subnets +#### Route Tables +- Routes you associate with your subnet + +#### NACLS +- Return/response traffic can come in through a range of ports, make sure to leave that range open +http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_NACLs.html + + +## S3 +- Buckets of storage yo + + +## CloudWatch +- Detailed monitoring vs regular is more granular interval of recording +- Has alarms and alerting +- https://github.com/jorgebastida/awslogs +- http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html +- http://blog.brianbeach.com/2014/09/cloudwatch-logs-push.html +- http://zoltanaltfatter.com/2017/01/13/centralized-logging-with-cloudwatch-logs/ +- https://cloudacademy.com/blog/centralized-log-management-with-aws-cloudwatch-part-1-of-3/ +- https://alestic.com/2010/12/ec2-user-data-output/ +- https://aws.amazon.com/blogs/devops/send-ecs-container-logs-to-cloudwatch-logs-for-centralized-monitoring/ + +## Route 53 +- DNS management +- Can configure health checks to request a specific domain or ip +- Allows you to create complex routing configurations with traffic policies etc + +## CloudFormation +Infrastructure as code! You can create yaml|json defintions of your infrastructure stack which creates your +stack. +- You can get a cost estimate from a CloudFormation template +http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-paying.html +- You can reference other CloudFormation templates for organization sake +- Use `AWS::CloudFormation::Init` for initial setup +- `aws cloudformation validate-template --template-body file:////home//local//test//sampletemplate.json` + +**Links** +- https://github.com/toddm92/aws/wiki/CloudFormation-Best-Practices +- http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-paying.html +- https://www.slideshare.net/AmazonWebServices/dvo304-aws-cloudformation-best-practices +- https://www.quora.com/Why-are-there-no-AWS-CloudFormation-success-stories +- https://paperairoplane.net/?p=680 +- http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-init.html +- https://blog.engelke.com/2012/12/28/provisioning-a-server-with-cloudformation/ + +## IAM +### Groups +- Have policies, up to 10 policies per group http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-limits.html +- A user can belong to many groups + +### Users +- Can have api only access or console access or both +- Can belong to one or many groups +- Can be assigned specific policies vs policies inherited from group membership + +### Policy +- Dictates what you can or can't do +- Inline policies are for one-off special snowflakes and should be avoided generally + +### Roles +- You can assign a role to a service, eg an EC2 instance can have a role of PublicWebServer which has its own policies +- Think of a role as an identity, not a group +- Leverage for cross account access + +### KMS +Storing encryption keys in the cloud + +- https://blog.fugue.co/2015-04-21-aws-kms-secrets.html +- https://github.com/fugue/credstash +- https://medium.com/@mda590/simple-secrets-management-via-aws-ec2-parameter-store-737477e19450 +- https://github.com/mozilla/sops + +## AWS Config +- Run a set of checks against resources to ensure adherance to certain standards +- $2 a rule + +## CloudTrail +Think of it as the auditd of linux but for AWS + +## CodeCommit +- Private git repos which can have triggers! +- http://www.paul-kearney.com/2015/09/migrating-from-github-to-aws-codecommit.html +- If you have an instance role that has access to a repos you need to use the credential helper + http://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-https-windows.html#setting-up-https-windows-credential-helper + ```shell + git config --global credential.helper '!aws codecommit credential-helper $@' + git config --global credential.UseHttpPath true + ``` + + +## Other +- Use Resource GRoups to get a birds eye view of everything, across AZs + +### AWS Certificate Mananger +- http://serverfault.com/questions/9708/what-is-a-pem-file-and-how-does-it-differ-from-other-openssl-generated-key-file +- https://docs.aws.amazon.com/acm/latest/userguide/acm-services.html +- http://docs.aws.amazon.com/acm/latest/userguide/import-certificate.html +- https://jamielinux.com/docs/openssl-certificate-authority/certificate-revocation-lists.html +- https://www.schneier.com/academic/paperfiles/paper-pki-ft.txt \ No newline at end of file diff --git a/influxdb.md b/influxdb.md new file mode 100644 index 0000000..a2fcbe9 --- /dev/null +++ b/influxdb.md @@ -0,0 +1,36 @@ +# Notes - Influxdb + + +- measurements ~ table +- fields are non-indexed columns +- If you are using a field & tag that are the same, use a different name + eg fields: {class: 1, sn: 23434, cpu 0.232}, tags: {class:1} vs tags:{pclass:1} + makes querying easier +- tags are indexed columns and should be used on columns that don't have lots of unique values make filtering faster (~10x) +- tune precision (seconds, minutes, etc) +- values default to int +- types are bool, int, float, str and after the first record the field to type mapping is set +- https://cobe.io/blog/posts/memory-adventures-with-influxdb/ +- https://vimeo.com/169742452 +- https://vimeo.com/170035101 +- https://vimeo.com/195058724 +- https://vimeo.com/198723778 +- https://vimeo.com/200898048 +- https://grisha.org/blog/2015/03/20/influxdb-data/ +- http://www.ryandaigle.com/a/time-series-db-design-with-influx +- http://roobert.github.io/2015/10/10/Columned-Graphite-Data-in-InfluxDB/ +- https://maxchadwick.xyz/blog/monitoring-magento-cron +- http://techblog.shutl.com/2016/06/moving-from-graphite-to-influxdb/ +- http://ryanfrantz.com/posts/solving-monitoring/ +- https://blog.karmawifi.com/the-software-behind-karma-metrics-and-alerting-57cb1d0ddaf#.1k9rpu78w +- http://blog.aiven.io/2016/04/monitoring-influxdb-telegraf-grafana.html +- https://www.digitalocean.com/community/tutorials/how-to-analyze-system-metrics-with-influxdb-on-centos-7 +- http://www.vishalbiyani.com/graphing-performance-with-collectd-influxdb-grafana/ +- https://milinda.svbtle.com/cluster-and-service-monitoring-using-grafana-influxdb-and-collecd +- https://streamsets.com/blog/streamsets-monitoring-grafana-influxdb-jmxtrans/ +- http://rmoff.net/2016/05/12/monitoring-logstash-ingest-rates-with-influxdb-and-grafana/ +- http://jansipke.nl/monitoring-hosts-with-collectd-influxdb-and-grafana/ +- https://markri.nl/custom-monitoring-stack/ +- http://tech.aabouzaid.com/2016/08/monitoring-processes-with-telegraf-influxdb-kapacitor-python.html +- https://github.com/mark-rushakoff/awesome-influxdb +- https://influxdata.com/tldr-influxdb-tech-tips-february-16-2017/ diff --git a/metrics.md b/metrics.md new file mode 100644 index 0000000..83709b6 --- /dev/null +++ b/metrics.md @@ -0,0 +1,29 @@ +# Metrics + +- https://kartar.net/2015/08/monitoring-survey-2015---metrics/ +- http://dieter.plaetinck.be/post/a_few_common_graphite_problems_and_how_they_are_already_solved/ +- https://matt.aimonetti.net/posts/2013/06/26/practical-guide-to-graphite-monitoring/ +- https://artofmonitoring.com/ +- http://le-huy.blogspot.com/2013/08/some-useful-tips-for-graphite.html +- http://stackoverflow.com/questions/18108047/statsd-graphite-naming-conventions-for-metrics +- http://obfuscurity.com/2012/05/Organizing-Your-Graphite-Metrics +- http://florin.myip.org/blog/massive-scalability-when-querying-metrics-stored-graphite +- https://prometheus.io/docs/practices/naming/ +- http://metrics20.org/ +- http://www.slideshare.net/Dieterbe/metrics-stack-20 +- https://docs.newrelic.com/docs/plugins/plugin-developer-resources/developer-reference/metric-naming-reference +- https://support.signalfx.com/hc/en-us/articles/201213425-Metrics-dimensions-and-metadata +- https://support.signalfx.com/hc/en-us/articles/201213445 +- http://www.codinginstinct.com/2013/03/metrics-and-graphite.html +- https://codeascraft.com/2011/02/15/measure-anything-measure-everything/ +- https://codeascraft.com/category/infrastructure/ +- http://www.xkyle.com/graph-everything-with-graphite/ +- http://shop.oreilly.com/product/0636920035794.do +- http://www.programblings.com/2013/11/14/sensu-checks-to-report-metrics/ +- https://github.com/ryandotsmith/l2met +- http://www.ryandaigle.com/a/time-series-db-design-with-influx +- https://github.com/lukevenediger/statsd.net/blob/master/statsd.net/Documentation/guidance/metric-anti-patterns.md +- https://influxdata.com/blog/getting-started-with-sending-statsd-metrics-to-telegraf-influxdb/ +- https://blog.raintank.io/logs-and-metrics-and-graphs-oh-my/ +- http://www.robustperception.io/monitoring-not-just-for-outages/ +- http://www.robustperception.io/rate-then-sum-never-sum-then-rate/ \ No newline at end of file diff --git a/mongodb.md b/mongodb.md new file mode 100644 index 0000000..e69de29 diff --git a/nodejs.md b/nodejs.md new file mode 100644 index 0000000..94fd725 --- /dev/null +++ b/nodejs.md @@ -0,0 +1,10 @@ +# NodeJS + +## Debugging node.js +- http://techblog.netflix.com/2014/11/nodejs-in-flames.html +- https://nodejs.org/en/docs/guides/simple-profiling/ +- https://blog.heroku.com/node-habits-2016 +- http://www.slideshare.net/yunongx/debugging-node-in-prod +- https://github.com/node-inspector/v8-profiler +- https://blog.risingstack.com/finding-a-memory-leak-in-node-js/ +- http://www.brendangregg.com/blog/2014-09-17/node-flame-graphs-on-linux.html \ No newline at end of file diff --git a/python.md b/python.md new file mode 100644 index 0000000..9feb315 --- /dev/null +++ b/python.md @@ -0,0 +1,23 @@ +# Notes - Python + + +## Python Resources +- http://jtushman.github.io/blog/2014/01/14/python-|-multiprocessing-and-interrupts/ +- http://www.programcreek.com/python/example/57787/multiprocessing.JoinableQueue +- https://noswap.com/blog/python-multiprocessing-keyboardinterrupt +- https://zhimingwang.org/blog/2015-05-05-graceful-handling-of-sigint-when-using-pythons-multiprocessingprocess.html +- http://stackoverflow.com/questions/2342397/pythons-time-sleep-never-waking-up +- https://blog.ionelmc.ro/2013/06/05/python-debugging-tools/ +- http://jessenoller.com/blog/2009/02/01/python-threads-and-the-global-interpreter-lock +- https://wiki.python.org/moin/DebuggingWithGdb +- http://stackoverflow.com/questions/11854519/python-multiprocessing-some-functions-do-not-return-when-they-are-complete-que +- https://www.troyfawkes.com/learn-python-multithreading-queues-basics/ +- https://mkaz.tech/python-string-format.html +- https://zapier.com/engineering/debugging-python-boss/ +- https://fangpenlin.com/posts/2012/08/26/good-logging-practice-in-python/ +- http://kronosapiens.github.io/blog/2014/07/28/understanding-package-imports-in-python.html +- http://docs.quantifiedcode.com/python-anti-patterns/correctness/method_could_be_a_function.html +- https://www.codementor.io/sheena/how-to-write-python-custom-exceptions-du107ufv9 +- http://tech.marksblogg.com/faster-django-testing.html +- http://alexmic.net/flask-sqlalchemy-pytest/ +- https://gist.github.com/Spindel/1d07533ef94a4589d348 \ No newline at end of file diff --git a/systemd.md b/systemd.md new file mode 100644 index 0000000..8cbfb3c --- /dev/null +++ b/systemd.md @@ -0,0 +1,48 @@ +# Notes - SystemD + +Important note, the `Environment=PATH=...` does not interopolate other environment variables +- https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files +- http://hokstadconsulting.com/devops/writing-systemd-units +- http://sysadvent.blogspot.com/2015/12/day-17-grokking-systemd-for-fun-and.html +- https://asylum.madhouse-project.org/blog/2015/09/09/systemd-job-monitoring/ +- https://fedoramagazine.org/systemd-template-unit-files/ +- http://www.tecmint.com/manage-services-using-systemd-and-systemctl-in-linux/ +- https://www.dynacont.net/documentation/linux/Useful_SystemD_commands/ +- `/usr/lib/systemd/system` + +```shell +# After installing config +sudo systemctl daemon-reload + +# Show available units +sudo systemctl + +sudo systemctl status tsdb-recorder.service +sudo systemctl restart tsdb-recorder.service +sudo systemctl show tsdb-recorder.service + +# Logging +journalctl -u tsdb-recorder.service + +# To test systemd restarting +# sudo kill -KILL $(ps aux | grep record | head -n1 | tr -s ' ' | cut -d ' ' -f 2) + + +# Utilizations +systemd-cgtop -m + +# Running services +systemctl -t service | grep -v systemd + +# CSV format of unit,load,active,sub +systemctl -t service | tr -s ' ' | tr -d '●' | awk '$1 ~ /^[a-z]/ {out=""; for(i=1;i<5;i++){out=out" "$i", "}; print out}' + +# If you have a failed service you removed, it will stay until you run this +systemctl reset-failed + +# Remove stopped 1 off jobs +sudo systemctl stop tsdb-alert@fail-cat.service.service + +# Show the config for the service +sudo systemctl cat tsdb-recorder.service +```