diff --git a/.gitignore b/.gitignore new file mode 100644 index 00000000..b0c519bf --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +usr/local/src/unit-tests/shunit2 diff --git a/COPYING b/COPYING index d71ac74c..39789d32 100644 --- a/COPYING +++ b/COPYING @@ -1,4 +1,4 @@ - Copyright 2011 Vizrt + Copyright 2011-2015 Escenic Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/README b/README index d5ca69e6..fbcd9ef3 100644 --- a/README +++ b/README @@ -1,10 +1,16 @@ +======== +Note! +======== +This version of ece-scripts is currently being modified to support java7, tomcat7 and content engine 5.6. During this transition the latest version of ece-scripts will not work correctly for previously released engine versions. + + ======== Overview ======== These are scripts to help template developers and administrators manage various components of the Escenic Content Engine. The scripts are experimental and not yet a part of the official distribution of -Escenic Content Engine from Vizrt Online. +Escenic Content Engine. ======================== Project directory layout @@ -13,60 +19,39 @@ The file structure ressembles where you'd typically put the various scripts and configuration files. That's all it is, only for documentational purposes :-) -========================== -Features of the ece script -========================== -$ ece help -Usage: /usr/bin/ece [-t ] [-i ] +=================================== +Features of the /usr/bin/ece script +=================================== +https://github.com/escenic/ece-scripts/blob/master/usr/share/doc/escenic/ece-guide.org -DESCRIPTION - -t --type - The following types are available: - engine - The Escenic Content Engine, this is the default - and is the assumed type if none is specified. - search - A standalone search indexer and solr instance - rmi-hub - The RMI hub responsible for the internal - communication between the ECE instances. - - -i --instance - The type instance, such as editor1, web1 or search1 - - -p --publication - Needed only for updating publication resources - - -r --resource - Needed only for updating publication resources. - Must be one of: content-type, feature, layout, layout-group - image-version, menu - -v --verbose - Prints out debug statements, useful for debugging. +=================================== +Features of the /usr/bin/ece-import script +=================================== +https://github.com/escenic/ece-scripts/blob/master/usr/share/doc/escenic/ece-import-guide.org -The following commands are available: - applog the type's app server log - assemble runs the Assembly Tool *) - clean removes temporary files created by /home/torstein/bin/ece *) - deploy deploys the assembled EAR *) - help prints this help screen - kill uses force to stop the type - log the type's Log4J log - outlog the [ece#engine] script log (system out log) - restart restarts the type - start starts the type - status checks if the type is running - stop stops the type - threaddump write a thread dump to standard out (system out log) - update update publication resources - versions lists the ECE component versions +============================================ +Features of the /usr/sbin/ece-install script +============================================ +https://github.com/escenic/ece-scripts/blob/master/usr/share/doc/escenic/ece-install-guide.org -*) only applicable if type is 'engine' +=========================================== +Features of the /usr/bin/system-info script +=========================================== +https://github.com/escenic/ece-scripts/blob/master/usr/share/doc/escenic/system-info-guide.org -========================== -Feedback -========================== +==================================== +Features of the /usr/bin/vosa script +==================================== +https://github.com/escenic/ece-scripts/blob/master/usr/share/doc/vizrt/vosa-guide.org +======== +Feedback +======== Hope you enjoy the ece /usr/bin and intit.d scripts. All feedback, come hither! --torstein@escenic.com +Escenic Support support(at)escenic.com +(The founder of this repo , Torstein, has move(d) on to other challenges) +Commits to this repo will also show in our vosa hipchat room diff --git a/TODO.md b/TODO.md new file mode 100644 index 00000000..4c039a9f --- /dev/null +++ b/TODO.md @@ -0,0 +1,65 @@ + +# Version 3.x aka "akbash" + +## /usr/sbin/ece-install + +### ✔ Install ECE from APT +Support for installing ECE & plugins using the escenic APT +repositories. + +- ✔ Editor profile (Escenic Content Engine) +- ✔ Presentation profile (Escenic Content Engine) +- ✔ Search profile (ECE's indexer-webapp + Solr) +- ✔ DB profile, ECE +- ✔ DB profile, ECE plugins +- ✔ Create publication(s) profile → Move to `/usr/bin/ece` +- ✔ Analysis profile (Escenic Analysis Engine) +- ✔ Analysis DB profile +- ✔ Cache server profile +- ✔ Assembly Tool + +### ✔ Install ECE from RPMs +Support for installing ECE & plugins using the escenic RPM packages. + +- ✔ Editor profile (Escenic Content Engine) +- ✔ Presentation profile (Escenic Content Engine) +- ✔ Search profile (ECE's indexer-webapp + Solr) +- ✔ DB profile, ECE +- ✔ DB profile, ECE plugins +- ✔ Create publication(s) profile → Move to `/usr/bin/ece` +- ✔ Analysis profile (Escenic Analysis Engine) +- ✔ Analysis DB profile +- ✔ Cache server profile +- ✔ Assembly Tool + +### ✔ Support for Varnish 4 + +Varnish 3 has reached its end of life and Varnish 4 is in the official +repositories of Debian stable, Ubuntu LTS and CentOS 7 and RedHat 7. + +### ✔ YAML configuration file format + +See the [unit tests](usr/local/src/unit-tests/ece-install-conf-file-reader-test.sh) for +configuration examples. + +### ✔ Remove the interactive mode. + +It's rarely used and complicates the source code and configuration +options unnecessarily. + +## ✔ /usr/bin/ece + +- ✔ New sub command: /usr/bin/ece create-publication + +- ✔ Easy to extend, e.g. put `create-publication.sh` in a directory, + e.g.`ece.d`, and get `ece create-publication`. The + `create-publication` sub command is then included in TAB completion, + help screens and `man` pages. + + +## Version 4.x +### Easier to extend `ece-install` + +Preferably in any language. Put a file in a directory and +`ece-install` will execute that file at a predictable point in time. + diff --git a/etc/bash_completion.d/ece b/etc/bash_completion.d/ece index f22da95f..0db3f9b4 100644 --- a/etc/bash_completion.d/ece +++ b/etc/bash_completion.d/ece @@ -1,40 +1,129 @@ -# auto completion for the /usr/bin/ece command. -_ece_commands() +# auto completion for the /usr/bin/ece command. Emacs: -*- sh -*- mode + +_get_sub_commands_for_ece() { + cat </dev/null + done + + declare -F | + grep -q -w "${cmd_completion_function}" && + "${cmd_completion_function}" +} +complete -F _get_completions_for_ece_command ece diff --git a/etc/bash_completion.d/ece-deploy b/etc/bash_completion.d/ece-deploy new file mode 100644 index 00000000..d1b7395f --- /dev/null +++ b/etc/bash_completion.d/ece-deploy @@ -0,0 +1,33 @@ +# auto completion for the ece-deploy command. -*- sh -*- + +_ece-deploy_commands() +{ + local cur=${COMP_WORDS[COMP_CWORD]} + local prev=${COMP_WORDS[COMP_CWORD-1]} + + options=" + --conf + --ear + --force + --list-deployments + --update-publication-resources + " + + # default completions is the list of options + case "$prev" in + --conf) + completions="http://" + ;; + --ear) + completions="http://" + ;; + *) + completions="$options" + ;; + esac + + COMPREPLY=( $(compgen -W "$completions" -- $cur) ) +} + +complete -o default -F _ece-deploy_commands ece-deploy + diff --git a/etc/bash_completion.d/ece-import b/etc/bash_completion.d/ece-import new file mode 100644 index 00000000..4ff3eafc --- /dev/null +++ b/etc/bash_completion.d/ece-import @@ -0,0 +1,32 @@ +# auto completion for the ece-import command. -*- sh -*- + +_ece-import_commands() +{ + local cur=${COMP_WORDS[COMP_CWORD]} + local prev=${COMP_WORDS[COMP_CWORD-1]} + + options=" + --directories-only + --escenic-group + --escenic-user + --import-archive + --name + --password + --publication + --uri + --user + --version + -V + -f + -n + -p + " + + # default completions is the list of options + completions="$options" + + COMPREPLY=( $(compgen -W "$completions" -- $cur) ) +} + +complete -o default -F _ece-import_commands ece-import + diff --git a/etc/bash_completion.d/ece-install b/etc/bash_completion.d/ece-install new file mode 100644 index 00000000..7ce24308 --- /dev/null +++ b/etc/bash_completion.d/ece-install @@ -0,0 +1,24 @@ +# auto completion for the ece-install command. -*- sh -*- + +_ece-install_commands() +{ + local cur=${COMP_WORDS[COMP_CWORD]} + local prev=${COMP_WORDS[COMP_CWORD-1]} + + options=" + --conf-file + --verbose + --version + -V + -f + -v + " + + # default completions is the list of options + completions="$options" + + COMPREPLY=( $(compgen -W "$completions" -- $cur) ) +} + +complete -o default -F _ece-install_commands ece-install + diff --git a/etc/bash_completion.d/vosa b/etc/bash_completion.d/vosa new file mode 100644 index 00000000..5e84442c --- /dev/null +++ b/etc/bash_completion.d/vosa @@ -0,0 +1,19 @@ +# auto completion for the vosa command. -*- sh -*- + +_vosa_commands() +{ + local cur=${COMP_WORDS[COMP_CWORD]} + local prev=${COMP_WORDS[COMP_CWORD-1]} + + commands=$( + vosa commands | cut -f3 -d' ' | cut -f1 -d':' + ) + + # default completions is the list of commands + completions="$commands" + + COMPREPLY=( $(compgen -W "$completions" -- $cur) ) +} + +complete -o default -F _vosa_commands vosa + diff --git a/etc/cron.d/periodic-check b/etc/cron.d/periodic-check new file mode 100644 index 00000000..9f3fae0b --- /dev/null +++ b/etc/cron.d/periodic-check @@ -0,0 +1,4 @@ +MAILTO=root + +* * * * * nagios /bin/sleep 20; if [ -x /usr/lib/periodic-check/main ]; then /usr/lib/periodic-check/main; fi + diff --git a/etc/cron.daily/remove-old-escenic-log-files b/etc/cron.daily/remove-old-escenic-log-files new file mode 100755 index 00000000..71d2a483 --- /dev/null +++ b/etc/cron.daily/remove-old-escenic-log-files @@ -0,0 +1,33 @@ +#! /usr/bin/env bash + +# cron script to remove old escenic related log files. It's safe to +# install this cron script on all servers, also the ones that don't +# have Escenic Content Engine or Escenic Analysis Engine installed. + +function remove_old_escenic_log_files() { + if [ ! -e /etc/init.d/ece ]; then + return + fi + + /etc/init.d/ece remove-old-log-files +} + +if [ -r /etc/default/ece ]; then + source /etc/default/ece + if [ ${remove_old_log_files-0} -eq 1 ]; then + remove_old_escenic_log_files + fi +fi + +# Added to take care of left over ear files in escenic cache by ece-install and ece-deploy +if [ -r /etc/escenic/ece.conf ]; then + source /etc/escenic/ece.conf + if [ ! -z "${cache_dir}" ] && [ -d "${cache_dir}" ] ; then + nice find "${cache_dir}" -maxdepth 1 -type f -mtime +5 -delete + fi + +fi + +# Now clean the /tmp directory as well +nice find /tmp -type f -mtime +3 -delete + diff --git a/etc/default/ece b/etc/default/ece index f7b2368b..ac916343 100644 --- a/etc/default/ece +++ b/etc/default/ece @@ -4,13 +4,13 @@ # instances. If you only have one # instance, use "default" as its # name. If you don't have any instances of that type on this host, # then just set the variable to an empty string. -engine_instance_list="app1 app2" -search_instance_list="search1" -analysis_instance_list="analysis1" +engine_instance_list="" +search_instance_list="" +analysis_instance_list="" # You only need one hub in an ECE cluster, so only set this one to -# true on one host. -enable_rmi_hub=true +# true on one host. 1 means on, 0 means off. +enable_rmi_hub=0 # The location of the ece script, default is /usr/bin/ece ece_script=/usr/bin/ece @@ -20,3 +20,7 @@ ece_script=/usr/bin/ece ece_unix_group=escenic ece_unix_user=escenic +# Do you want cron to remove (5 days) old log files every day? +# (default is yes, i.e. 1. Set 0 to turn this off) +remove_old_log_files=1 + diff --git a/etc/default/system-info b/etc/default/system-info new file mode 100644 index 00000000..ec078780 --- /dev/null +++ b/etc/default/system-info @@ -0,0 +1,3 @@ +# If you have blockdiag installed, but don't want system-info to +# generate diagrams, uncomment the following line: +# do_not_generate_diagram=1 diff --git a/etc/escenic/ece.conf b/etc/escenic/ece.conf index 2f3ad3bf..5aca995c 100644 --- a/etc/escenic/ece.conf +++ b/etc/escenic/ece.conf @@ -1,57 +1,88 @@ -# ece script configruation -*- sh -*- +# ece script configruation -*- conf -*- # version: $Revision: #1 $ $Date: 2011/02/18 $ # author: torstein@escenic.com # last update by: $Author: shud $ -######################################################################## +######################################################################### # Section I: Variables you probably want to change or at least verify # make sense. -######################################################################## +######################################################################### -######################################################################## +######################################################################### # Where the escenic content engine itself is installed -######################################################################## +######################################################################### ece_home=/opt/escenic/engine -######################################################################## +######################################################################### # Set this one if you're running more than one ECE instance on your # host. Usually, you'll set it to the instance name. If you're running # one ECE instance on your system, you may omit this one. -######################################################################## +######################################################################### # ece_server=rolling # ece_server_hostname=quanah.escenic.com -######################################################################## +######################################################################### +# Set this one if you wish to have a Nursery configuration layer +# special to the kind of enviornment you're running. An enviornment in +# this context is: development, test, staging and prodution. +######################################################################### +# ece_environment=production + +######################################################################### # Setting the java home, yes lowercase is correct ;-) -######################################################################## -java_home=/usr/lib/jvm/java-6-sun +######################################################################### +java_home=/usr/lib/jvm/jdk-8-oracle-x64 -######################################################################## +######################################################################### # Possible options are: tomcat, jboss, resin & oc4j -######################################################################## +######################################################################### appserver=tomcat -######################################################################## +######################################################################### +# App server port. If this variable is unset, ece will try to figure +# out the port number by itself. ece-.conf should override +# this when there's more than one ECE instance on the same host. +######################################################################### +appserver_port=8080 + +######################################################################### # Section II: Variables that are sensible defaults but may be changed # to accommodate the conventions or tastes of your OS or company. -######################################################################## +######################################################################### -######################################################################## +######################################################################### # Related to the ece script/ECE/hub/search -######################################################################## +######################################################################### assemblytool_home=/opt/escenic/assemblytool cache_dir=/var/cache/escenic -ece_security_configuration_dir=/etc/escenic/engine/common/security + +######################################################################### +# Configuration directories +######################################################################### +# The root of all Escenic configuration files (not only ECE) +escenic_conf_dir=/etc/escenic + +# The directory for the Escenic Analysis Engine/Stats' configuration +# files. Note that this is different from the EAE plugin for Content +# Studio, which has its configuration with the other ECE Nursery +# components. +analysis_conf_dir=$escenic_conf_dir/analysis +ece_security_configuration_dir=${escenic_conf_dir}/engine/common/security +rmi_hub_conf=${escenic_conf_dir}/rmi-hub + +######################################################################### +# Other important directories +######################################################################### +data_dir=/var/lib/escenic log_dir=/var/log/escenic -pid_dir=/var/run/escenic -rmi_hub_conf=/etc/escenic/rmi-hub +run_dir=/var/run/escenic rmi_hub_home=$ece_home/contrib/rmi-hub solr_home=/var/lib/escenic/solr tmp_dir=/tmp -######################################################################## +######################################################################### # When making a deployment, ece will default wise, deploy the EAR # generated with "ece assemble" on the application server. The # webapps included in the EAR can be filtered with the @@ -65,12 +96,19 @@ tmp_dir=/tmp # webapps listed here, will be deployed. # # Note: this is currently only supported when appserver=tomcat. -# +# # deploy_webapp_white_list="escenic-admin pbe" -######################################################################## +######################################################################### +# Memcached support. 'ece -i deploy' will per default add +# support for using memcached with your publication. If you wish to +# turn this off, you can set this value to 0. +######################################################################### +# enable_memcached_support=1 + +######################################################################### # Where to find the various application servers -######################################################################## +######################################################################### jboss_home=/opt/jboss oc4j_home=/opt/oc4j resin_home=/opt/resin @@ -81,48 +119,133 @@ tomcat_home=/usr/share/tomcat6 # binaries, just with different data. # tomcat_base=/opt/tomcat-editor1 -######################################################################## +###################################################################### +# If you've compiled or installed APR support (native Tomcat +# libraries, using the Apache APR library), you specify the install +# directory here, if not comment it out. +###################################################################### +apr_lib_dir=/usr/lib + +######################################################################### # JVM related (the Java Virtual Machine) -######################################################################## +######################################################################### -######################################################################## +######################################################################### # The minimum and maximum heap sizes for the Java process -######################################################################## -min_heap_size=2048m -max_heap_size=2048m +######################################################################### +min_heap_size=256m +max_heap_size=256m -######################################################################## +######################################################################### # The permgen size is the amount of memory the JVM sets aside to class # defintions. If you are experiencing constant OutOfMemory # exceptions/errors and are using EAR deployment, you may want to # increase these. -######################################################################## +######################################################################### min_permgen_size=64m max_permgen_size=256m -enable_heap_dump=0 enable_remote_debugging=0 remote_debugging_port=5005 enable_remote_monitoring=0 remote_monitoring_port=5792 -##################################################################### +######################################################################### +# Settings related to the JVM garbage collection, don't touch these if +# you're not sure of what you're doing and have tested your changes +# properly. Note that ece takes care of the GC logging, so you don't +# have to cater for that here. +######################################################################### +jvm_gc_settings=" +-XX:+CMSClassUnloadingEnabled +-XX:+CMSIncrementalPacing +-XX:+CMSPermGenSweepingEnabled +-XX:+ExplicitGCInvokesConcurrent +-XX:+UseCMSInitiatingOccupancyOnly +-XX:+UseConcMarkSweepGC +-XX:CMSInitiatingOccupancyFraction=65 +-XX:InitialCodeCacheSize=50m +-XX:MaxNewSize=250m +-XX:NewSize=250m +-XX:ParallelGCThreads=1 +-XX:ReservedCodeCacheSize=50m +-XX:SurvivorRatio=1 +" + +###################################################################### +# If you want ECE/EAE to use an HTTP proxy to access the internet, set +# the parameters you need here and remove the rest. Default is no HTTP +# proxy. See +# http://docs.oracle.com/javase/6/docs/technotes/guides/net/proxies.html +# for more details on the available options here. +###################################################################### +# jvm_proxy_settings=" +# -Dhttp.proxyHost=proxyserver +# -Dhttp.proxyPort=3128 +# -Dhttp.nonProxyHosts=localhost +# -Dhttps.proxyHost=proxyserver +# -Dhttps.proxyPort=3128 +# " + +###################################################################### +# This overrides the default connection settings of all classes that +# directly or indirectly open TCP connections inside the JVM. +# +# To ensue more stable connections to EAE, we set max timeouts +# here. If you're not using EAE, you probably don't need these and can +# comment out the variable. +# +# The values are in milliseconds and are set to 5 seconds (5000 +# milliseconds). +jvm_connection_settings=" + -Dsun.net.client.defaultConnectTimeout=1000 + -Dsun.net.client.defaultReadTimeout=5000 +" + +###################################################################### +# Tell the JVM to use this encoding everywhere (in stead of falling +# back to the system default). +###################################################################### +jvm_encoding=UTF-8 + +###################################################################### # Java will use IPv6 if it's available on the OS. This however, can # cause problems with applications that insists on using IPv4 # sockets. # # If e.g. Tomcat cannot create a socket on a given port (which is not # taken), you may need to set this one to '1'. Default is '0'. -##################################################################### +###################################################################### force_ipv4=0 -##################################################################### -# Is this a production system? -##################################################################### -is_production=1 - -##################################################################### -# Ask the JVM to create a heap dump when/if it crashes. -##################################################################### +###################################################################### +# Ask the JVM to create a heap dump when/if it crashes. +###################################################################### enable_heap_dump=1 heap_dump_dir=/var/crash/escenic + +###################################################################### +# Where 'ece -i backup' will put its snapshot +# backups. Default is /var/backups/escenic. +###################################################################### +backup_dir=/var/backups/escenic + +###################################################################### +# When removing old log files, how old should the oldest file be? +###################################################################### +old_log_file_max_age_in_days=5 + +###################################################################### +# HTTP authentication for the escenic-admin webapp (default is no HTTP +# auth) +###################################################################### +# escenic_admin_http_user= +# escenic_admin_http_password= + +###################################################################### +# HTTP authentication for the build server (default is no HTTP auth), +# this only used for when you use 'ece deploy' with a URI for an EAR +# a the build server. +###################################################################### +# builder_http_user= +# builder_http_password= diff --git a/etc/init.d/ece b/etc/init.d/ece index dc2a515c..c6602280 100755 --- a/etc/init.d/ece +++ b/etc/init.d/ece @@ -5,8 +5,8 @@ ### BEGIN INIT INFO # Provides: ece -# Required-Start: $remote_fs $network $syslog -# Required-Stop: $remote_fs $network $syslog +# Required-Start: $local_fs $network $syslog +# Required-Stop: $local_fs $network $syslog # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Escenic Content Engine @@ -15,7 +15,7 @@ ################################################################# # Usage: # * copy this script to /etc/init.d/ece -# +# # * copy its configuration file to /etc/default/ece (or # * /etc/conf.d/ece on Gentoo based disitributions) # @@ -25,31 +25,41 @@ # would be: update-rc.d ece defaults ####################################################################### +if [ $(whoami) != "root" ]; then + echo "Sorry, you must be root for running $0" + exit 1 +fi + ####################################################################### # Default values ####################################################################### dir_list=" /var/cache/escenic +/var/backups/escenic /var/crash/escenic /var/log/escenic +/var/lib/escenic /var/run/escenic " ece_script=/usr/bin/ece -# the values above may be overidden a file named the same as this -# init.d script, located in: +# The values above may be overidden a file named the same as this +# init.d script. This init.d configuration must also hold the +# variables controlling which ECE instances to start. The list of +# locations per default caters (at least) for Debian, RedHat & Gentoo +# based systems: conf_file_location_list=" /etc/default /etc/conf.d -/tmp +/etc/sysconfig " function read_conf_file() { for el in $conf_file_location_list; do - if [ -r $el/`basename $0` ]; then - source $el/`basename $0` + if [ -r $el/ece ]; then + source $el/ece found_conf=1 break fi @@ -67,7 +77,12 @@ function ensure_dirs_are_ok() if [ ! -d $el ]; then mkdir -p $el fi - chown -R $ece_unix_user:$ece_unix_group $el + + if [[ ${el} == "/var/cache"* || ${el} == "/var/lib"* ]]; then + chown --changes "${ece_unix_user-escenic}:${ece_unix_group-escenic}" $el + else + chown --recursive --changes "${ece_unix_user-escenic}:${ece_unix_group-escenic}" $el + fi done } @@ -80,25 +95,25 @@ function ensure_ece_script_is_ok() echo "$ece_script needs to be executable (do: chmod +x $ece_script)" exit 1 fi - + } function execute_command() { - if [ $enable_rmi_hub = "true" ]; then - su - $ece_unix_user -c "$ece_script -t rmi-hub $1" + if [ "${enable_rmi_hub-0}" -eq 1 ]; then + su - ${ece_unix_user-escenic} -c "$ece_script -t rmi-hub $1" fi - + for el in $engine_instance_list; do - su - $ece_unix_user -c "$ece_script -t engine -i $el $1" + su - "${ece_unix_user-escenic}" -c "$ece_script -t engine -i $el $1" done - + for el in $search_instance_list; do - su - $ece_unix_user -c "$ece_script -t search -i $el $1" + su - "${ece_unix_user-escenic}" -c "$ece_script -t search -i $el $1" done for el in $analysis_instance_list; do - su - $ece_unix_user -c "$ece_script -t analysis -i $el $1" + su - "${ece_unix_user-escenic}" -c "$ece_script -t analysis -i $el $1" done } diff --git a/etc/init.d/vosa b/etc/init.d/vosa new file mode 100644 index 00000000..575c5009 --- /dev/null +++ b/etc/init.d/vosa @@ -0,0 +1,114 @@ +#! /usr/bin/env bash + +# version: $Revision: #1 $ $Date: 2011/02/18 $ +# author: mogsie@vizrt.com + +### BEGIN INIT INFO +# Provides: vosa +# Required-Start: $remote_fs $network $syslog +# Required-Stop: $remote_fs $network $syslog +# Default-Start: 2 3 4 5 +# Default-Stop: 0 1 6 +# Short-Description: Vizrt Online Components +### END INIT INFO + +################################################################# +# Usage: +# * copy this script to /etc/init.d/vosa +# +# * copy its configuration file to /etc/default/vosa (or +# * /etc/conf.d/vosa on Gentoo based disitributions) +# +# * make sure it's executable, chmod +x /etc/init.d/vosa +# +# * add it to the desired runlevels. On Debian based systems this +# would be: update-rc.d vosa defaults +####################################################################### + +if [ $(whoami) != "root" ] ; then + echo "Sorry, you must be root to run $0" + exit 1 +fi + +####################################################################### +# Default values +####################################################################### +dir_list=" +/var/run/vizrt/vosa +" + +vosa_instance_list= +for a in /etc/vizrt/vosa/enabled.d/* ; do + if [ -L "$a" -a -r "$a" ] ; then + vosa_instance_list="$vosa_instance_list $(basename "$a")" + fi +done + +vosa_script=/usr/bin/vosa + +# The values above may be overidden a file named the same as this +# init.d script. This init.d configuration must also hold the +# variables controlling which vosa instances to start. The list of +# locations per default caters (at least) for Debian, RedHat & Gentoo +# based systems: +conf_file_location_list=" +/etc/default +/etc/conf.d +/etc/sysconfig +" + +function read_conf_file() +{ + for el in $conf_file_location_list; do + if [ -r $el/`basename $0` ]; then + source $el/`basename $0` + found_conf=1 + break + fi + done + +# if [ -z $found_conf ]; then +# echo "Couldn't find configuration for $0, exiting :-(" +# exit 1 +# fi +} + +function ensure_dirs_are_ok() +{ + for el in $dir_list; do + if [ ! -d $el ]; then + mkdir -p $el + fi +# don't change owner; root is fine... +# chown --changes -R $vosa_unix_user:$vosa_unix_group $el + done +} + +function ensure_vosa_script_is_ok() +{ + if [ ! -r $vosa_script ]; then + echo "Couldn't read $vosa_script" + exit 1 + elif [ ! -x $vosa_script ]; then + echo "$vosa_script needs to be executable (do: chmod +x $vosa_script)" + exit 1 + fi + +} + +function execute_command() +{ + for el in $vosa_instance_list; do + $vosa_script -i $el $1 + done +} + +if [ $1 ]; then + # todo: whitelist to start, stop, status. + read_conf_file + ensure_vosa_script_is_ok + ensure_dirs_are_ok + execute_command $1 +else + echo "Usage: `basename $0` " +fi diff --git a/etc/logrotate.d/tomcat b/etc/logrotate.d/tomcat new file mode 100644 index 00000000..95e40834 --- /dev/null +++ b/etc/logrotate.d/tomcat @@ -0,0 +1,9 @@ +/var/log/escenic/*.out{ + copytruncate + daily + rotate 7 + compress + missingok + size 5M +} + diff --git a/etc/munin/plugins/indexer_running b/etc/munin/plugins/indexer_running new file mode 100755 index 00000000..405ce9f5 --- /dev/null +++ b/etc/munin/plugins/indexer_running @@ -0,0 +1,37 @@ +#!/bin/bash +# + +#. $MUNIN_LIBDIR/plugins/plugin.sh +info_file="/var/cache/periodic-check/indexer_running" +state=$(cat $info_file | cut -f 3 -d ' ') +age=$(cat $info_file | cut -f 4 -d ' ') + +if [ "$1" = "config" ]; then + echo 'graph_title indexing delay' + echo "graph_args --base 1000 -r --lower-limit 0 " + echo 'graph_vlabel Minutes' + echo 'graph_order indexing_delay' +# echo 'graph_scale no' + echo 'graph_info Shows the time to get a document indexed by Escenic search. This is basically time difference between update time of head-tail file and entryUpdated field in SearchIndex table.' + echo 'graph_category system' + echo 'graph_period second' + echo "indexing_delay.label time to index current document" + echo "indexing_delay.draw LINE2" + echo "indexing_delay.min 0" + echo "indexing_delay.type GAUGE" + echo "indexing_delay.info time in $site in Bytes" + exit 0; +fi + +# Note: Counters/derive need to report integer values. Also we need +# to avoid 10e+09 and the like %.0f should do this. + +cur_epoch=$(date +%s) +sample_epoch=$(cat "$info_file" | head -n 1| cut -f 1 -d ' ') +#add 90 seconds to compensate freshness. +sample_epoch=$(echo "$sample_epoch + 90"| bc) + +if [ "$sample_epoch" -gt "$cur_epoch" ] && [ "$state" == "yes" ] && [ "$age" -ge 0 ]; then + printf "indexing_delay.value $age\n" +fi + diff --git a/etc/munin/plugins/site_length b/etc/munin/plugins/site_length new file mode 100755 index 00000000..2c4ce04b --- /dev/null +++ b/etc/munin/plugins/site_length @@ -0,0 +1,52 @@ +#!/bin/bash +# + +#. $MUNIN_LIBDIR/plugins/plugin.sh +info_file="/var/cache/periodic-check/site_length" +sites=($(cat $info_file | cut -f 2 -d ' ')) +lengths=($(cat $info_file | cut -f 3 -d ' ')) + +if [ "$1" = "autoconf" ]; then + if [ -r /proc/stat ]; then + echo yes + exit 0 + else + echo no + exit 0 + fi +fi + +if [ "$1" = "config" ]; then + + echo 'graph_title sites length' + echo "graph_order " ${sites[@]//./_} + echo "graph_args --base 1000 -r --lower-limit 0 " + echo 'graph_vlabel Bytes' +# echo 'graph_scale no' + echo 'graph_info This graph shows length of different pages.' + echo 'graph_category system' + echo 'graph_period second' + for site in "${sites[@]}"; do + echo "${site//./_}.label $site" + echo "${site//./_}.draw LINE2" + echo "${site//./_}.min 0" + echo "${site//./_}.type GAUGE" + echo "${site//./_}.info length/size of $site in Bytes" + done + exit 0 +fi + +# Note: Counters/derive need to report integer values. Also we need +# to avoid 10e+09 and the like %.0f should do this. + +cur_epoch=$(date +%s) +sample_epoch=$(cat "$info_file" | head -n 1| cut -f 1 -d ' ') +#add 90 seconds to compensate freshness. +sample_epoch=$(echo "$sample_epoch + 90"| bc) + +if [ "$sample_epoch" -gt "$cur_epoch" ]; then + for i in $(seq 1 ${#sites[@]}); do + printf "%s.value %.0f\n" ${sites[$i-1]//./_} ${lengths[$i-1]} + done +fi + diff --git a/etc/puppet/manifests/global-variables.pp b/etc/puppet/manifests/global-variables.pp new file mode 100644 index 00000000..badaf59e --- /dev/null +++ b/etc/puppet/manifests/global-variables.pp @@ -0,0 +1,13 @@ +# global variables +$control_host = "control" + +$apt_vizrt_user = "" +$apt_vizrt_password = "" +$apt_proxy_host = "$control_host" + +$db_user = "ece5user" +$db_vip = "" + +$privileged_ip_list = "" + +$mobilize_vip = "" diff --git a/known-issues.org b/known-issues.org new file mode 100644 index 00000000..31c64b2d --- /dev/null +++ b/known-issues.org @@ -0,0 +1,26 @@ +#+TITLE: Known Issues +#+AUTHOR: Escenic Cloud Team + +* ece-install +** 2013-01-17 cron script removes all 15 day old files on the system +- *Fixed in Version* :: 1.0-2013-01-17-468 + +- *Versions affected* :: 1.0-2013-01-12-x => 1.0-2013-01-17-x + +If you have installed and run either the =ece-install= all-in-one, +presentation or editorial profile between *2013-01-12* +and *2013-01-17*, you have encountered this issue. + +If you have =escenic-content-engine-installer= of any of the versions +affected, first remove the cron job if it's present: +#+BEGIN_SRC sh +# rm /etc/cron.daily/remove-old-escenic-cache-files +#+END_SRC + +Then upgrade to the latest version of =ece-install= so that you're +sure new runs of it will set up a proper cron job. If you're using +our APT repository at apt.escenic.com, this as easy as: +#+BEGIN_SRC sh +# apt-get update +# apt-get install escenic-content-engine-scripts +#+END_SRC diff --git a/usr/bin/ece b/usr/bin/ece index 24575a55..acd98e55 100755 --- a/usr/bin/ece +++ b/usr/bin/ece @@ -4,22 +4,43 @@ # the RMI hub and the indexer standalone instances. Type "ece help" # for a complete list of supported operations. # -# echo comments and suggestions > tkj@vizrt.com +# echo comments and suggestions > torstein@escenic.com +###################################################################### +# Needs to be here to bootstrap the ece script itself when it looks +# for the ece[-[a-z0-9]*].conf files. +# +# There is one to avoid this bootstrap dependency, and that is to set +# the ECE_CONF_LOCATIONS environment variable, however this is quite +# uncommon and generally speaking not recommended. +###################################################################### +escenic_conf_dir=/etc/escenic ###################################################################### # Script defaults, my be overriden in any of the configration files ###################################################################### -verbose=0 force_ipv4=0 +quiet=0 +everything_but_the_kitchen_sink=0 ###################################################################### # Dear user, don't touch anyting in this script, especially below this # line :-) ###################################################################### +ece_scripts_version="straight-from-github" type=engine -instance=default +instance=engine1 command="" +backup_exclude_binaries=0 +backup_exclude_conf=0 +backup_exclude_db=0 +backup_exclude_init=0 +backup_exclude_multimedia=0 +backup_exclude_solr=0 +backup_exclude_state=0 +wget_appserver_auth="" +curl_appserver_auth="" +log="" type_list=" engine @@ -27,13 +48,6 @@ search rmi-hub " -common_settings_list=" -ece_home -java_home -log_dir -pid_dir -" - hub_required_fields=" ece_server_hostname rmi_hub_conf @@ -44,116 +58,125 @@ engine_required_fields=" appserver assemblytool_home cache_dir +data_dir +escenic_conf_dir ece_home ece_security_configuration_dir enable_heap_dump enable_remote_debugging enable_remote_monitoring -is_production java_home solr_home +log_dir +run_dir +tmp_dir " search_required_fields=" appserver java_home solr_home +tmp_dir " analysis_required_fields=" appserver java_home +tmp_dir " -#################################################################### -# Will exit the ece execution if the last operation failed. While -# failing, it will print the message passed to the function. -#################################################################### -function exit_on_error() -{ - if [ $? -eq 1 ]; then - print $1 "FAILED, exiting :-(" - exit 1 - fi -} -#################################################################### -# debug/verbose method -#################################################################### -function debug() -{ - if [ $verbose -eq 1 ]; then - print $@ - fi -} - -function print() -{ - echo "$id" $@ -} - -function log() -{ - echo "$id" `date` $@ 2>/dev/null >> $log_file -} - -function verify_that_directory_and_file_are_writeable() -{ - dir=`dirname $1` - if [ ! -e $dir ]; then - print $dir " doesn't exist" - exit 1 - fi - if [ ! -w $dir ]; then - print $dir " isn't writable for user $USER" - exit 1 - fi +## Bootstrapping, load files from /usr/share/escenic/ece-scripts The +## method will first try to be smart, in case the user has copied the +## ece-scripts somewhere else., e.g.: moved everything to ~/ece-scrpts +## or /opt/escenic/ece-scripts, this should also work. +function init() { + # first, try to be nice + local dir=$(dirname $0)/../share/escenic/ece-scripts + + # then check the standard location + if [ ! -d $dir ]; then + dir=/usr/share/escenic/ece-scripts + fi + + if [ -d $dir ]; then + # load common libraries + common_libraries="common-bashing.sh common-io.sh common-os.sh common-ece.sh" + for el in $common_libraries; do + source $dir/${el} + done - if [ -e $1 ]; then - if [ ! -w $1 ]; then - print $1 "exists, " - print "but isn't write-able for user $USER" - print "check the permissions and that $USER is the correct one" - print "to run "`basename $0` - exit 1 - fi - fi + # load ece-install modules + for el in $dir/ece.d/*.sh; do + source $el + done + else + echo "I cannot find $(basename $0)'s dependencies, exiting :-(" + exit 1 + fi } #################################################################### # Ensures that all required fields are set. Will report all missing # required fields before failing. #################################################################### -function ensure_that_required_fields_are_set() -{ +function ensure_that_required_fields_are_set() { requirements_failed=0 - + for el in $@; do if [ -n "$(eval echo $`echo $el`)" ]; then continue fi - - print "You need to specifiy '$el' in your `basename $0`.conf" + + print "You need to specifiy '$el' in one of ${ece_conf_files_read[@]}" requirements_failed=1 done - if [ $requirements_failed -eq 1 ]; then + if [ "$requirements_failed" -eq 1 ]; then exit 1 fi } -function read_conf_file() -{ +ece_conf_files_read=() + +function read_conf_file() { for el in $conf_file_location_list; do if [ -r $el/$1 ]; then debug "found $1 in $el, reading it" source $el/$1 + ece_conf_files_read=($el/$1 ${ece_conf_files_read}) break fi done } -function set_common_settings() -{ +## If the system is installed using the recommended paths, the method +## will return a list of the instances configured in +## ${escenic_conf_dir}/ece-*.conf +export root_allowed_list_instances=1 +function get_instance_list() { + local allowed_types="engine analysis changelogd search rmi-hub" + instance_list="" + for el in $(\ls /etc/escenic/ece-*.conf 2>/dev/null); do + + local instance=$(basename $el .conf) + instance=${instance##ece-} + + for ele in $allowed_types; do + if [[ $instance == $ele ]]; then + instance="" + continue + fi + + local away="${ele}-" + instance=${instance##${away}} + done + instance_list="$instance_list $instance" + done + + echo $instance_list +} + +function set_common_settings() { # This behaviour can be overridden by specifying a list of # locations in the environment variable ECE_CONF_LOCATIONS if [ -n "$ECE_CONF_LOCATIONS" ] ; then @@ -161,43 +184,79 @@ function set_common_settings() else conf_file_location_list=" `dirname $0` - /etc/escenic/$type/instance/$instance - /etc/escenic/$type/host/`echo $HOSTNAME | tr '[A-Z]' '[a-z]'` - /etc/escenic/$type/common - /etc/escenic/$type - /etc/escenic + ${escenic_conf_dir}/$type/instance/$instance + ${escenic_conf_dir}/$type/host/`echo $HOSTNAME | tr '[A-Z]' '[a-z]'` + ${escenic_conf_dir}/$type/common + ${escenic_conf_dir}/$type + ${escenic_conf_dir} `dirname $0`/../etc " fi # main configuration file, may be overridden in the type and # instance specific ones. - read_conf_file `basename $0`.conf + read_conf_file $(basename $0).conf - if [ $instance = "default" ]; then - log_file=$log_dir/$type.out - pid_file=$pid_dir/$type.pid - else - log_file=$log_dir/$type-$instance.out - pid_file=$pid_dir/$type-$instance.pid - fi + log=$log_dir/$instance.out + pid_file=$run_dir/$instance.pid + + gc_log=${log_dir}/${instance}-gc.log - log_file_list=" + log4j_file_list=" + $log_dir/${HOSTNAME}-${instance}-messages + $log_dir/${instance}-messages $log_dir/messages $log_dir/Escenic-error.log " +} + +function set_type_settings() { + # optional: possible to have type specific conf file, + # will take precedences over the common one. + read_conf_file $(basename $0)-$type.conf +} + +function set_type_pid() { + if [ "$(uname)" == "Linux" ]; then + ps_options="auxww" + else + ps_options="-ef" + fi + + # Get a hold of the PID of the process. Although we've got the PID + # file, we stil get the PID from the OS here as we used it for + # sanity checking later on, see stop_type(). + if [ "$type" == "rmi-hub" ]; then + # we need to be able to differentiate between an ECE instance + # and an rmi hub, for this we use java.security.policy which + # the hub doesn't have. + type_pid=`ps $ps_options | grep -v java.security.policy | \ + awk "/Djava.rmi.server.hostname=$ece_server_hostname / && \ + !/awk/"' {print $2}'` + else + type_pid=`ps $ps_options | awk "/Descenic.server=$ece_server / && !/awk/"' {print $2}'` + fi + + debug "type_pid is now=$type_pid" +} + +function set_type_port() { host=localhost - if [ $appserver != "tomcat" ]; then + # port set in ece[-instance].conf takes precedence + if [ -n "${appserver_port}" ]; then + port=${appserver_port} + debug "appserver_port set in .conf files=${port}" + elif [ "$appserver" != "tomcat" ]; then debug "Only tomcat is supported for reading host/port right now, " debug "trying to make an educated guess" port=8080 else if [ -r $tomcat_base/logs/catalina.out ]; then # for tomcat6-user packaged tomcats (and perhaps others) - out_log=$tomcat_base/logs/catalina.out + out_log=$log_dir/${instance_name}-tomcat-catalina.out else - out_log=$log_file + out_log=$log fi if [ -r $out_log ]; then @@ -207,59 +266,34 @@ function set_common_settings() cut -d'-' -f2 ) fi - fi - -} -function set_type_settings() -{ - # optional: possible to have type specific conf file, - # will take precedences over the common one. - read_conf_file `basename $0`-$type.conf + debug "set_type_port=$port" } -function set_type_pid() -{ - # Get a hold of the PID of the process. Although we've got the PID - # file, we stil get the PID from the OS here as we used it for - # sanity checking later on, see stop_type(). - if [ $type = "rmi-hub" ]; then - # we need to be able to differentiate between an ECE instance - # and an rmi hub, for this we use java.security.policy which - # the hub doesn't have. - type_pid=`ps -ef | grep -v java.security.policy | \ - awk "/Djava.rmi.server.hostname=$ece_server_hostname / && \ - !/awk/"' {print $2}'` - else - type_pid=`ps -ef | awk "/Descenic.server=$ece_server / && !/awk/"' {print $2}'` - fi - - debug "type_pid is now=$type_pid" -} - -function set_instance_settings() -{ +function set_instance_settings() { # optional: possible to have instance specific conf files, # these will take precedence over the other two - read_conf_file `basename $0`-$instance.conf - read_conf_file `basename $0`-$type-$instance.conf + read_conf_file $(basename $0)-$instance.conf + read_conf_file $(basename $0)-$type-$instance.conf # at this point in the script flow, we have now read all the # possible combinations of conf files (common, type and instance) # and can now ensure that required fields are set and apply them. - if [ $type = "rmi-hub" ]; then + if [ "$type" == "rmi-hub" ]; then ensure_that_required_fields_are_set $hub_required_fields - elif [ $type = "search" ]; then + elif [ "$type" == "search" ]; then ensure_that_required_fields_are_set $search_required_fields - elif [ $type = "engine" ]; then + elif [ "$type" == "engine" ]; then ensure_that_required_fields_are_set $engine_required_fields + elif [ "$type" == "analysis" ]; then + ensure_that_required_fields_are_set $analysis_required_fields fi # We respect ece_server if it's set in any of the configuration # files. If it's not set there, it sets some sensible defaults. - if [ -z $ece_server ]; then - if [ $instance = "default" ]; then + if [ -z "$ece_server" ]; then + if [ "$instance" == "engine1" ]; then ece_server=${HOSTNAME} else ece_server=${HOSTNAME}-${instance} @@ -267,9 +301,9 @@ function set_instance_settings() fi set_type_pid - + # if type is rmi-hub, we don't' need more configuration. - if [ $type = "rmi-hub" ]; then + if [ "$type" == "rmi-hub" ]; then return fi @@ -282,74 +316,118 @@ function set_instance_settings() # * java.awt.headless is to avoid potential problems with graphics # handling/generation, causing 0x0 bitmaps etc. # * java.security for configuring the Java security framework with ECE. - ece_args="-Descenic.server=$ece_server\ - -Dsolr.solr.home=$solr_home\ - -Djava.awt.headless=true\ - -Djava.security.auth.login.config=$ece_security_configuration_dir/jaas.config\ - -Djava.security.policy=$ece_security_configuration_dir/java.policy" - - if [ $instance != "default" ]; then - ece_args=$ece_args" -Dcom.escenic.instance=$instance" + # * garbage collection log: this is paramount to keep an eye on + # when running in production. + ece_args=" + -Descenic.server=$ece_server + -Dcom.escenic.instance=$instance + -Djava.awt.headless=true + -Djava.security.auth.login.config=$ece_security_configuration_dir/jaas.config + -Djava.security.policy=$ece_security_configuration_dir/java.policy + -Dsolr.solr.home=$solr_home + -XX:+PrintGCDetails + -XX:+PrintGCTimeStamps + -Xloggc:${gc_log} + " + + if [ "$jvm_gc_settings" ]; then + ece_args=$ece_args" "$jvm_gc_settings fi - if [ $ece_server_hostname ]; then + jvm_rolling_gc=${jvm_rolling_gc-1} + if [ "$jvm_rolling_gc" == 1 ]; then + gc_rolling_args=" + -XX:+UseGCLogFileRotation + -XX:NumberOfGCLogFiles=${number_of_gc_log_files-5} + -XX:GCLogFileSize=${gc_log_file_size-5m} + " + ece_args=$ece_args" "$gc_rolling_args + fi + + if [ "$jvm_proxy_settings" ]; then + ece_args=$ece_args" "$jvm_proxy_settings + fi + + if [ "$jvm_connection_settings" ]; then + ece_args=$ece_args" "$jvm_connection_settings + fi + + if [ -n "$ece_environment" ]; then + ece_args=$ece_args" -Dcom.escenic.environment=${ece_environment}" + fi + + if [ "$ece_server_hostname" ]; then ece_args=$ece_args" -Djava.rmi.server.hostname=$ece_server_hostname" fi - if [ $apr_lib_dir ]; then + if [ "$apr_lib_dir" ]; then ece_args=$ece_args" -Djava.library.path=$apr_lib_dir" fi - if [ $min_heap_size ]; then + if [ "$min_heap_size" ]; then ece_args=$ece_args" -Xms$min_heap_size" fi - if [ $max_heap_size ]; then + if [ "$max_heap_size" ]; then ece_args=$ece_args" -Xmx$max_heap_size" fi - if [ $min_permgen_size ]; then + if [ "$min_permgen_size" ]; then ece_args=$ece_args" -XX:PermSize=$min_permgen_size" fi - if [ $max_permgen_size ]; then + if [ "$max_permgen_size" ]; then ece_args=$ece_args" -XX:MaxPermSize=$max_permgen_size" fi - - # settings specific to a production environment - if [ $is_production -eq 1 ]; then - ece_args=$ece_args" -server" - fi - - if [ $jvm_encoding ]; then + + if [ "$jvm_encoding" ]; then ece_args=$ece_args" -Dsun.jnu.encoding=$jvm_encoding" ece_args=$ece_args" -Dfile.encoding=$jvm_encoding" fi - if [ $enable_remote_debugging -eq 1 ]; then + if [ "$analysis_conf_dir" ]; then + ece_args=$ece_args" -Dcom.escenic.eae.config=${analysis_conf_dir}" + fi + + if [ "$enable_remote_debugging" -eq 1 ]; then ece_args=$ece_args" -Xdebug -Xnoagent -Djava.compiler=NONE \ -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=$remote_debugging_port" - fi + fi - if [ $enable_remote_monitoring -eq 1 ]; then + if [ "$enable_remote_monitoring" -eq 1 ]; then ece_args=$ece_args" -Dcom.sun.management.jmxremote" ece_args=$ece_args" -Dcom.sun.management.jmxremote.authenticate=false" ece_args=$ece_args" -Dcom.sun.management.jmxremote.ssl=false" - ece_args=$ece_args" -Dcom.sun.management.jmxremote.port=$remote_monitoring_port" + ece_args=$ece_args" -Dcom.sun.management.jmxremote.port=$remote_monitoring_port" fi - if [ $enable_heap_dump -eq 1 ]; then + if [ "$enable_heap_dump" -eq 1 ]; then ece_args=$ece_args" -XX:+HeapDumpOnOutOfMemoryError" ece_args=$ece_args" -XX:HeapDumpPath=$heap_dump_dir" fi - if [ $force_ipv4 -eq 1 ]; then + if [ "$force_ipv4" -eq 1 ]; then ece_args=$ece_args" -Djava.net.preferIPv4Stack=true" fi - + + if [[ -n "$escenic_admin_http_user" && -n "$escenic_admin_http_password" ]]; then + wget_appserver_auth=" + --http-user $escenic_admin_http_user + --http-password $escenic_admin_http_password + " + curl_appserver_auth="-u ${escenic_admin_http_user}:${escenic_admin_http_password}" + fi + + if [[ -n "$builder_http_user" && -n "$builder_http_password" ]]; then + wget_builder_auth=" + --http-user $builder_http_user + --http-password $builder_http_password + " + fi + # Resin needs some more arguments as its XML parser is not # compatible with ECE. - if [ $appserver = "resin" ]; then + if [[ "$appserver" == "resin" ]]; then ece_args="$ece_args\ -Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser -Djavax.xml.stream.XMLInputFactory=com.sun.xml.stream.ZephyrParserFactory @@ -358,8 +436,8 @@ function set_instance_settings() -Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl" debug "resin_home=$resin_home" - elif [ $appserver = "tomcat" ]; then - if [ -z $tomcat_base ]; then + elif [ "$appserver" == "tomcat" ]; then + if [ -z "$tomcat_base" ]; then tomcat_base=$tomcat_home else export CATALINA_BASE=$tomcat_base @@ -371,706 +449,369 @@ function set_instance_settings() export ECE_HOME=$ece_home export JAVA_OPTS="$JAVA_OPTS $ece_args" export JAVA_HOME=$java_home - + debug ECE_HOME=$ECE_HOME debug JAVA_HOME=$JAVA_HOME debug JAVA_OPTS=$JAVA_OPTS - } -function sanity_check() -{ - verify_that_directory_and_file_are_writeable $log_file - verify_that_directory_and_file_are_writeable $pid_file - - if [ -z "$command" ]; then - print "You must specificy a command, see 'ece help'." - exit 1 - fi -} - -function deploy() -{ - ear=$cache_dir/engine.ear - if [ ! -e $ear ]; then - print "$ear does not exist. Did you run '"`basename $0`" assemble'?" - exit 1 - fi - - # extract EAR to a temporary area - dir=$tmp_dir/`basename $0`-`date +%s` - (mkdir -p $dir && cd $dir && jar xf $ear) - exit_on_error "extracting $ear to $dir" - - print "Deploying $ear on $appserver ..." - - case $appserver in - tomcat) - # We do not want the Escenic jars to share the same - # classloader folder as tomcat does We thereby want - # clients to use a separate escenic classloader to avoid - # strange upgrade problems i.e wrong versions of certain - # libraries. - if [ -d $tomcat_base/escenic/lib ]; then - if [ `ls $tomcat_base/escenic/lib | grep .jar | wc -l` -gt 0 ]; then - rm $tomcat_base/escenic/lib/*.jar - exit_on_error "removing previous deployed libraries" - fi - cp $dir/lib/*.jar \ - $tomcat_base/escenic/lib \ - 1>>$log_file \ - 2>>$log_file - exit_on_error "deploying jar files to $tomcat_base/escenic/lib" - else - print "Could not find $tomcat_base/escenic/lib. Exiting." - print "Also make sure that you have defined this directory in" - print " $tomcat_base/conf/catalina.properties" - print "see sample configuration in the engine distribution" - print " contrib/appserver/tomcat/catalina-sample.properties" - exit 1 - fi - - exit_on_error "copying lib files to app server classpath" - - rm -rf $tomcat_base/work/* \ - 1>>$log_file \ - 2>>$log_file - - exit_on_error "removing work files from tomcat" - - for war in $dir/*.war ; do - if [ -d $tomcat_base/webapps/`basename $war .war` ] ; then - rm -rf $tomcat_base/webapps/`basename $war .war` \ - 1>>$log_file \ - 2>>$log_file - exit_on_error "removing already deployed escenic wars in $tomcat_base/webapps/" - fi - done - - # this scenario is likely when running many minimal - # instances of tomcat and some of these are not properly - # initialised. - if [ ! -d $tomcat_base/webapps ]; then - print $tomcat_base/webapps "doesn't exist, exiting." - exit 1 - fi - - if [ -n "$deploy_webapp_white_list" ]; then - deploy_this_war=0 - message="Deployment white list active, only deploying: " - message=$message"$deploy_webapp_white_list" - print $message - log $message +function sanity_check() { + if [ "$(whoami)" == "root" ]; then + local next_is_ok=0 + for subcommand in ${command}; do + # ignore options, i.e. things starting with a hyphen. + if [[ "${subcommand}" =~ ^-.* ]]; then + # Allow values to options, must use this since not + # everything is using getopt (yet) + next_is_ok=1 + continue fi - - for war in $dir/*.war ; do - name=`basename $war .war` - - deploy_this_war=1 - if [ -n "$deploy_webapp_white_list" ]; then - deploy_this_war=0 - - for el in $deploy_webapp_white_list; do - if [ $el = $name ]; then - debug "found $war in white list, will deploy it" - deploy_this_war=1 - fi - done - fi - - if [ $deploy_this_war -eq 0 ]; then - continue - fi - - (cd $tomcat_base/webapps && - mkdir $name && - cd $name && - jar xf $war \ - 1>>$log_file \ - 2>>$log_file) - exit_on_error "extracting $war to $tomcat_base/webapps/" - done - ;; - - resin) - if [ ! -d $resin_home/deploy ]; then - mkdir -p $resin_home/deploy \ - 1>>$log_file \ - 2>>$log_file - fi - cp $ear $resin_home/deploy \ - 1>>$log_file \ - 2>>$log_file - ;; - *) - print "Deployment is only implemented for Resin and Tomcat so far." - ;; - esac -} - - -function start_type() -{ - unset CLASSPATH - message="Starting the $instance instance of $type on $HOSTNAME ..." - print $message - log $message - - if [ $type = "rmi-hub" ]; then - ensure_that_required_fields_are_set $hub_required_fields - - if [ -r $rmi_hub_conf ]; then - export CLASSPATH=$rmi_hub_conf:$CLASSPATH - else - print $rmi_hub_conf "must point to a valid Nursery configuration" - print "for the rmi-hub, you may copy the one found in" - print "$ece_home/contrib/rmi-hub/config." - print "Exiting :-(" - exit 1 - fi - - for el in $rmi_hub_home/lib/*.jar; do - export CLASSPATH=$CLASSPATH:$el - done - - $java_home/bin/java \ - -Djava.rmi.server.hostname=${ece_server_hostname} \ - neo.nursery.GlobalBus /Initial \ - 1>>$log_file \ - 2>>$log_file & pid=$! - - echo $pid > $pid_file - exit 0 - elif [ $type = "search" ]; then - # TODO trim & tun the default parameters for the search - # instance. - ensure_that_required_fields_are_set $engine_required_fields - elif [ $type = "engine" ]; then - ensure_that_required_fields_are_set $engine_required_fields - elif [ $type = "analysis" ]; then - ensure_that_required_fields_are_set $analysis_required_fields - fi - - # indexer and engine are treated the same - case $appserver in - tomcat) - # Tomcat respects JAVA_OPTS set in configure(), so no need - # to set them here. - - if [ ! -x $tomcat_home/bin/catalina.sh ]; then - print "$tomcat_home/bin/catalina.sh was not executable" - print "unable to start tomcat" - exit 1 + if [ ${next_is_ok-0} -eq 1 ]; then + next_is_ok=0 + continue fi - # We call run here to get the log output to the stdout log - $tomcat_home/bin/catalina.sh run \ - 1>>$log_file \ - 2>>$log_file & pid=$! - ;; - oc4j) - export OC4J_JVM_ARGS=$ece_args - $oc4j_home/bin/oc4j -start\ - 1>>$log_file\ - 2>>$log_file & pid=$! - ;; - resin) - # Resin has stared insisting on a -J prefix of the -D - # prefixes :-) Tested with Resin 3.0.25 - resin_ece_args=`echo $ece_args | sed 's/-D/-J-D/g'` - - # works for Resin 3.0 - if [ -e $resin_home/bin/wrapper.pl ]; then - exec perl $resin_home/bin/wrapper.pl \ - -chdir \ - -name httpd \ - -class com.caucho.server.resin.Resin \ - $resin_ece_args ${1+"$@"} \ - 1>>$log_file \ - 2>>$log_file & pid=$! - else - # works for Resin 3.1 - $java_home/bin/java $ece_args \ - -jar $resin_home/lib/resin.jar \ - start \ - 1>>$log_file \ - 2>>$log_file & pid=$! - fi - ;; - jboss) - $jboss_home/bin/run.sh \ - -b 0.0.0.0 \ - -c $jboss_conf \ - 1>>$log_file \ - 2>>$log_file & pid=$! - ;; - *) - echo "" # extra line feed, because of the echo -n above - print "No appserver is defined in $ece_conf" - exit 1 - esac - - if [ $pid ]; then - verify_that_directory_and_file_are_writeable $pid_file - echo $pid > $pid_file + declare | grep -q -w root_allowed_${subcommand//-/_} || { + print "Sorry, you cannot be root when running " \ + "'${BASH_SOURCE[0]##*/} ${subcommand}'." \ + "root can only use /etc/init.d/ece" + exit 1 + } + done fi - - exit_on_error $message -} -function stop_type() -{ - message="Stopping the $instance instance of $type on $HOSTNAME ..." - - if [ -n "$type_pid" ]; then - log $message - print $message - - if [ -r $pid_file ]; then - if [ "$type_pid" != "`cat $pid_file`" ]; then - print "Is running, but was not started with `basename $0`" - print "system PID $ece_pid differs from the PID in $pid_file" - print "removing dangling PID file $pid_file. " - print "In future, be sure to use $0 to start " - print "and stop your $type" - kill $type_pid 1>>$log_file 2>>$log_file - rm $pid_file - return - fi - fi - - kill $type_pid \ - 1>>$log_file \ - 2>>$log_file - - if [ -e $pid_file ]; then - rm $pid_file \ - 1>>$log_file \ - 2>>$log_file - fi - else - print "The $instance instance of $type on $HOSTNAME is NOT running" + if [ "$(echo $command)" == "list-instances" ]; then + return fi - - exit_on_error $message -} -function kill_type() -{ - if [ -n "$type_pid" ]; then - message="Using force to stop the $instance instance of $type on $HOSTNAME ..." - log $message - print $message - kill -9 $type_pid - if [ -w $pid_file ]; then - rm $pid_file - fi - else - print "No $instance instance of $type on $HOSTNAME to be killed" - fi - - exit_on_error "kill_type" -} + verify_that_directory_and_file_are_writeable $log + verify_that_directory_and_file_are_writeable $pid_file -function restart_type() -{ - stop_type - - # I've tested this with various values, 8 seconds seems to - # guarantee that the previous JVM process has stopped - # properly before starting the new one. - sleep 8 - - # sometimes the JVM refuses to shot down when doing a graceful - # kill, therefore, if it's still running after the sleep above, we - # use brute force to kill it. - set_type_pid - if [ -n "$type_pid" ]; then - message="The $instance instance of $type failed to stop gracefully" - debug $message - log $message >> $log_file - print $message - kill_type - fi - - start_type -} - -function assemble() -{ - if [ ! $type = "engine" ]; then - print "You cannot assemble $type" + if [ -z "$command" ]; then + print "You must specificy a command, see 'ece help'." exit 1 fi - message="Assembling your EAR file ..." - print $message - log $message >> $log_file - - cd $assemblytool_home && \ - ant -q ear -DskipRedundancyCheck=true \ - 1>>$log_file \ - 2>>$log_file - exit_on_error "$message" - - mkdir -p $ear_cache_dir/ - exit_on_error "creating $ear_cache_dir" - - cp $assemblytool_home/dist/engine.ear $cache_dir - exit_on_error "copying ear to $ear_cache_dir" - - debug $assemblytool_home/dist/engine.ear "is now ready" -} - -function tail_messages_log() -{ - for el in $log_file_list; do - if [ -r $el ]; then - print "tailing $el" - tail -f $el - break - fi + local instance_list=$(get_instance_list) + local instance_exists=0 + for el in $instance_list; do + if [ $el == $instance ]; then + instance_exists=1 + break + fi done -} -function tail_out_log() -{ - tail_list=$log_file - - # if needs be, we can add more system out logs here. For now, - # we're sticking with the default one. - - print "Tailing the system out log $tail_list" - tail -f $tail_list -} + if [ $instance_exists -eq 0 ]; then + print "Instance '$instance' doesn't exist on $HOSTNAME" + exit 1 + fi -function tail_app_log() -{ - if [ $type = "rmi-hub" ]; then - print "There is no application server log for $type" + # verifies that java_home exists and has the java executable + if [ ! -d $java_home ]; then + print "java_home $java_home doesn't exist" + print "check your setting in one of: ${ece_conf_files_read[@]}" exit 1 - fi - - if [ $appserver = "tomcat" ]; then - log_file=$tomcat_base/logs/localhost.`date +%F`.log - elif [ $appserver = "resin" -a -e $resin_home/log/jvm-default.log ]; then - log_file=$resin_home/log/jvm-default.log - else - print "I don't know where the logs for $appserver are." - print "Ask support@escenic.com to add support for $appserver in " - print "tail_app_log()" + elif [ ! -x $java_home/bin/java ]; then + print "$java_home/bin/java isn't executable for $ece_user" exit 1 fi - - print "Tailing the application server log $log_file" - tail -f $log_file } -function make_thread_dump() -{ - - if [ -n "$type_pid" ]; then - print "Thread dump (PID" $type_pid") written to system out log." - print "Type 'ece -t $type -i default outlog' to see it or view" - print $log_file "directly." +# Returns the file (can be a directory) passed to the function only +# if it's the actual file/directory and not a link to it. If the +# passed file is a link, the link target is returned instead. +# +# $1 - the file (which could be a link) +function get_actual_file() { + if [ -h ${1} ]; then + dir=$(dirname $1) + real_file=$(ls -l ${1} | awk '{print $11}') + + # Because of the test if the file we want to returns is + # absolute, we go to the root before testing. We want to + # preserve the cwd, therefore, we're doing the "cd /" in a + # subshell. + real_file=$( + cd / + if [ ! -e ${real_file} ]; then + real_file=${dir}/${real_file} + fi - if [ -x $java_home/bin/jstack ]; then - jstack -l $type_pid >> $log_file - else - kill -QUIT $type_pid >> $log_file - fi + # this is the return value from the sub process + echo ${real_file} + ) else - get_status + real_file=${1} fi + + echo ${real_file} } -function set_type_command_and_instance() -{ - next_is_type=0 - next_is_instance=0 - next_is_publication=0 - next_is_resource=0 - +function set_type_command_and_instance() { + local next_is_type=0 + local next_is_instance=0 + local next_is_publication=0 + local next_is_resource=0 + local next_is_http_user=0 + local next_is_http_password=0 + local next_is_file=0 + for el in $@; do - if [ $el = "-v" -o $el = "--verbose" ]; then - verbose=1 + if [ "$el" == "-v" -o "$el" == "--verbose" ]; then + debug=1 + continue + fi + + if [ "$el" == "--full" ]; then + everything_but_the_kitchen_sink=1 continue fi - if [ $el = "--help" ]; then + + if [ "$el" == "-f" -o $el == "--file" -o $el == "--uri" ]; then + next_is_file=1 + continue + fi + + if [ "$el" == "-q" -o $el == "--quiet" ]; then + quiet=1 + continue + fi + + if [ "$el" == "--help" ]; then command=help continue fi - if [ $next_is_type -eq 1 ]; then + if [ "$next_is_file" -eq 1 ]; then + file=$el + next_is_file=0 + continue + fi + + if [ "$next_is_type" -eq 1 ]; then type=$el next_is_type=0 continue fi - - if [ $next_is_instance -eq 1 ]; then + + if [ "$next_is_http_user" -eq 1 ]; then + http_user=$el + next_is_http_user=0 + continue + fi + + if [ "$next_is_http_password" -eq 1 ]; then + http_password=$el + next_is_http_password=0 + continue + fi + + if [ "$next_is_instance" -eq 1 ]; then instance=$el next_is_instance=0 continue fi - - if [ $next_is_publication -eq 1 ]; then + + if [ "$next_is_publication" -eq 1 ]; then publication=$el next_is_publication=0 continue fi - - if [ $next_is_resource -eq 1 ]; then + + if [ "$next_is_resource" -eq 1 ]; then resource=$el next_is_resource=0 continue fi - - if [ $el = "-t" -o $el = "--type" ]; then + + if [ "$el" == "-t" -o "$el" == "--type" ]; then next_is_type=1 continue else next_is_type=0 fi - if [ $el = "-i" -o $el = "--instance" ]; then + if [ "$el" == "-i" -o "$el" == "--instance" ]; then next_is_instance=1 continue else next_is_instance=0 fi - if [ $el = "-p" -o $el = "--publication" ]; then + if [ "$el" == "-p" -o "$el" == "--publication" ]; then next_is_publication=1 continue else next_is_publication=0 fi - - if [ $el = "-r" -o $el = "--publication-resource" ]; then + + if [ "$el" == "-r" -o "$el" == "--publication-resource" ]; then next_is_resource=1 continue else next_is_resource=0 fi - + + if [ "$el" == "-u" -o "$el" == "--user" ]; then + next_is_http_user=1 + continue + else + next_is_http_user=0 + fi + + if [ "$el" == "-w" -o "$el" == "--password" ]; then + next_is_http_password=1 + continue + else + next_is_http_password=0 + fi + + if [ "$el" == "--exclude-binaries" ]; then + backup_exclude_binaries=1 + continue + elif [ "$el" == "--exclude-solr" ]; then + backup_exclude_solr=1 + continue + elif [ "$el" == "--exclude-conf" ]; then + backup_exclude_conf=1 + continue + elif [ "$el" == "--exclude-multimedia" ]; then + backup_exclude_multimedia=1 + continue + elif [ "$el" == "--exclude-db" ]; then + backup_exclude_db=1 + continue + elif [ "$el" == "--exclude-init" ]; then + backup_exclude_init=1 + continue + elif [ "$el" == "--exclude-state" ]; then + backup_exclude_state=1 + continue + elif [ "$el" == "--version" -o "$el" == "-V" ]; then + echo "Version:" $ece_scripts_version + exit 0 + fi + # the only thing left at this point, is the command - command="$command $el" + if [ -n "${command}" ]; then + command="$command $el" + else + command=${el} + fi done -} - -function clean_up() -{ - if [ $type = "engine" ]; then - print "Cleaning up generated files in $assemblytool_home ..." - cd $assemblytool_home - ant clean \ - 1>>$log_file \ - 2>>$log_file - fi - tmp_dir_prefix="`basename $0`-" - if [ `ls $tmp_dir | grep $tmp_dir_prefix | wc -l` -gt 0 ]; then - print "Cleaning up generated files in $tmp_dir ..." - rm -rf $tmp_dir/$tmp_dir_prefix-[0-9]* \ - 1>>$log_file \ - 2>>$log_file - fi - - if [ -e /var/cache/escenic/ -a \ - `ls /var/cache/escenic | grep ece- | wc -l` -gt 0 ]; then - print "Cleaning up "`basename $0`" files in /var/cache/escenic/ ..." - rm -rf /var/cache/escenic/* \ - 1>>$log_file \ - 2>>$log_file + # If the user didn't specify which instance to use and if there's + # only one available instance, use that instead of asking the user + # to provide it in sanity_check. + if [ "$instance" == "engine1" ]; then + debug "Trying to determine instance name" + local instance_list=$(get_instance_list) + + if [ $(echo $instance_list | grep ' ' | wc -l) -eq 0 ]; then + instance=$instance_list + if [ -z "$instance" ] ; then + instance=engine1 + fi + debug "setting instance=$instance as there's only one" + fi fi - } -function set_id() -{ - if [ $instance = "default" ]; then - id="["`basename $0`"#${type}]" - else - id="["`basename $0`"#${type}-${instance}]" - fi - +function set_id() { + id="["$(basename $0)"#${type}-${instance}]" + debug type is $type \ and command is $command \ and instance is $instance } -function get_status() -{ +function get_status() { if [ -z "$type_pid" ]; then - print "DOWN" + echo "DOWN" exit 0 elif [ -r $pid_file ]; then - + if [ "$type_pid" != `cat $pid_file` ]; then - print "Is running, but was not started with `basename $0`" - print "system PID $ece_id differs from the PID in $pid_file" + echo "Is running, but was not started with $(basename $0)" + echo "system PID $ece_id differs from the PID in $pid_file" exit 1 fi else - print $pid_file "did not exist, " - print "the ${instance} instance of ${type} is running with PID $type_pid" - print "but hasn't been started properly with `basename $0`" + echo $pid_file "did not exist, " + echo "the ${instance} instance of ${type} is running with PID $type_pid" + echo "but hasn't been started properly with $(basename $0)" exit 1 fi - now=`date +%s` - started=`stat -c %Y $pid_file` - seconds=$(( now - started )) - days=$(( seconds / ( 60 * 60 * 24 ) )) - seconds_left=$(( seconds - ( $days * 60 * 60 * 24 ) )) - hours=$(( seconds_left / ( 60 * 60 ) )) - seconds_left=$(( seconds_left - ( $hours * 60 * 60 ) )) - minutes=$(( seconds_left / 60 )) - seconds_left=$(( seconds_left - $minutes * 60 )) - - print "UP" ${days}d ${hours}h ${minutes}m ${seconds_left}s + local now=`date +%s` + local started=`stat -c %Y $pid_file` + local seconds=$(( now - started )) + local days=$(( seconds / ( 60 * 60 * 24 ) )) + local seconds_left=$(( seconds - ( $days * 60 * 60 * 24 ) )) + local hours=$(( seconds_left / ( 60 * 60 ) )) + local seconds_left=$(( seconds_left - ( $hours * 60 * 60 ) )) + local minutes=$(( seconds_left / 60 )) + local seconds_left=$(( seconds_left - $minutes * 60 )) + + echo "UP" ${days}d ${hours}h ${minutes}m ${seconds_left}s } -function list_versions() -{ - if [ -z "$type_pid" ]; then - print "$instance instance of $type on $HOSTNAME is NOT running" - exit 1 +function read_rc_file_if_present() { + local rc_file=$HOME/.ecerc + if [ -r $rc_file ]; then + source $rc_file fi - - version_manager=escenic-admin/browser/Global/neo/io/managers/VersionManager - url=http://$host:$port/$version_manager - - print "Installed on the ${type} running on ${host}:${port} is:" - wget -O - $url 2>/dev/null | \ - grep "\[\[" | \ - sed 's/\[//g' | \ - sed 's/\]//g' | \ - sed 's/Name\=io/Name\=content-engine/g' | \ - sed 's/Name\=//g' | \ - sed 's/\;\ Version\=/\ /g' | \ - awk -F',' '{for (i = 1; i <= NF; i++) print " * " $i;}' | \ - # This is a hack since that for some reason, cannot get - # sub(/^[ \t]+/, "") to work inside the loop for $i, it seems - # to always operate on the incoming line. - sed 's/\*\ \ \ /\*/g' | \ - sort } -function update_publication_resources() -{ - url=http://${host}:${port}/escenic-admin/publication-resources +function get_escenic_admin_url() { + set_type_port + local url=http://${host}:${port}/escenic-admin + echo ${url} +} - if [ ! -r $resource ]; then - print $resource "doesn't exist. I will exit :-(" - exit 1 - fi +export root_allowed_list_publications=1 +function show_all_publications() { + set_type_port + print "These are all the publications on ${instance}:" + get_publication_list ${port} +} - if [ -x $publication ]; then - print "You must specify which publication to update (-p )" - exit 1 - fi - - case "$(basename $resource)" in - content-type) - url=${url}/${publication}/escenic/content-type - ;; - feature) - url=${url}/${publication}/escenic/feature - ;; - layout) - url=${url}/${publication}/escenic/layout - ;; - layout-group) - url=${url}/${publication}/escenic/layout-group - ;; - image-version) - url=${url}/${publication}/escenic/image-version - ;; - menu) - url=${url}/${publication}/escenic/plugin/menu - ;; - - *) - print "Invalid resource: $(basename $resource) :-(" - exit 1 - - esac - - print "Updating the $(basename $resource) resource for the $publication" \ - "publication" - - debug POSTing $resource to $url - wget -O - \ - --post-file ${resource} \ - $url \ - 1>>$log_file 2>>$log_file - +function show_all_deployments() { + print "These are all the deployments on ${instance}:" + cat $(get_deployment_log) +} + +## $1 :: command +## $..n :: the rest of the arguments passed to /usr/bin/ece +function lookup_and_execute_dynamic_command_if_found() { + local command=$1 + local requested_cmd=cmd_${command//-/_} + local actual_cmd= + actual_cmd=$(declare -F | grep -w "${requested_cmd}" | cut -d' ' -f3) + if [ -n "${actual_cmd}" ]; then + shift 1 + "${actual_cmd}" "$@" + exit 0 + fi } -set_type_command_and_instance $@ +init +read_rc_file_if_present +set_type_command_and_instance "$@" set_id set_common_settings set_type_settings set_instance_settings sanity_check -function print_help() -{ - echo "Usage: $0 [-t ] [-i ] " - echo "" - echo "DESCRIPTION" - echo " -t --type " - echo " The following types are available:" - echo " engine - The Escenic Content Engine, this is the default" - echo " and is the assumed type if none is specified." - echo " search - A standalone search indexer and solr instance" - echo " rmi-hub - The RMI hub responsible for the internal " - echo " communication between the ECE instances." - echo " analysis - The Escenic Analysis Engine also knows as 'Stats'" - echo "" - echo " -i --instance " - echo " The type instance, such as editor1, web1 or search1" - echo "" - echo " -p --publication " - echo " Needed only for updating publication resources" - echo "" - echo " -r --resource " - echo " Used for updating publication resources." - echo " Must be one of: content-type, feature, layout, layout-group" - echo " image-version, menu" - echo "" - echo " -v --verbose" - echo " Prints out debug statements, useful for debugging." - echo "" - echo "The following commands are available:" - echo " applog the type's app server log" - echo " assemble runs the Assembly Tool *)" - echo " clean removes temporary files created by $0 *)" - echo " deploy deploys the assembled EAR *)" - echo " help prints this help screen" - echo " kill uses force to stop the type" - echo " log the type's Log4J log" - echo " outlog the $id script log (system out log)" - echo " restart restarts the type" - echo " start starts the type" - echo " status checks if the type is running" - echo " stop stops the type" - echo " threaddump write a thread dump to standard out (system out log)" - echo " update update publication resources" - echo " versions lists the ECE component versions" - echo "" - echo "*) only applicable if type is 'engine'" -} - for el in $command; do case "$el" in start) start_type ;; status) - get_status + print "$(get_status)" ;; stop) stop_type @@ -1078,9 +819,21 @@ for el in $command; do restart) restart_type ;; + info) + get_info_for_type + ;; log) tail_messages_log ;; + list-logs) + show_all_log_paths + ;; + list-publications) + show_all_publications + ;; + list-deployments) + show_all_deployments + ;; outlog) tail_out_log ;; @@ -1096,29 +849,50 @@ for el in $command; do deploy) deploy ;; + repackage) + repackage "${file}" + ;; assemble) assemble ;; + package) + create_packages + ;; clean) clean_up ;; versions) list_versions ;; + list-instances) + print $(get_instance_list) + ;; + remove-old-log-files) + remove_old_log_files + ;; update) update_publication_resources ;; + edit) + update_publication_resources "edit_first" + ;; + create) + update_publication_resources "create_first" + ;; + backup) + backup_type + ;; + flush) + flush_caches + ;; + top) + run_ece_top + ;; help) - if [ -x $(which less) ]; then - print_help | less - elif [ -x $(which more) ]; then - print_help | more - else - print_help - fi - + print_help ;; *) + lookup_and_execute_dynamic_command_if_found "${el}" "$@" print "Invalid command: '$el' :-(" print "Try 'ece help' to get a list of all commands available." exit 1 @@ -1126,4 +900,3 @@ for el in $command; do done exit 0 - diff --git a/usr/bin/ece-import b/usr/bin/ece-import new file mode 100755 index 00000000..e32e930b --- /dev/null +++ b/usr/bin/ece-import @@ -0,0 +1,418 @@ +#! /usr/bin/env bash + +## Runs one VOSA import job one time. The output is XML suitable for +## the standard Escenic Syndication XML import job. +## +## The script can also create an import job from an import job archive +## +## See /usr/share/doc/vizrt/vosa-handbook/import-jobs.org for more +## details on the structures this command operates on. + +function bootstrap_thyself() { + # first, try to be nice, then check the standard location + local dir=$(dirname $0)/../share/escenic/ece-scripts + if [ ! -d $dir ]; then + dir=/usr/share/escenic/ece-scripts + fi + + local common_libraries=" + common-bashing.sh + common-ece.sh + common-io.sh + common-os.sh + " + + for el in $common_libraries; do + source $dir/$el 2>/dev/null || { + echo "$(basename $0): Could not load the library $el," \ + "and I can't live without it :-(" | fmt + exit 1 + } + done + + for el in $dir/$(basename $0).d/*.sh; do + source $el 2>/dev/null || { + echo "$(basename $0): Could not load the library $el," \ + "and I can't live without it :-(" | fmt + exit 1 + } + done +} + +bootstrap_thyself + +# internal variables +escenic_group=escenic +escenic_spool_base_dir=/var/spool/escenic/import +escenic_user=escenic +job_name="" +log_base_dir=/var/log/escenic +ece_scripts_version="straight-from-github" + +log=$log_base_dir/$(basename $0).log +nursery_base_dir=/etc/escenic/engine/common +publication_name="" +raw_spool_base_dir=/var/spool/escenic/raw +raw_state_base_dir=/var/lib/escenic/raw +raw_transformation_base_dir=/var/cache/escenic/import +raw_transformed_base_dir=/var/backups/escenic/import +transformers_base_dir=/usr/share/escenic/import +arg_regex_of_file="^(.*)$" +arg_write_url="" + +# available commands/operations for ece-import +COMMAND_IMPORT=1 +COMMAND_CREATE_IMPORT_CONFIGURATION=2 +COMMAND_DOWNLOAD_RAW_DATA=3 +command=$COMMAND_IMPORT + +function get_user_input() { + local next_is_name=0 + local next_is_publication=0 + local next_is_import_archive=0 + local next_is_nursery_base_dir=0 + local next_is_escenic_user=0 + local next_is_escenic_group=0 + local next_is_user=0 + local next_is_password=0 + local next_is_uri=0 + local next_is_http_proxy=0 + local next_is_regex_of_file=0 + local next_is_write_url=0 + + for el in $@; do + if [[ "$el" == "-n" || "$el" == "--name" ]]; then + next_is_name=1 + elif [[ "$el" == "-p" || "$el" == "--publication" ]]; then + next_is_publication=1 + elif [[ "$el" == "-f" || "$el" == "--import-archive" ]]; then + next_is_import_archive=1 + elif [[ "$el" == "--escenic-user" ]]; then + next_is_escenic_user=1 + elif [[ "$el" == "--escenic-group" ]]; then + next_is_escenic_group=1 + elif [[ "$el" == "--nursery-base-dir" ]]; then + next_is_nursery_base_dir=1 + elif [[ "$el" == "--user" ]]; then + next_is_user=1 + elif [[ "$el" == "--password" ]]; then + next_is_password=1 + elif [[ "$el" == "--uri" ]]; then + next_is_uri=1 + elif [[ "$el" == "--regex-of-file" ]]; then + next_is_regex=1 + elif [[ "$el" == "--write_url" ]]; then + next_is_write_url=1 + elif [[ "$el" == "--directories-only" ]]; then + directories_only=1 + elif [[ "$el" == "--http-proxy" ]]; then + next_is_http_proxy=1 + elif [ "$el" == "--version" -o "$el" == "-V" ]; then + echo "Version:" $ece_scripts_version + exit 0 + elif [ $next_is_name -eq 1 ]; then + job_name=$el + next_is_name=0 + elif [ $next_is_publication -eq 1 ]; then + publication_name=$el + next_is_publication=0 + elif [ $next_is_import_archive -eq 1 ]; then + import_archive=$el + next_is_import_archive=0 + elif [ $next_is_nursery_base_dir -eq 1 ]; then + nursery_base_dir=$el + next_is_nursery_base_dir=0 + elif [ $next_is_escenic_user -eq 1 ]; then + escenic_user=$el + next_is_escenic_user=0 + elif [ $next_is_escenic_group -eq 1 ]; then + escenic_group=$el + next_is_escenic_group=0 + elif [ $next_is_user -eq 1 ]; then + user=$el + next_is_user=0 + elif [ $next_is_password -eq 1 ]; then + password=$el + next_is_password=0 + elif [ $next_is_uri -eq 1 ]; then + uri=$el + next_is_uri=0 + elif [ $next_is_regex_of_file -eq 1 ]; then + arg_regex_of_file=$el + next_is_regex_of_file=0 + elif [ $next_is_write_url -eq 1 ]; then + arg_write_url=$el + next_is_write_url=0 + elif [ $next_is_http_proxy -eq 1 ]; then + the_http_proxy="$el" + next_is_http_proxy=0 + else + if [[ "$el" == "create" ]]; then + command=$COMMAND_CREATE_IMPORT_CONFIGURATION + elif [[ "$el" == "download-import-data" ]]; then + command=$COMMAND_DOWNLOAD_RAW_DATA + fi + fi + done + + local errors=0 + if [ -z "$job_name" -a -z "${import_archive}" ]; then + print_and_log "You must specify which import job to run" + print_and_log "E.g.: $(basename $0) --name video" + errors=1 + fi + if [ -z "$publication_name" -a -z "${import_archive}" ]; then + print_and_log "You must specify the publication name" + print_and_log "E.g.: $(basename $0) --publication mypub" + errors=1 + fi + + if [ -n "${import_archive}" -a ! -r "${import_archive}" ]; then + print_and_log "You have specified an import job archive file" \ + "but it doesn't exist :-(" + errors=1 + fi + + if [ $command -eq $COMMAND_DOWNLOAD_RAW_DATA ]; then + if [ -z "$user" ]; then + print_and_log "You must specify the the user" + print_and_log "E.g.: $(basename $0) --user lisa" + errors=1 + fi + if [ -z "$password" ]; then + print_and_log "You must specify the the user" + print_and_log "E.g.: $(basename $0) --password foo" + errors=1 + fi + if [ -z "$uri" ]; then + print_and_log "You must specify the the URI" + print_and_log "E.g.: $(basename $0) --uri http://feeds.com/myfeed" + errors=1 + fi + fi + + if [ $errors -eq 1 ]; then + remove_pid_and_exit_in_error + fi +} + +## $1 :: the transformer (file name, relative or absoulte) +function is_transformer_supported() { + if [ -z $1 ]; then + return + fi + + local supported_transformer_list="pl py sh xsl" + for el in $supported_transformer_list; do + if [[ "$1" == *"${el}" ]]; then + echo 1 + return + fi + done + + echo 0 +} + +## $1 :: file +function perform_transformations() { + for el in $transformers_base_dir/$publication_name/$job_name/transformers/[0-9]*; do + if [ $(is_transformer_supported $el) -eq 0 ]; then + log "$(yellow WARNING) The transformer $el isn't supported by $(basename $0)" + continue + fi + + log "Applying transformation $(basename $el) to $1" + + if [[ "$el" == *".sh" ]]; then + bash $el $1 >> $log 2>> $log + + if [ $? -gt 0 ]; then + handle_transformation_error $el $1 + return + fi + elif [[ "$el" == *".xsl" ]]; then + xsltproc --output ${1}.tmp ${el} ${1} >> $log 2>> $log + + if [ $? -gt 0 ]; then + handle_transformation_error $el $1 + return + else + run mv ${1}.tmp ${1} + fi + elif [[ "$el" == *".pl" ]]; then + perl $el $1 >> $log 2>> $log + if [ $? -gt 0 ]; then + handle_transformation_error $el $1 + return + fi + elif [[ "$el" == *".py" ]]; then + python $el $1 >> $log 2>> $log + if [ $? -gt 0 ]; then + handle_transformation_error $el $1 + return + fi + fi + transformation_count=$(( transformation_count + 1 )) + done +} + +## Will log the transformer error and move it to the error archive. +## +## $1 :: transformer +## $2 :: the raw/input file +function handle_transformation_error() { + local dir=$raw_transformed_base_dir/$publication_name/$job_name/failed + log "$(red FAILED) The transformation $1 on file $2" \ + "moving $2 to $dir and skipping to the next XML file" + run mv $2 $dir +} + +import_error_count=0 + +## $1 :: the directory to check for multimedia files. +## $2 :: the directory to move any of these multimedia files to +function move_any_multimedia_files_if_present() { + if [ ! -d $1 -o ! -d $2 ]; then + return + fi + + local multimedia_file_count=$( + ls $1 | egrep -i ".(png|gif|jpg|jpeg|pdf)$" | wc -l + ) + if [ $multimedia_file_count -gt 0 ]; then + log "Moving ${multimedia_file_count} multimedia files from $1 to $2" + mv $1/*.{png,gif,jpg,jpeg,pdf} $2 >> $log 2>/dev/null + fi +} + +function import_raw_files() { + raw_file_count=0 + for f in $(find $raw_spool_base_dir/$publication_name/$job_name -type f); do + raw_file_count=$(( raw_file_count + 1 )) + transformation_count=0 + print_and_log "Importing raw XML #${raw_file_count}: $(basename $f) ..." + local file=$raw_transformation_base_dir/$publication_name/$job_name/$(basename $f) + run cp $f $file + perform_transformations $file + + log "Applied $transformation_count transformations to $file" + if [ $(is_escenic_xml_ok $file) -eq 1 ]; then + local dir=$escenic_spool_base_dir/$publication_name/$job_name/new + log "Transformed XML is OK, moving transformed file to" $dir + move_any_multimedia_files_if_present $(dirname $file) $dir + run mv $file $dir + dir=$raw_transformed_base_dir/$publication_name/$job_name/succeeded + log "Transformed XML is OK, moving original raw XML to" $dir \ + "and gzip-ing it." + run mv $f $dir + run gzip --force $dir/$(basename $f) + else + local dir=$raw_transformed_base_dir/$publication_name/$job_name/failed + log $(red ERROR) "Transformed XML #${raw_file_count}," \ + $file "isn't valid Escenic Syndication XML, so moving it to" $dir + run mv $f $dir + import_error_count=$(( import_error_count + 1 )) + fi + done +} + +function verify_import_job_configuration() { + verify_writable_dir_list \ + $raw_spool_base_dir/$publication_name/$job_name \ + $raw_state_base_dir/$publication_name/$job_name \ + $raw_transformation_base_dir/$publication_name/$job_name \ + $raw_transformed_base_dir/$publication_name/$job_name \ + $escenic_spool_base_dir/$publication_name/$job_name/new \ + $escenic_spool_base_dir/$publication_name/$job_name/archive \ + $escenic_spool_base_dir/$publication_name/$job_name/error + verify_readable_dir_list $transformers_base_dir/$publication_name/$job_name + + local dir=$transformers_base_dir/$publication_name/$job_name/transformers + local tranformation_count=$( + ls $dir | \ + grep ^[0-9] | \ + egrep ".sh$|.pl$|.py$|.xsl$" | \ + wc -l + ) + + if [ $command -eq $COMMAND_IMPORT -a $tranformation_count -lt 1 ]; then + print_and_log "$(yellow WARNING) No transformers found in" \ + "$dir/, I'm assuming the incoming" \ + "data is already tranformed into Escenic Syndication XML" + fi + + print_and_log "Running import" $job_name \ + "for publication" $publication_name +} + +function print_report() { + if [ $command -eq $COMMAND_IMPORT ]; then + print_and_log "Number of raw XML files processed:" $raw_file_count + print_and_log "Number of raw XML successes:" \ + $(green $(( raw_file_count - import_error_count ))) + print_and_log "Number of raw XML errors:" $(red $import_error_count) + fi +} + +assert_commands_available xsltproc xmllint xml_grep +get_user_input $@ + +function run_import() { + pid_file=${pid_file/%.pid/-run-import.pid} + lock_file=${lock_file/%.lock/-run-import.lock} + common_pre_run + + verify_import_job_configuration + import_raw_files + common_post_run +} + +function run_create_import_configuration() { + pid_file=${pid_file/%.pid/-create.pid} + lock_file=${lock_file/%.lock/-create.lock} + common_pre_run + + if [ -z $import_archive ]; then + if [ ${directories_only-0} -eq 0 ]; then + create_import_configuration $publication_name $job_name + fi + create_import_directories $publication_name $job_name + else + apply_import_archive + fi + + common_post_run +} + +function run_download_raw_data() { + pid_file=${pid_file/%.pid/-download.pid} + lock_file=${lock_file/%.lock/-download.lock} + + common_pre_run + verify_import_job_configuration + download_latest_files + common_post_run +} + +function common_pre_run() { + print_and_log "Started @ $(date), I'm logging to $log" + create_pid + create_lock_or_fail +} + +function common_post_run() { + print_manual_steps + print_report + + print_and_log "Finished @ $(date), enjoy thyself!" + remove_pid + remove_lock +} + +if [[ "$command" == $COMMAND_IMPORT ]]; then + run_import +elif [[ "$command" == $COMMAND_CREATE_IMPORT_CONFIGURATION ]]; then + run_create_import_configuration +elif [[ "$command" == $COMMAND_DOWNLOAD_RAW_DATA ]]; then + run_download_raw_data +fi diff --git a/usr/bin/generate-changelog b/usr/bin/generate-changelog new file mode 100755 index 00000000..99fc980e --- /dev/null +++ b/usr/bin/generate-changelog @@ -0,0 +1,207 @@ +#! /usr/bin/env bash + +log=$HOME/.$(basename $0).log +archive_base_dir=$HOME/.$(basename $0) + +function bootstrap_thyself() { + # first, try to be nice, then check the standard location + local dir=$(dirname $0)/../share/escenic/ece-scripts + if [ ! -d $dir ]; then + dir=/usr/share/escenic/ece-scripts + fi + + local common_libraries=" + common-bashing.sh + common-io.sh + " + + for el in $common_libraries; do + source $dir/$el 2>/dev/null || { + echo "$(basename $0): Could not load the library $el," \ + "and I can't live without it :-(" | fmt + exit 1 + } + done +} + +function read_user_settings() { + local file=$HOME/.$(basename $0).conf + if [ -r $file ]; then + run source $file + else + log $(blue INFO) $file "doesn't exist, will default to Atlassian on Demand" + fi +} + +function get_header_from_jira() { + local body=$(curl -u ${user}:${password} -s ${jira_base_url}/browse/${1}) + echo "$body" | grep '' | sed -e 's/<title>//g' -e 's/<\/title>//g' +} + +function get_to_revision() { + echo ${to-COMMITTED} +} + +function get_from_revision() { + echo ${from-PREV} +} + +function get_user_input() { + local next_is_from=0 + local next_is_to=0 + local next_is_project=0 + local next_is_user=0 + local next_is_password=0 + + for el in "$@"; do + if [[ "$el" == "-s" || "$el" == "--from" ]]; then + next_is_from=1 + elif [[ "$el" == "-u" || "$el" == "--user" ]]; then + next_is_user=1 + elif [[ "$el" == "-p" || "$el" == "--password" ]]; then + next_is_password=1 + elif [[ "$el" == "-t" || "$el" == "--to" ]]; then + next_is_to=1 + elif [[ "$el" == "-p" || "$el" == "--project" ]]; then + next_is_project=1 + elif [[ "$el" == "-f" || "$el" == "--full" ]]; then + full_listing=1 + elif [ ${next_is_from-0} -eq 1 ]; then + from=$el + next_is_from=0 + elif [ ${next_is_user-0} -eq 1 ]; then + user=$el + next_is_user=0 + elif [ ${next_is_password-0} -eq 1 ]; then + password=$el + next_is_password=0 + elif [ ${next_is_to-0} -eq 1 ]; then + to=$el + next_is_to=0 + elif [ ${next_is_project-0} -eq 1 ]; then + project_code=$el + next_is_project=0 + fi + done +} + +function should_regenerate() { + if [[ $(is_number $(get_from_revision)) -eq 1 && \ + $(is_number $(get_to_revision)) ]]; then + echo 0 + return + fi + + echo 1 +} + +function get_commit_information_from_vcs() { + the_diff=$(get_archive_dir)/from-$(get_from_revision)-to-$(get_to_revision).diff + + if [[ ! -e $the_diff || $(should_regenerate) -eq 1 ]]; then + svn diff -r $(get_from_revision):$(get_to_revision) > $the_diff + exit_on_error "svn diff -r $(get_from_revision):$(get_to_revision)" + fi + + if [ ${full_listing-0} -eq 1 ]; then + cat $the_diff + else + echo "Full diff of all" $(egrep '^(\+|\-)' $the_diff | wc -l) \ + "changes:" $the_diff | fmt + fi +} + +## $@ :: svn revision number or tag name +function get_date_from_svn_log() { + echo $(svn log -r "$@" | sed -n '2p' | cut -d'|' -f3) +} + +function get_svn_location() { + svn info | grep URL | cut -d':' -f2- +} + +function get_info_from_jira() { + the_report=$(get_archive_dir)/from-$(get_from_revision)-to-$(get_to_revision).report + + if [[ ! -e $the_report || $(should_regenerate) -eq 1 ]]; then + echo "Changes in $(get_svn_location)" > $the_report + echo "From: revision $(get_from_revision) @" \ + $(get_date_from_svn_log $(get_from_revision)) >> $the_report + echo "To : revision $(get_to_revision) @" \ + $(get_date_from_svn_log $(get_to_revision)) >> $the_report + + local commit_log=$(svn log -r $(get_from_revision):$(get_to_revision)) + echo "$commit_log" | \ + grep ${project_code}-[0-9]* | \ + sed "s#.*\(${project_code}-[0-9]*\).*#\1#g" | \ + sort | \ + uniq | while read f; do + echo " *" $(get_header_from_jira $f) | fmt >> $the_report + echo " URL: ${jira_base_url}/browse/$(basename $f)" >> $the_report + echo "" >> $the_report + done + + add_risk_assemsment_to_report + fi + + echo "Report:" $the_report +} + +function sanity_check() { + if [ ! -e $(pwd)/.svn ]; then + print "This directory, $(pwd), " \ + "does not contain a working version control checkout." + exit 1 + fi + + # defaulting to Atlassian on demand + jira_base_url=${jira_base_url-https://vizrtcustomers.jira.com} + if [ -z "$svn_base_url" ]; then + # bash default string substitution terminates with slashes, hence + # have to set this manually here. + svn_base_url=${jira_base_url}/svn + fi + + conf_file=$HOME/.$(basename $0).conf + ensure_variable_is_set \ + project_code \ + user \ + password \ + jira_base_url \ + svn_base_url + + if [[ "$(get_from_revision)" == "$(get_to_revision)" ]]; then + print_and_log "From and to revision are the same," \ + "will no create any change log" + exit 0 + fi +} + +function get_project() { + echo $(lowercase ${project_code}) +} + +function get_archive_dir() { + local project_context=$(lowercase $(get_svn_location | \ + sed -e "s#${svn_base_url}/##g" -e 's#[ ]*##g') + ) + + local dir=${archive_base_dir}/${project_context} + make_dir $dir + echo $dir +} + +function add_risk_assemsment_to_report() { + echo "Risk assessment score: " \ + $(wc -l $the_diff 2>/dev/null | cut -d' ' -f1) \ + >> $the_report +} + +bootstrap_thyself +read_user_settings +get_user_input "$@" +sanity_check +get_commit_information_from_vcs +get_info_from_jira + + diff --git a/usr/bin/generate-git-changelog b/usr/bin/generate-git-changelog new file mode 100755 index 00000000..48dabf38 --- /dev/null +++ b/usr/bin/generate-git-changelog @@ -0,0 +1,194 @@ +#! /usr/bin/env bash + +log=$HOME/.$(basename $0).log +archive_base_dir=$HOME/.$(basename $0) + +function bootstrap_thyself() { + # first, try to be nice, then check the standard location + local dir=$(dirname $0)/../share/escenic/ece-scripts + if [ ! -d $dir ]; then + dir=/usr/share/escenic/ece-scripts + fi + + local common_libraries=" + common-bashing.sh + common-io.sh + " + + for el in $common_libraries; do + source $dir/$el 2>/dev/null || { + echo "$(basename $0): Could not load the library $el," \ + "and I can't live without it :-(" | fmt + exit 1 + } + done +} + +function get_header_from_jira() { + local body=$(curl -u ${user}:${password} -s ${jira_base_url}/browse/${1}) + echo "$body" | grep '<title>' | sed -e 's/<title>//g' -e 's/<\/title>//g' +} + +# --to revision number +function get_to_revision() { + echo ${to-COMMITTED} +} + +# --from revision number +function get_from_revision() { + echo ${from-PREV} +} + +function get_user_input() { + local next_is_from=0 + local next_is_to=0 + local next_is_project=0 + local next_is_user=0 + local next_is_password=0 + local next_is_jirabaseurl=0 + + for el in "$@"; do + if [[ "$el" == "-s" || "$el" == "--from" ]]; then + next_is_from=1 + elif [[ "$el" == "-u" || "$el" == "--user" ]]; then + next_is_user=1 + elif [[ "$el" == "-p" || "$el" == "--password" ]]; then + next_is_password=1 + elif [[ "$el" == "-t" || "$el" == "--to" ]]; then + next_is_to=1 + elif [[ "$el" == "-p" || "$el" == "--project" ]]; then + next_is_project=1 + elif [[ "$el" == "-j" || "$el" == "--jirabaseurl" ]]; then + next_is_jirabaseurl=1 + elif [[ "$el" == "-f" || "$el" == "--full" ]]; then + full_listing=1 + elif [ ${next_is_from-0} -eq 1 ]; then + from=$el + next_is_from=0 + elif [ ${next_is_user-0} -eq 1 ]; then + user=$el + next_is_user=0 + elif [ ${next_is_password-0} -eq 1 ]; then + password=$el + next_is_password=0 + elif [ ${next_is_to-0} -eq 1 ]; then + to=$el + next_is_to=0 + elif [ ${next_is_project-0} -eq 1 ]; then + project_code=$el + next_is_project=0 + elif [ ${next_is_jirabaseurl-0} -eq 1 ]; then + jirabaseurl=$el + next_is_jirabaseurl=0 + fi + done +} + + +function get_commit_information_from_vcs() { + the_diff=$(get_archive_dir)/from-$(get_from_revision)-to-$(get_to_revision).diff + + if [[ ! -e $the_diff ]]; then + git diff $(get_from_revision) $(get_to_revision) > $the_diff + exit_on_error "git diff $(get_from_revision) $(get_to_revision)" + fi + + if [ ${full_listing-0} -eq 1 ]; then + cat $the_diff + else + echo "Full diff of all" $(egrep '^(\+|\-)' $the_diff | wc -l) \ + "changes:" $the_diff | fmt + fi + + if [[ ! -z $jirabaseurl ]]; then + get_info_from_jira + fi + +} + +## $@ :: git last revision date +#Fri Sep 27 11:18:05 2013 +0600 +function get_date_from_git_log() { + echo $(git log -1 --format="%cd") +} + +# e.g. ssh://git@git.vizrtsaas.com/ccipoc +function get_git_location() { + echo $(git config --get remote.origin.url) +} + + +function get_info_from_jira() { + the_report=$(get_archive_dir)/from-$(get_from_revision)-to-$(get_to_revision).report + + if [[ ! -e $the_report ]]; then + echo "Changes in branch:$(get_project_branch) and location:$(get_git_location) " > $the_report + echo "From: revision $(get_from_revision) @" \ + $(get_date_from_git_log $(get_from_revision)) >> $the_report + echo "To : revision $(get_to_revision) @" \ + $(get_date_from_git_log $(get_to_revision)) >> $the_report + + local commit_log=$(git log $(get_from_revision)..$(get_to_revision)) + echo "$commit_log" | \ + grep $(get_jira_project_name)-[0-9]* | \ + sed "s#.*\($(get_jira_project_name)-[0-9]*\).*#\1#g" | \ + sort | \ + uniq | while read f; do + echo " *" $(get_header_from_jira $f) | fmt >> $the_report + echo " URL: ${jira_base_url}/browse/$(basename $f)" >> $the_report + echo "" >> $the_report + done + + add_risk_assemsment_to_report + fi + + echo "Report:" $the_report +} + +function sanity_check() { + #if [ ! -e $(pwd)/src/.git ]; then + # print "This directory, $(pwd), " \ + # "does not contain a working version control checkout." + #exit 1 + #fi + + # defaulting to Atlassian on demand + jira_base_url=${jirabaseurl} + + if [[ "$(get_from_revision)" == "$(get_to_revision)" ]]; then + print_and_log "From and to revision are the same," \ + "will no create any change log" + exit 0 + fi +} + +#e.g. ccipoc.git +function get_project_name() { + echo ${project_code} | awk '{split($0,array,"/")} END{print array[1]}' +} + +function get_jira_project_name() { + echo ${project_code} | awk '{split($0,array,"/")} END{print array[1]}' | sed 's/.git//g' | tr [a-z] [A-Z] + +} + +function get_project_branch() { + echo ${project_code} | awk '{split($0,array,"/")} END{print array[2]}' +} + +function get_archive_dir() { + local directory=${archive_base_dir}/$(get_project_name)/$(get_project_branch) + make_dir $directory + echo $directory +} + +function add_risk_assemsment_to_report() { + echo "Risk assessment score: " \ + $(wc -l $the_diff 2>/dev/null | cut -d' ' -f1) \ + >> $the_report +} + +bootstrap_thyself +get_user_input "$@" +sanity_check +get_commit_information_from_vcs diff --git a/usr/bin/sync-network-drive b/usr/bin/sync-network-drive new file mode 100755 index 00000000..6cec78cf --- /dev/null +++ b/usr/bin/sync-network-drive @@ -0,0 +1,198 @@ +#! /usr/bin/env bash + +# Script made for syncing the NFS exports between the serving +# hosts. The script ensures that there is only one instance of it +# running and thus that two simultaneous runs it will not cause +# corruption of data. +# +# +# Example usage with source (-s) and target (-t): +# +# $ sync-network-drive \ +# -s remote-server:/var/lib \ +# -t /var/backups/remote-server +# +# The full path on the remote server is replicated locally here, so +# after this command, you'll have remote-server's /var/lib directory +# under your local directory /var/backups/remote-server/var/lib And +# one last thing: trailing slashes are not needed + +log=/var/log/escenic/$(basename $0).log + +function bootstrap_thyself() { + # first, try to be nice, then check the standard location + local dir=$(dirname $0)/../share/escenic/ece-scripts + if [ ! -d $dir ]; then + dir=/usr/share/escenic/ece-scripts + fi + + for el in common-bashing.sh common-io.sh; do + source $dir/$el 2>/dev/null || { + echo "$(basename $0): Could not load the library $el," \ + "and I can't live without it :-(" | fmt + exit 1 + } + done +} + +bootstrap_thyself + +tmp_known_hosts_file=$(mktemp) + +rsync_opts=" +--recursive +--links +--perms +--group +--owner +--compress +--itemize-changes +--ignore-times +--checksum +--cvs-exclude +" +rsync_additional_outgoing_opts=" +--dry-run +--verbose +" +# the backslashes need to be here +rsync_ssh_opts="\ +-o BatchMode=yes \ +-o UserKnownHostsFile=${tmp_known_hosts_file} \ +-o StrictHostKeyChecking=no \ +-p 22 \ +" + +outgoing=0 +verbose=0 + +function ensure_src_is_sane() { + # checking that the src is <host>:<dir> + if [[ $(dirname $(echo "$src" | cut -d':' -f2 )) == /** ]]; then + return + else + print "Source $src is insane, should be <host>:<absolute dir>" + remove_lock + tidy_up + exit 1 + fi +} + +function ensure_target_dir_is_there() { + mkdir -p $target_dir || (echo "failed creating target dir"; exit 1 ) + + if [ ! -w $target_dir ]; then + print $USER "cannot write to $target_dir :-(" + remove_lock + tidy_up + exit 1 + fi +} + +function sync_it() { + if [ $outgoing -eq 1 ]; then + rsync_opts=${rsync_opts}" "${rsync_additional_outgoing_opts} + fi + + if [ $verbose -eq 1 ]; then + rsync_opts=${rsync_opts}" --verbose " + fi + + print "Sync on $HOSTNAME started @ $(stat -c %y $lock_file)" \ + "${src} -> ${target_dir}" + run rsync $rsync_opts \ + -e "ssh $rsync_ssh_opts" \ + $src/ \ + $target_dir/ +} + +function tidy_up() { + run rm $tmp_known_hosts_file +} + +function print_report() { + local now=$(date +%s) + local started=$(stat -c %Y $lock_file) + local seconds=$(( now - started )) + local days=$(( seconds / ( 60 * 60 * 24 ) )) + local seconds_left=$(( seconds - ( $days * 60 * 60 * 24 ) )) + local hours=$(( seconds_left / ( 60 * 60 ) )) + local seconds_left=$(( seconds_left - ( $hours * 60 * 60 ) )) + local minutes=$(( seconds_left / 60 )) + local seconds_left=$(( seconds_left - $minutes * 60 )) + + print "Hi! $src has now been synced to $target_dir" + print "It took" ${days}d ${hours}h ${minutes}m ${seconds_left}s + +} + +function get_user_options() { + while getopts ":s:t:ov" opt; do + case $opt in + s) + src=${OPTARG} + ;; + t) + target_dir_root=${OPTARG} + + if [[ $target_dir_root == "/" ]]; then + target_dir_root="" + fi + + target_dir=${target_dir_root}$(echo $src | cut -d":" -f2) + ;; + o) + outgoing=1 + ;; + v) + verbose=1 + ;; + \?) + print "Invalid option: -$OPTARG" >&2 + exit 1 + ;; + :) + print "Option -$OPTARG requires an argument." >&2 + exit 1 + ;; + *) + print "Command is $OPTARG" + exit 0 + esac + done +} + +function ensure_user_options_are_ok() { + if [ -z "${src}" ]; then + cat <<EOF +You must specify a source with -s <host>:<dir>, e.g.: +$(basename $0) -s app1:/var/log -t /var/backups/app1 +EOF + +tidy_up +exit 1 + fi + + if [ -z "${target_dir}" ]; then + cat <<EOF +You must specify a local target directorywith -t <target dir>, e.g.: +$(basename $0) -s app1:/var/log -t /var/backups/app1 +EOF + +tidy_up +exit 1 + fi + +} + +get_user_options $@ +ensure_user_options_are_ok +create_pid +create_lock +ensure_src_is_sane +ensure_target_dir_is_there +sync_it +print_report +remove_lock +tidy_up +remove_pid diff --git a/usr/bin/system-info b/usr/bin/system-info index 3df03f28..bc1297c0 100755 --- a/usr/bin/system-info +++ b/usr/bin/system-info @@ -1,193 +1,394 @@ #! /usr/bin/env bash -# Getting system info, especially useful before reporting to Escenic -# select Support. - -# can be: ascii, confluence -output_format=ascii -on_debian_or_derivate=0 -on_gentoo_or_derivate=0 -on_redhat_or_derivate=0 -on_linux=0 - -important_packages_on_debian=" -ant -apache2 -libapr1 -libmysql-java -libtcnative-1 -maven2 -mysql-server -nginx -percona-server-server -slapd -sun-java6-jdk -sun-java6-jre -tomcat6 -tomcat6-user -varnish -" -important_packages_on_gentoo=" -dev-db/percona-server -dev-java/ant-contrib -dev-java/ant-nodeps -dev-java/maven-bin -dev-libs/apr -net-misc/memcached -net-nds/openldap -virtual/jre-1.6.0 -www-servers/apache -www-servers/nginx -www-servers/varnish -" - -if [ `uname -s` = "Linux" ]; then - on_linux=1 -fi - -if [ -x /usr/bin/dpkg -a -e /etc/debian_version ]; then - on_debian_or_derivate=1 -elif [ -x /usr/bin/emerge -a -e /etc/gentoo-release ]; then - on_gentoo_or_derivate=1 -fi - - -function print_ruler() -{ - if [ $output_format = "ascii" ]; then - for i in {0..72}; do - echo -n $1 - done - fi +## Script which creates an overview of system information related to +## the specified ECE/EAE/Search instance, example invocation: +## +## $ system-info -i engine1 | xmllint --format - > /var/www/engine1.html +## +## Add this to /etc/cronttab if you want it to create an HTML report +## every minute: +## +## echo '* * * * * root system-info -f html > /var/www/index.html' >> /etc/crontab + +ece_user="" +ece_scripts_version="straight-from-github" + +## possible values: html, org, confluence, yaml, json (not complete) +format=yaml +generate_output_file_per_module=0 +output_dir=$(pwd) +output_file="" + +current_indent_level=0 +INDENT=" " +verbose=0 +temporaries=1 +log=$HOME/.$(basename $0).log + +function init() { + # first, try to be nice + ece_scripts_dir=$(dirname $0)/../share/escenic/ece-scripts + + # then check the standard location + if [ ! -d $ece_scripts_dir ]; then + ece_scripts_dir=/usr/share/escenic/ece-scripts + fi + + source $ece_scripts_dir/common-ece.sh + source $ece_scripts_dir/common-bashing.sh + source $ece_scripts_dir/common-io.sh } -function print_pre_start() -{ - if [ $output_format = "confluence" ]; then - echo "{code}" +## Runs extra system-info modules +function run_system_info_modules() { + if [ -d $ece_scripts_dir ]; then + # load system-info modules + if [ ! -d $ece_scripts_dir/system-info.d ]; then + return fi + + for el in $(\ls $ece_scripts_dir/system-info.d/*.sh 2>/dev/null); do + if [ $generate_output_file_per_module -eq 1 ]; then + source $el | tee ${output_dir}/$(basename $el).${format} + else + source $el + fi + done + fi } -function print_pre_end() -{ - if [ $output_format = "confluence" ]; then - echo "{code}" - fi +## $1 : indent level, optional, if not set, will use the +## current_indent_level +function get_indent() { + local result="" + + if [[ "${1}x" != "x" ]]; then + local number_of_indents=$1 + else + local number_of_indents=$current_indent_level + fi + + for (( i = 0; i < $number_of_indents; i++ )); do + result="${INDENT}$result" + done + + echo "$result" } -function print_header() +function create_header() { + local title="Overview of $HOSTNAME" + + if [ $format == "html" ]; then +cat <<EOF + +<html xmlns="http://www.w3.org/1999/xhtml"> + <head> + <title>$title @ $(date) + + + +$(cat $ece_scripts_dir/vizrt-logo-svg.html) +EOF + elif [ $format == "json" ]; then + cat </dev/null | grep ^ii | sed 's/ii\ \ //g' - done - elif [ $on_gentoo_or_derivate -eq 1 ]; then - for el in $important_packages_on_gentoo; do - equery --no-color --no-pipe list $el 2>/dev/null | \ - grep -v "Searching for" | \ - cut -d']' -f3- 2>/dev/null - done +function create_footer() { + if [ $format == "html" ]; then + cat < + +EOF + + elif [ $format == "json" ]; then + cat <$el" + else + result="$result $el" + fi + done + else + result="$@" + fi + + echo "$result" } -function list_os_info() -{ - print_header "Kernel version" - uname -a - - print_header "Distribution information" - if [ $on_debian_or_derivate -eq 1 ]; then - echo "Debian or derivate, version "`cat /etc/debian_version` - elif [ -r /etc/gentoo-release ]; then - cat /etc/gentoo-release - fi +function print_un_ordered_list_start() { + if [ $format == "org" ]; then + echo "" + elif [ $format == "html" ]; then + echo "
    " + elif [ $format == "confluence" ]; then + echo "" + elif [ $format == "json" ]; then + echo " [" + fi } -function list_db_information() -{ - print_header "Database details" - if [ -x /usr/sbin/mysqld ]; then - /usr/sbin/mysqld -V - else - mysql -V 2>/dev/null - mysql5 -V 2>/dev/null - fi +function print_un_ordered_list_end() { + if [ $format == "org" ]; then + echo "" + elif [ $format == "html" ]; then + echo "
" + elif [ $format == "confluence" ]; then + echo "" + elif [ $format == "json" ]; then + echo " ]," + fi } -function list_java_information() -{ - print_header "Java version" - java -version +function print_list_item() { + if [ $format == "org" ]; then + echo "- $@" + elif [ $format == "html" ]; then + echo "
  • " \ + "$@" \ + "
    • " + elif [ $format == "confluence" ]; then + echo "* $@" + elif [ $format == "json" ]; then + echo " \"$@\", " + elif [ $format == "yaml" ]; then + echo "$(get_indent) - $@" + fi } -function list_hardware_information() -{ - if [ $on_linux -eq 1 ]; then - print_header "Processors" - echo "Processor type: " `grep "model name" /proc/cpuinfo | head -1 | cut -d':' -f2` - echo "Number of processors/cores:" `grep processor /proc/cpuinfo | wc -l` - - print_header "Memory" - echo "Total memory: " `grep MemTotal /proc/meminfo | cut -d':' -f2` - echo "Free memory: " `grep MemFree /proc/meminfo | cut -d':' -f2` - fi +function json_escape_string() { + echo "$@" | sed 's/\"/\\"/g' +} - print_header "Disk Storage" - print_pre_start - if [ $(df --version | grep GNU | wc -l) -gt 0 ]; then - df -hT - else - df - fi - print_pre_end +function print_pre_text() { + if [ $format == "org" ]; then + cat < +$@ + +EOF + elif [ $format == "confluence" ]; then + cat <$@" + elif [ $format == "confluence" ]; then + echo "h3. $@" + elif [ $format == "json" ]; then + cat <$@" + elif [ $format == "confluence" ]; then + echo "h4. $@" + elif [ $format == "yaml" ]; then + echo "" + echo "$(get_indent 3)${@}:" + current_indent_level=3 + fi +} + +function print_h2_header() { + if [ $format == "org" ]; then + echo "" + echo "** $@" + elif [ $format == "html" ]; then + echo "

    $@

    " + elif [ $format == "confluence" ]; then + echo "h2. $@" + elif [ $format == "json" ]; then + cat <$@" + elif [ $format == "confluence" ]; then + echo "h1. $@" + elif [ $format == "yaml" ]; then + echo "${@}:" + current_indent_level=0 + fi } -for el in $@; do - if [ $el = "-c" -o $el = "--confluence" ]; then - output_format=confluence +function print_p_text() { + if [ $format == "org" ]; then + echo -e "$@" + elif [ $format == "html" ]; then + cat < + $@ +

    +EOF + elif [ $format == "confluence" ]; then + echo -e "$@" + elif [ $format == "json" ]; then + echo " \"message\": \"$text\"," + elif [ $format == "yaml" ]; then + local LINE_WRAP=72 + local line="" + text="" + + for el in $@; do + local tmp_line="${line} ${el}" + # wrap the line when it becomes too long + if [ ${#tmp_line} -gt $LINE_WRAP ]; then + text="${text}\n$(get_indent)${el}" + line="" + else + line="${line} ${el}" + fi + done + + # adding the last line line here. For one liners, text will be + # empty here, so the indent is straght on. For multi line input, + # the text variable already contains a new line at this point. + text="${text}\n$(get_indent)${line}" + echo -e "$text" + fi +} + +function get_user_options() { + # starting to add support for --long-version options + for el in $@; do + if [ $el = "-V" -o $el = "--version" ]; then + echo "Version:" $ece_scripts_version + exit 0 fi -done + done + + while getopts ":d:o:i:f:u:mvst" opt; do + case $opt in + v) + verbose=1 + ;; + s) + temporaries=0 + ;; + d) + output_dir=${OPTARG} + mkdir -p $output_dir + ;; + u) + ece_user=${OPTARG} + ;; + f) + format=${OPTARG} + ;; + o) + output_file=${OPTARG} + ;; + m) + generate_output_file_per_module=1 + ;; + t) + generate_import_job_overview=1 + ;; + \?) + echo "Invalid option: -$OPTARG" >&2 + exit 1 + ;; + :) + echo "Option -$OPTARG requires an argument." >&2 + exit 1 + ;; + esac + done -print_report_header -list_os_info -list_db_information -list_java_information -list_useful_package_info -list_hardware_information +} - - \ No newline at end of file +init +get_user_options $@ +create_header +run_system_info_modules +create_footer diff --git a/usr/bin/vosa b/usr/bin/vosa new file mode 100755 index 00000000..46ca34b0 --- /dev/null +++ b/usr/bin/vosa @@ -0,0 +1,1448 @@ +#!/bin/bash + +# /usr/bin/vosa --- command to manage vizrt on-line system-administration +# managed instances of virtual machines. + +# vosa list --- list all available virtual machines +# vosa -i /etc/vizrt/vosa/available.d/vm03 enable --- enable a specific vm +# vosa -i /etc/vizrt/vosa/available.d/vm03 disable --- disable a specific vm +# vosa -i /etc/vizrt/vosa/enabled.d/vm03 disable --- disable a vm +# vosa -i /etc/vizrt/vosa/enabled.d/vm03 install --- creates a new disk image etc. +# vosa -i /etc/vizrt/vosa/enabled.d/vm03 uninstall --- removes the disk image etc. +# vosa -i ... start +# vosa -i ... status --- tells you about the VM, if it's enabled, running, alive, its uptime. + +instance_dir= +instance_name= +usage=0 +ece_scripts_version="straight-from-github" + +### To add an option, add it to the optstring, as defined in getopts, in alphabetical order. +### Also add a function "option-x" where x is the option. $1 will be the option value, if any. +### set any variables needed to default values as globals first. +OPTSTRING=":i:v:hdV" + +function option-h() { + usage=1 +} + +function option-d() { + debug=$((debug + 1)) +} + +function option-V() { + echo "Version:" $ece_scripts_version + exit 0 +} + +function option-i() { + if [[ "${1}" =~ ^[-a-z0-9]*$ ]] ; then + instance_dir=${available_dir}/$1 + else + instance_dir="${1}" + fi + if [ ! -r $instance_dir ] ; then + echo "Instance $1 does not exist. Exiting" + exit 1 + else + instance_dir=$(readlink -f "$instance_dir") + instance_name=$(basename $instance_dir) + fi +} + +function option-v() { + uec_version="${1}" +} + + +function unknown-option() { + echo "Unknown option $@" + usage=1 +} + +### To add a command, simply define a function with a do_ prefix. +available_dir=$(dirname $0)/../../etc/vizrt/vosa/available.d +available_dir=$(readlink -f ${available_dir}) +enabled_dir=$(dirname $0)/../../etc/vizrt/vosa/enabled.d +enabled_dir=$(readlink -f ${enabled_dir}) + + +function requires_instance_dir() { + if [ -z "$instance_dir" -o -z "${instance_name}" ] ; then + echo "Instance is required" + usage=1 + return 1 + fi + + aws=0 + [ -r $instance_dir/amazon.conf ] && aws=1 + return 0 +} +function prohibits_instance_dir() { + if [ ! -z "$instance_dir" -o ! -z "${instance_name}" ] ; then + echo "Instance cannot be specified" + usage=1 + return 1 + fi +} + +## Provides instant gratification. +function do_help() { + cat < /etc/vizrt/vosa/skeleton-kvm/install.conf < /etc/vizrt/vosa/skeleton-kvm/boot.conf < /etc/vizrt/vosa/skeleton-kvm/ova.conf < /etc/vizrt/vosa/skeleton-kvm/sdp.xml < + + + + NAME + TIMEZONE + LOCATION + SHORTNAME + REALM + DOMAIN + + + + http://technet.escenic.com/ + + USERNAME + PASSWORD + + + + + + + demopub + demopub + demopub.\${domain} + + + + root + + PASSWORD + + + + escenic + adm + + PASSWORD + + + + + + http://no.archive.ubuntu.com/ubuntu/ + + + + unixodbc + defoma + java-common + xsltproc + http://secrets.vizrtsaas.com/static/packages/java/8/server.urilist + + PASSPHRASE + + + + python-dev + libxml2-dev + libxslt1-dev + lxml + + + + + + + + + + \${name} + \${timezone} + \${realm} + + + + + + + http://BUILDER-IP:PORT/escenic/CUSTOMER/releases/CUSTOMER-branch-master-latest.ear + + + + + + http://BUILDER-IP:PORT/escenic/CUSTOMER/releases/vosa-conf-CUSTOMER-1-CUSTOMER-branch-master-latest.deb + + +--> + + http://maven.escenic.com + + MAVEN-USER + MAVEN-PASSWORD + + + + http://maven.escenic.com/unstable + + MAVEN-USER + MAVEN-PASSWORD + + + + + + + + + + + + + + + http://ftp.heanet.ie/mirrors/www.apache.org/dist/tomcat/tomcat-7/ + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - -
    - - -

    Guide for the /usr/bin/ece Script

    - - - -
    -

    1 TAB completion

    -
    - -

    The ece script offers TAB completion, given that your version of BASH -supports this and that you have enabled it (some distributions hasn't -turned on BASH completion per default). -

    - -
    - -
    -

    1.1 Completing ece commands

    -
    - -

    You will find yourself using this regularly, both for speed, but also -to remember all the different options and their correct wording: -

    - - -
    $ ece TAB TAB
-applog      deploy      log         start       threaddump  
-assemble    help        outlog      status      update      
-clean       kill        restart     stop        versions 
-
-
    - - - -

    -The commands are all described in detail in "ece help" -

    -
    - -
    - -
    -

    1.2 Completing types of servers the ece scripts can operate on

    -
    - - - - -
    $ ece -t TAB TAB
-analysis  engine    rmi-hub   search 
-
-
    - - - - -
    - -
    - -
    -

    1.3 Completing the publication resources available

    -
    - -

    The ece script can update the publication resources for a given -publication. -

    -

    -To help selecting the correct publication resource, you can make ece -list and complete the available resource names: -

    - - -
    $ ece -p mypub -r TAB TAB
-content-type   image-version  layout-group   
-feature        layout         menu
-
-
    - - - -
    -
    - -
    - -
    -

    2 Getting an overview of all available options

    -
    - - - - -
    $ ece help
-Usage: /home/torstein/bin/ece [-t <type>] [-i <instance>] <command>
-
-DESCRIPTION
- -t --type <type>
-      The following types are available:
-      engine  -  The Escenic Content Engine, this is the default
-                 and is the assumed type if none is specified.
-      search  -  A standalone search indexer and solr instance
-      rmi-hub -  The RMI hub responsible for the internal 
-                 communication between the ECE instances.
-      analysis - The Escenic Analysis Engine also knows as 'Stats'
-
- -i --instance <instance>
-      The type instance, such as editor1, web1 or search1
-
- -p --publication <publication>
-      Needed only for updating publication resources
-
- -r --resource <resource>
-      Used for updating publication resources.
-      Must be one of: content-type, feature, layout, layout-group
-                      image-version, menu
-
- -v --verbose
-      Prints out debug statements, useful for debugging.
-
-The following commands are available:
-   applog     the type's app server log
-   assemble   runs the Assembly Tool *)
-   clean      removes temporary files created by /home/torstein/bin/ece *)
-   deploy     deploys the assembled EAR *)
-   help       prints this help screen
-   kill       uses force to stop the type
-   log        the type's Log4J log
-   outlog     the [ece#engine] script log (system out log)
-   restart    restarts the type
-   start      starts the type
-   status     checks if the type is running
-   stop       stops the type
-   threaddump write a thread dump to standard out (system out log)
-   update     update publication resources
-   versions   lists the ECE component versions
-
-*) only applicable if type is 'engine'
-
-
    - - - -
    -
    -
    -

    Date: 2011-08-25 12:08:05 CST

    -

    Author: Torstein Krause Johansen

    -

    Org version 7.6 with Emacs version 23

    -Validate XHTML 1.0 -
    -
    - - diff --git a/usr/share/doc/escenic/ece-guide.org b/usr/share/doc/escenic/ece-guide.org index f4f8e808..104c08d6 100644 --- a/usr/share/doc/escenic/ece-guide.org +++ b/usr/share/doc/escenic/ece-guide.org @@ -1,83 +1,199 @@ -Guide for the /usr/bin/ece Script +#+TITLE: ece +#+AUTHOR: Escenic Cloud Team +#+OPTIONS: H:6 num:5 toc:2 +* NAME +ece - command for managing your Escenic instances. + +* SYNOPSIS +ece [[-i --instance instance]] [[[-t --type type]]] [[[-p --publication +publication]]] [[[--uri ear-to-deploy]]] COMMAND + +* DESCRIPTION +Command which will let you manage your Escenic Content Engine, Escenic +Analysis Engine and search instances with ease. The script does +everything from starting & stopping the instance to building & +deploying EAR files and detailed information about the running +instances. + +* OPTIONS +** -t --type type +The following types are available: +- engine :: The Escenic Content Engine, this is the default and is the + assumed type if none is specified. + +- search :: A standalone search indexer and solr instance + +- analysis :: The Escenic Analysis Engine also knows as 'Stats' + +- rmi-hub :: The RMI hub responsible for the internal communication + between the ECE instances. This is not needed for ECE >= + 5.3. + +** -i --instance instance +The type instance, such as editor1, engine1 or search1 +** -p --publication publication +Needed only for updating publication resources +** -r --resource resource +Used for updating publication resources. Must be one of: +content-type, feature, layout, layout-group image-version, menu +** -v --verbose +Prints out debug statements, useful for debugging. + +* COMMANDS +You can specify one or more commands. The commands are executed in +the same order that you pass them to the ece command. +** applog +The type's app server log +** assemble +Runs the Assembly Tool. +** backup +Allows you to create backup of your entire Escenic installation, or +just specific parts of it. Without any additional options, ece +backup, will create a backup with: +- Solar & Nursery configuration (everything in /etc/escenic) +- DB dump +- Solar index +- Multimedia files +- The entire app server +- init.d script and its configuration +- state files + +You may exclude any parts you don't want: + +| Backup specific option | Explanation | +|------------------------+----------------------| +| --exclude-binaries | No software binaries | +| --exclude-solr | No Solar index | +| --exclude-multimedia | No multimedia files | +| --exclude-init | No init files | +| --exclude-state | No state files | + +** clean +Clears the temporary files from the instance's application server as +well as Assembly Tool work files (if available). +** deploy +Deploys the locally assembled EAR, available in +/var/cache/escenic/engine.ear. + +Optionally, you can specify an EAR on a remote server by passing the +--uri parameter. If the server is password protected, you can either +set builder_http_user and builder_http_password in ece.conf or pass +it on the command line: + +| Deploy specific option | Explanation | +|------------------------+------------------------------------| +| --file | Local EAR file to deploy | +| --uri | URI to the EAR to deploy | +| --user | HTTP User which may access the URI | +| --password | HTTP password to the user above. | + +** edit +Edit a publication resources. This command requires you to specify +both the publication and the resource which you wish to edit: +#+BEGIN_SRC sh +$ ece -i engine1 -p my-pub -r content-type edit +#+END_SRC +If you don't change anything, ece will not change the publication +resource. + +** flush +Clear all the ECE caches of the given instance. +** help +Prints this help screen +** info +Get information about the instance. +** kill +Uses force to stop the type +** list-deployments +Lists all the deployments done on this instance (granted that you've +used the ece script to make the deployment). +** list-instances +List the instances installed on your machine. +** list-logs +List all the log files for a given instance +** log +The type's Log4J log +** outlog +The [ece#engine] command log (system out log) +** restart +Restarts the type +** start +Starts the type +** status +Checks if the type is running +** stop +Stops the type +** threaddump +Write a thread dump to standard out (system out log) +** top +Watch the JSP top on your instance. Here, you can watch which JSPs +are consuming the most CPU time. + +** update +Update publication resources +** versions +Lists the version of the ECE and all the plugins. * TAB completion -The ece script offers TAB completion, given that your version of BASH +The ece command offers TAB completion, given that your version of BASH supports this and that you have enabled it (some distributions hasn't turned on BASH completion per default). -** Completing ece commands -You will find yourself using this regularly, both for speed, but also -to remember all the different options and their correct wording: +You can auto complete all options and commands, as well as the +instance names, publication names and resources types. + +* Log files +There are a number of log files associated with the different +instances. To get an overview of all the log files for a particular +instance, do: #+BEGIN_SRC sh -$ ece TAB TAB -applog deploy log start threaddump -assemble help outlog status update -clean kill restart stop versions +$ ece -i engine1 list-logs #+END_SRC -The commands are all described in detail in "ece help" -** Completing types of servers the ece scripts can operate on +To tail all of these, pass the -q parameter and pipe it to tail: #+BEGIN_SRC sh -$ ece -t TAB TAB -analysis engine rmi-hub search +$ ece -i engine1 -q list-logs | xargs tail -f #+END_SRC -** Completing the publication resources available -The ece script can update the publication resources for a given -publication. +* Configuration files +| Path | Explanation | +|----------------------------------+-----------------------------| +| /etc/escenic/ece.conf | The main configuration file | +| /etc/escenic/ece-.conf | Instance specific settings | +| $HOME/.ecerc | User specific settings. | -To help selecting the correct publication resource, you can make ece -list and complete the available resource names: +* Examples +Starting an instance: #+BEGIN_SRC sh -$ ece -p mypub -r TAB TAB -content-type image-version layout-group -feature layout menu +$ ece -i engine1 start #+END_SRC -* Getting an overview of all available options + +Creating a thread dump from an instance: +#+BEGIN_SRC sh +$ ece -i engine1 threaddump +#+END_SRC + +Building a new EAR file using the a locally available Assembly Tool, +deploy this new EAR and restart the instance: #+BEGIN_SRC sh -$ ece help -Usage: /home/torstein/bin/ece [-t ] [-i ] - -DESCRIPTION - -t --type - The following types are available: - engine - The Escenic Content Engine, this is the default - and is the assumed type if none is specified. - search - A standalone search indexer and solr instance - rmi-hub - The RMI hub responsible for the internal - communication between the ECE instances. - analysis - The Escenic Analysis Engine also knows as 'Stats' - - -i --instance - The type instance, such as editor1, web1 or search1 - - -p --publication - Needed only for updating publication resources - - -r --resource - Used for updating publication resources. - Must be one of: content-type, feature, layout, layout-group - image-version, menu - - -v --verbose - Prints out debug statements, useful for debugging. - -The following commands are available: - applog the type's app server log - assemble runs the Assembly Tool *) - clean removes temporary files created by /home/torstein/bin/ece *) - deploy deploys the assembled EAR *) - help prints this help screen - kill uses force to stop the type - log the type's Log4J log - outlog the [ece#engine] script log (system out log) - restart restarts the type - start starts the type - status checks if the type is running - stop stops the type - threaddump write a thread dump to standard out (system out log) - update update publication resources - versions lists the ECE component versions - -*) only applicable if type is 'engine' +$ ece -i engine1 assemble deploy restart #+END_SRC + +Stop the instance, deploy an EAR from the build server, clean all app +server temporary files including the JSP compilation directory and +start the instance again: +#+BEGIN_SRC sh +$ ece -i engine1 \\ + --uri http://builder.example.com/myproject-1.2.ear \\ + stop clean deploy start +#+END_SRC + + +* COPYRIGHT +Copyright 2011-2015 Escenic + +Licensed under the Apache License, Version 2.0, see +https://github.com/escenic/ece-scripts/COPYING for further details. + +* AUTHOR +Torstein Krause Johansen diff --git a/usr/share/doc/escenic/ece-guide.txt b/usr/share/doc/escenic/ece-guide.txt deleted file mode 100644 index d4fefd5d..00000000 --- a/usr/share/doc/escenic/ece-guide.txt +++ /dev/null @@ -1,74 +0,0 @@ - Guide for the ece Script - ======================== - -Author: Torstein Krause Johansen -Date: 2011-08-24 13:30:04 CST - - -Table of Contents -================= -1 TAB completion - 1.1 Completing ece commands - 1.2 Completing types of servers the ece scripts can operate on - 1.3 Completing the publication resources available -2 Getting an overview of all available options - - -1 TAB completion ------------------ -The ece script offers TAB completion, given that your version of BASH -supports this and that you have enabled it (some distributions hasn't -turned on BASH completion per default). - -1.1 Completing ece commands -============================ -You will find yourself using this regularly, both for speed, but also -to remember all the different options and their correct wording: - - - - $ ece TAB TAB - applog deploy log start threaddump - assemble help outlog status update - clean kill restart stop versions - - - - -The commands are all described in detail in "ece help" - -1.2 Completing types of servers the ece scripts can operate on -=============================================================== - - - $ ece -t TAB TAB - analysis engine rmi-hub search - - - - -1.3 Completing the publication resources available -=================================================== -The ece script can update the publication resources for a given -publication. - -To help selecting the correct publication resource, you can make ece -list and complete the available resource names: - - - - $ ece -p mypub -r TAB TAB - content-type image-version layout-group - feature layout menu - - - - -2 Getting an overview of all available options ------------------------------------------------ - - - $ ece help - - - diff --git a/usr/share/doc/escenic/ece-import-guide.org b/usr/share/doc/escenic/ece-import-guide.org new file mode 100644 index 00000000..1fe2a8a9 --- /dev/null +++ b/usr/share/doc/escenic/ece-import-guide.org @@ -0,0 +1,412 @@ +#+TITLE: The ece-import Guide +#+AUTHOR: Escenic Cloud Team +#+OPTIONS: H:6 num:5 toc:2 + +* NAME +ece-import - command for managing integration with and imports of +external content into the Escenic Content Engine (ECE). + +* SYNOPSIS +ece-import [[[--name name]]] [[[--publication publication]]] [[[--user user]]] [[[--password password]]] COMMAND + +* OPTIONS +** --name name +The name of the import job +** --publication publication +The publication name that this job belongs to. +** --user user +The HTTP user to access the download data (optional). This option only +applies when running the [[download-import-data]] command. +** --password password +The HTTP password to access the download data. This option only +applies when running the [[download-import-data]] command. + +* COMMANDS +** create +ece-import will happily set up a new import job for you. +The preferred way is to use an import archive, which you typically get +from the web site developers who've created the XSL files and so +on. When you have such an import archive, you apply it like this: +#+BEGIN_SRC text +# ece-import --import-archive import-archive.zip create +#+END_SRC + +See [[Structure of an import archive]] for a detailed specification on how +to create such an archive yourself. + +Alternatively, you can set up an import job without an import archive: +#+BEGIN_SRC text +# ece-import \\ + --name myjob \\ + --publication mypub \\ + create +#+END_SRC + +In both cases, *ece-import* will set up the file structure for your +import job. If you're not using an import archive you'll have to +either put your transformers & [[cron]] jobs on the system yourself, or +make sure that the XML you want import already is in the Escenic +Syndication XML format. + +** download-import-data +There's built-in support in *ece-import* for downloading files from +any FTP or HTTP server. Everything it needs to run this, is passed +from the command line: +#+BEGIN_SRC text +$ ece-import \\ + --name myjob \\ + --publication mypub \\ + --user my-ftp-user \\ + --password my-ftp-password \\ + --uri http://news-feed.com/my-import-job-feed \\ + download-import-data +#+END_SRC + +You can safely re-run this as many times as you wish, *ece-import* +will keep track of previously downloaded files and will not allow you +to run parallel processes to download date. If you ever need to +re-download third party data, you can remove the corresponding +resource from */var/lib/escenic/raw/mypub/myjob/download.state* + +** Running an import job +This is the standard command/operation of *ece-import*, to take 3rd +party XML present in the raw spool, apply all transformations on it +and move the finished XML to the Escenic import spool. + +When running it, you just need to pass the import job name and +publication name: +#+BEGIN_SRC text +$ ece-import \\ + --name myjob \\ + --publication mypub +#+END_SRC + +* DESCRIPTION +*ece-import* is the center piece of your infrastructure when +integrating and importing third party content with ECE. Here follows +is a quick run through of how it all fits together. + +#+BEGIN_SRC text +(1) ece-import create + → new directories + → new Nursery configuration files + → instructions on how to add the new job to ECE + → new cron jobs + +(2) cron → ece-import download-import-data + → /var/spool/escenic/raw/mypub/myjob/1.xml + +(3) cron → ece-import → + transformers/01-download-images-referenced-in-xml.sh → + transformers/02-fix-image-colour-spaces.pl → + transformers/03-tidy-up-xml.py → + transformers/04-convert-to-escenic-xml.xsl + → /var/spool/escenic/import/mypub/myjob/1.xml + +(4) ECE/XMLImportSchedule → imports article into ECE +#+END_SRC + +** (1) +Call *ece-import create* with or with an import archive to set up all +the necessary directories, Nursery configuration files, [[cron]] jobs +and so on for a new job. + +** (2) +Cron calls *ece-import download-import-data* which downloads any new +XML (or other) file to */var/spool/escenic/raw/mypub/myjob*. + +** (3) +Again, [[cron]] (this can be the same cron script, which runs after a +successful run of (2)) runs *ece-import*, this time without a +command. This task will iterate through all the transformers +inside */usr/share/escenic/import/mypub/myjob/transformers* and pass +the *1.xml* file to each of them. + +Once this is done, *ece-import* checks if the resulting file is a well +formed XML file and that it looks like the Escenic syndication XML +format. + +If it passes these tests, *ece-import* moves the *1.xml* to the spool +directory for the ECE Import service to notice it. The file is now +in */var/spool/escenic/import/mypub/myjob/1.xml* + +** (4) +The standard ECE Import service detects that there is a new XML file +for it in the Escenic Syndication XML format and imports it into +ECE. Users should now see the fresh article in e.g. Escenic Content +Studio. + +** Further reading +For further explanation of why *ece-import* was created and rationale +for how it was implemented, see the [[BACKGROUND]] section. + +* Structure of an import archive +When creating a new import configuration for your project, the +following directory structure is required: + +#+BEGIN_SRC text +/ +//transformers/-.xsl +//cron.hourly/ +//cron.every.five.minutes/ +#+END_SRC + +- publication name :: the name of the publication for which the import + job(s) are defined. You can have more than one publication in + each zip archive. +- import job name :: lowercase with hyphens between words (if more + than one) +- transformers :: directory with files prefixed with *-*, + indicating the order of transformation to apply to + your import job. If this is a xsl file, *ece-import* + will run *xsltproc* on the file, whereas .sh files + will be run in a bash wrapper. + + Each of the transformers will be called with one + argument, namely the input XML data. Each + transformer is responsible to write changes back to + the file. +- cron.hourly :: commands to be run every our. These will be put + in */etc/cron.hourly* on the import server. Be sure + to set the execute bit on the file and note that as + with all cron jobs, the file cannot have a file + suffix. +- cron.every.five.minutes :: commands to run every five minutes. + +Here, you see we have one publication called *mypub* with one import +job called *myjob* and a second publication with the name *otherpub* +which has the import job *otherjob*. + +#+BEGIN_SRC text +$ unzip -t my-great-import-archive.zip.zip +mypub/myjob/transformers/01-fix-encoding.sh +mypub/myjob/transformers/02-convert-all-cows-to-ducks.xsl +mypub/myjob/transformers/02-convert-duck-to-escenic-xml.xsl +mypub/myjob/cron.hourly/get-files-from-myjob-ftp +mypub/myjob/cron.every.five.minutes/ask-for-public-ip +otherpub/otherjob/transformers/01-from-other-to-escenic-xml.xsl +#+END_SRC + +As you can guess from the file names, +the *02-convert-all-cows-to-ducks.xsl* stylesheet will be first +applied to the incoming data (normally XML) and +the *02-convert-duck-to-escenic-xml.xsl* will be applied next before +the resulting Escenic XML will be imported into the Escenic Content +Engine. + +* BACKGROUND +The motivation for creating *ece-import* was to tackle a number of +problems we have seen over and over again in countless projects +related to importing third party content into ECE. + +** Easy to set up new import jobs +Integrating ECE with external systems to get the external data into +ECE has always been a challenging task. First off, it's the setting up +of an ECE import job, which demands several *.properties* files to be +written, put in the correct path on the file system, and added to the +correct *Initial.properties* to be bootstrapped. + +Another problem we've seen over and over again, is that all the +relevant directories either were not present, had the wrong +permissions or the import directories were created under the wrong +paths. All of which made the ECE import fail. + +Using ece-import [[create]], you don't have to worry about creating the +relevant Nursery components, create sufficient directory structure +with correct permissions or what the new stanza +in *Initial.properties* must look like to load your new import job +Nursery component. + +** Easier to make error free import data +But this is just the start of problems so many projects have faced up +through the years. The transformation of the external data, say a feed +from a new agency or video meta data from a streaming video service, +is another challenge. To transform these external formats, often a +different XML format, which sadly very often neither validates, has +the correct file encoding or is not even well formed (!). The related +images are also often faulty, having the wrong colour space or has +file names containing non-printable characters, which again are +impossible, or difficult to refer to in an error free way from the XML +files. + +All of the above make the ECE import fail. Since such errors easily +drowns in the other ECE log errors and are time consuming to replay, +we wanted to move the complexity of the import job away from the ECE +XML import framework so that the Escenic import configuration could be +as simple as possible, only relating to the standard Escenic XML +syndication format. + +By using *ece-import* it is really easy for you to write simple, small +BASH, Python or Perl scripts which can perform small "massaging" tasks +that you'd otherwise would have to do in a manual step prior to +feeding the external data to the ECE importer. For instance, to change +the encoding on the incoming XML files from ISO-8859-1 to UTF-8, you +could write create a new transformer +called *01-convert-from-iso-8859-1-to-utf-8.sh* like this: +#+BEGIN_SRC text +#! /usr/bin/env bash +iconv --from-code "iso-8859-1" --to-code "utf-8" $1 +#+END_SRC +and put it +in */usr/share/escenic/import/mypub/myjob/transformers*. Now, this +script would be called once for each new XML file. + +Before, all of these "massaging scripts" were called manually on a +fresh batch of imports (or say migration data), before the data was +passed on to the ECE import job. Now, all of these scripts can be +incorporated into the main import process, having a clear structure, +order of execution, error handling and logging. + +** Easy to debug transformations +In a standard ECE import job setup, you will have one or more import +filters which in turn run an XSL sheet. If something goes wrong with +one of them, most Escenic engineers and consultants (and probably +customer admins too), would test the transformation using a command +line tool such as *xsltproc* +#+BEGIN_SRC text +$ xsltproc from-reuters-to-escenic.xsl some-reuters-article.xml +#+END_SRC + +The reason that most people use this approach as this is far easier +than re-configuring the import job to use the ECE import job debug +filter, which involves editing *.properties* file(s), turning on debug +in the running app server log configuration and peeking in the correct +log file. + +By using *ece-import*, everything is set up to be easy to retry +manually. The whole concept is that it should be easy to take just one +step and execute it from the command line. *ece-import* provides +extensive messages about everything it's doing +in */var/log/escenic/ece-import.log* and you should have no problems +checking each of the transformation steps by yourself by reading it. + +** Easy to download data from FTP and HTTP sources +Downloading data from external sources via FTP or HTTP has been done +so many times, often with buggy shell code, leaving the scripts +running multiple times in paralell, re-downloading the same files +over and over again, or failing because the output of the directory index +listing has changed. + +By using [[download-import-data][ece-import download-import-data]] you get tried and tested code +for doing this which addresses all of the above problems: +- Works with all web servers and all FTP servers +- Doesn't depend on the directory listing format, the locale of the + web server or any other, non-standard pre-requisite. +- Keeps track of the state of previous runs/downloads for the + different import jobs +- Ensures that only one instance of the download job is running at + any given time. +- Runs on any Linux or UNIX system, using standard BASH and cron + building blocks. + +** No additional configuration +There are enough configuration files on a production system, so +the *ece-import* itself doesn't have any configuration files to add +to that burden. Everything it needs is read from the command line +options or from configuration files already present on the system. + +* CREATING TRANSFORMERS +You can write a transformer in either XSL, Perl, Python or BASH. You +put it in *//transformers/-.<{xsl,pl,py,sh}>* +and specify the order in which it should run by setting the +before/after your other transformers (if any, many folks only have one +transformer which is an XSL file). + +All transformers are run by the ece-import command and they get one +argument, namely the raw XML file from the 3rd party system. All +transformers work on the same XML file, so that changes done in +e.g. *01-first.sh* are passed on to *02-second.pl*. Each transformer +must read the file and write to the same file. That's the contract. + +Here's an example of a transformer that downloads all the pictures +mentioned in the raw XML. ece-import will copy any multimedia files +to the ECE import spool for you, so the transformer only needs to +worry about downloading these to the directory of the incoming XML +file: + +#+BEGIN_SRC sh +#! /usr/bin/env bash + +# Command which will download all thumbnails listed in the XML +# exported from VMEO. +# +# It is normally called from ece-import, but can also be called +# directly (when debugging). + +xpath_to_get_thumbnail_urls"/didl:DIDL/didl:Item/didl:Component/didl:Decommandor/didl:Statement/mpeg7:Mpeg7/mpeg7:Decommandion/mpeg7:Summarization/mpeg7:Summary/mpeg7:VisualSummaryComponent/mpeg7:ImageLocator/mpeg7:MediaUri" + +# $1 is the raw XML +cat "$1" | \\ + xml_grep --nowrap --cond $xpath_to_get_thumbnail_urls | \\ + sed 's/>\n\(.*\)<.*#\1#g" | while read url; do + wget \\ + --quiet \\ + --continue \\ + --output-document $(dirname $1)/$(basename $url) \\ + $url; +done +#+END_SRC + +* FILES +- /usr/share/escenic/import// :: Directory holding + the [[cron]] scripts and transformers of the import job. Normally, + set up by the *ece-import* [[create]] command. + +- /var/spool/escenic/raw// :: The *ece-import* + [[download-import-data]] command will download files to this + directory and when *ece-import* is invoked with the standard + operation, it will look in here for new 3rd party XML files. + +- /var/spool/escenic/import// :: Once *ece-import* has + applied all the transformations on the 3rd party XML and it has + been checked to be valid Escenic XML, the file is put in + here. The ECE XML Import Service looks in this directory for new + files to import. + +- /var/cache/escenic/import/// :: This is the work + directory of *ece-import* where raw 3rd party XML files are put + while being processed by the transformers. + +- /var/backups/escenic/import///succeeded/.gz :: When + 3rd party XML has been successfully transformed, the original + XML is compressed with [[gzip]] and put here. + +- /var/backups/escenic/import///failed/ :: When + 3rd party XML fails to be transformed, the original + XML is moved here. It's not compressed to make it as easy as + possible to retry it. + +- /var/log/escenic/ece-import.log :: *ece-import* writes detailed + transcripts of everything it does to this file making it easy + for you to replay any of the steps. + +- /var/run/escenic/ece-import-.lock :: *ece-import* will + create different lock files depending on the three + commands/operations it supports. I.e. a long running download of + 3rd party data (*ece-import download-import-data*) will not stop a + regular import to run (*ece-import*). + +- /var/lib/escenic/raw///download.state :: State file + holding all the URLs (HTTP or FTP) that *ece-import* has + downloaded before. + +* TROUBLESHOOTING + +To trigger a (re)import of an already imported file, +- edit the /var/lib/escenic/raw///download.state file as the escenic user and remove the respective filenames +or +- use sed '/pattern to match/d' sed /var/lib/escenic/raw///download.state > /var/lib/escenic/raw///download.newstate +- verify that /var/lib/escenic/raw///download.newstate is good +- mv /var/lib/escenic/raw///download.newstate /var/lib/escenic/raw///download.state + +then wait a while for the import to happen again. or trigger it with the ece-import command. + +* COPYRIGHT +Copyright 2012-2015 Escenic + +Licensed under the Apache License, Version 2.0, see +https://github.com/escenic/ece-scripts/COPYING for further details. + +* AUTHOR +Torstein Krause Johansen diff --git a/usr/share/doc/escenic/ece-install-conf-reference.md b/usr/share/doc/escenic/ece-install-conf-reference.md new file mode 100644 index 00000000..afd0a9c6 --- /dev/null +++ b/usr/share/doc/escenic/ece-install-conf-reference.md @@ -0,0 +1,373 @@ +# Configuration Reference for ece-install + +`ece-install` can be configured either in YAML (new) or `.conf` +(old). You pass the file the same way regardless of format: +``` +# ece-install -f .yaml +# ece-install -f .conf +``` + +Both versions are listed below, grouped by topic. + +## environment + +``` +environment: + type: ${environment_type} + java_home: ${foo_java_home} + java_version: ${foo_java_version} + skip_password_checks: true + conf_url: ${conf_url} + apt: + escenic: + pool: ${apt_pool} + maven: + repositories: + - ${mvn_repo1} + - ${mvn_repo2} +``` + +- `ece-install.conf` equivalent: `java_home` +- `ece-install.conf` equivalent: `fai_environment` +- `ece-install.conf` equivalent: `fai_server_java_version` +- `ece-install.conf` equivalent: `fai_maven_repositories` +- `ece-install.conf` equivalent: `fai_conf_url` + +## editor + +``` +profiles: + editor: + install: yes + port: ${editor_port} + host: ${editor_host} + name: ${editor_name} + redirect: ${editor_redirect} + shutdown: ${editor_shutdown} +``` + +- `ece-install.conf` equivalent: `fai_editor_install` +- `ece-install.conf` equivalent: `fai_editor_port` +- `ece-install.conf` equivalent: `fai_editor_shutdown` +- `ece-install.conf` equivalent: `fai_editor_redirect` +- `ece-install.conf` equivalent: `fai_editor_name` + + +## presentation + +``` +profiles: + presentation: + install: yes + ear: ${presentation_ear} + environment: ${presentation_environment} + host: ${presentation_host} + name: ${presentation_name} + port: ${presentation_port} + redirect: ${presentation_redirect} + shutdown: ${presentation_shutdown} + deploy_white_list: ${presentation_deploy_white_list} + search_indexer_ws_uri: ${presentation_search_indexer_ws_uri} +``` + +- `ece-install.conf` equivalent: `fai_presentation_ear` +- `ece-install.conf` equivalent: `fai_presentation_environment` +- `ece-install.conf` equivalent: `fai_presentation_install` +- `ece-install.conf` equivalent: `fai_presentation_name` +- `ece-install.conf` equivalent: `fai_presentation_port` +- `ece-install.conf` equivalent: `fai_presentation_redirect` +- `ece-install.conf` equivalent: `fai_presentation_shutdown` +- `ece-install.conf` equivalent: `fai_presentation_deploy_white_list` +- `ece-install.conf` equivalent: `fai_presentation_search_indexer_ws_uri` + + +## db + +``` +profiles: + db: + install: ${db_install} + master: true + user: ${db_user} + ear: ${db_ear} + password: ${db_password} + schema: ${db_schema} + host: ${db_host} + port: ${db_port} + drop_old_db_first: yes + replication: yes +``` + +- `ece-install.conf` equivalent: `fai_db_install` +- `ece-install.conf` equivalent: `fai_db_master` +- `ece-install.conf` equivalent: `fai_db_user` +- `ece-install.conf` equivalent: `fai_db_password` +- `ece-install.conf` equivalent: `fai_db_schema` +- `ece-install.conf` equivalent: `fai_db_host` +- `ece-install.conf` equivalent: `fai_db_port` +- `ece-install.conf` equivalent: `fai_db_ear` +- `ece-install.conf` equivalent: `fai_db_drop_old_db_first` +- `ece-install.conf` equivalent: `fai_db_replication` + +## cache +``` +profiles: + cache: + install: yes +``` + +- `ece-install.conf` equivalent: `fai_presentation_install` + +## monitoring +``` +profiles: + monitoring: + install: yes +``` + +- `ece-install.conf` equivalent: `fai_monitoring_install` + +## credentials + +``` +credentials: + - site: maven.escenic.com + user: ${escenic_download_user} + password: ${escenic_download_password} + - site: builder + user: ${builder_download_user} + password: ${builder_download_password} +``` + +- `ece-install.conf` equivalent: `technet_user` +- `ece-install.conf` equivalent: `technet_password` +- `ece-install.conf` equivalent: `fai_builder_http_user` +- `ece-install.conf` equivalent: `fai_builder_http_password` +- `ece-install.conf` equivalent: `fai_conf_builder_http_user` +- `ece-install.conf` equivalent: `fai_conf_builder_http_password` + + +## create_publication + + +``` +profiles: + publications: + - name: ${publication1_name} + war: ${publication1_war} + war_remove_list: + - ${publication1_remove_file1} + - ${publication1_remove_file2} + webapps: + - ${publication_webapp1} + - ${publication_webapp2} + domain: ${publication1_domain} + ear: ${publication_ear} + environment: ${publication1_environment} + aliases: + - ${publication1_alias1} + - ${publication1_alias2} +``` + +- `ece-install.conf` equivalent: `fai_publication_domain_mapping_list` +- `ece-install.conf` equivalent: `fai_publication_ear` +- `ece-install.conf` equivalent: `fai_publication_war_remove_file_list` +- `ece-install.conf` equivalent: `fai_publication_environment` +- `ece-install.conf` equivalent: `fai_publication_webapps` +- `ece-install.conf` equivalent: `fai_publications_webapps # arg, the plural` + + +## publication + + +``` +profiles: + publications: + - name: ${publication1_name} + war: ${publication1_war} + domain: ${publication1_domain} + aliases: + - ${publication1_alias1} + - ${publication1_alias2} + - name: ${publication2_name} + war: ${publication2_war} + domain: ${publication2_domain} + aliases: + - ${publication2_alias1} + - ${publication2_alias2} +``` + +- `ece-install.conf` equivalent: `fai_publication_domain_mapping_list` + +## packages + +``` +packages: + - name: ${package_name} + version: ${package_version} +``` +- `ece-install.conf` equivalent: `fai_package_map` + declare -A fai_package_map + +## packages_multiple + + +``` +packages: + - name: ${package_name} + version: ${package_version} + - name: ${package_name_without_version} +``` +- `ece-install.conf` equivalent: `fai_package_map` + declare -A fai_package_map + +## use_escenic_packages + +``` +packages: + foo: 1 +``` + +- `ece-install.conf` equivalent: `fai_package_enabled` + +## restore + +``` +profiles: + restore: + pre_wipe_solr: true + pre_wipe_all: true + pre_wipe_logs: true + pre_wipe_cache: true + pre_wipe_crash: true + from_backup: true + data_files: true + software_binaries: true + db: true + configuration: true + from_file: ${restore_from_file} +``` +- `ece-install.conf` equivalent: `fai_restore_pre_wipe_solr` +- `ece-install.conf` equivalent: `fai_restore_pre_wipe_all` +- `ece-install.conf` equivalent: `fai_restore_pre_wipe_logs` +- `ece-install.conf` equivalent: `fai_restore_pre_wipe_cache` +- `ece-install.conf` equivalent: `fai_restore_pre_wipe_crash` +- `ece-install.conf` equivalent: `fai_restore_from_backup` +- `ece-install.conf` equivalent: `fai_restore_data_files` +- `ece-install.conf` equivalent: `fai_restore_software_binaries` +- `ece-install.conf` equivalent: `fai_restore_db` +- `ece-install.conf` equivalent: `fai_restore_configuration` +- `ece-install.conf` equivalent: `fai_restore_from_file` + +## analysis + + +``` +profiles: + analysis: + install: yes + name: ${analysis_name} + port: ${analysis_port} + host: ${analysis_host} + shutdown: ${analysis_shutdown} + redirect: ${analysis_redirect} +``` + +- `ece-install.conf` equivalent: `fai_analysis_install` +- `ece-install.conf` equivalent: `fai_analysis_name` +- `ece-install.conf` equivalent: `fai_analysis_port` +- `ece-install.conf` equivalent: `fai_analysis_host` +- `ece-install.conf` equivalent: `fai_analysis_shutdown` +- `ece-install.conf` equivalent: `fai_analysis_redirect` + +## analysis_db + + +``` +profiles: + analysis_db: + install: yes + user: ${analysis_db_user} + password: ${analysis_db_password} + schema: ${analysis_db_schema} +``` + +- `ece-install.conf` equivalent: `fai_analysis_db_install` +- `ece-install.conf` equivalent: `fai_analysis_db_user` +- `ece-install.conf` equivalent: `fai_analysis_db_password` +- `ece-install.conf` equivalent: `fai_analysis_db_schema` + +## search + +``` +profiles: + search: + install: yes + legacy: yes + ear: ${search_ear} + for_editor: true + indexer_ws_uri: ${search_indexer_ws_uri} + port: ${search_port} + host: ${search_host} + name: ${search_name} + redirect: ${search_redirect} + shutdown: ${search_shutdown} +``` + +- `ece-install.conf` equivalent: `fai_search_install` +- `ece-install.conf` equivalent: `fai_search_host` +- `ece-install.conf` equivalent: `fai_search_port` +- `ece-install.conf` equivalent: `fai_search_shutdown` +- `ece-install.conf` equivalent: `fai_search_redirect` +- `ece-install.conf` equivalent: `fai_search_name` +- `ece-install.conf` equivalent: `fai_search_legacy` +- `ece-install.conf` equivalent: `fai_search_for_editor` +- `ece-install.conf` equivalent: `fai_search_ear` +- `ece-install.conf` equivalent: `fai_search_indexer_ws_uri` + + +## editor_install_multi_profiles +``` +profiles: + editor: + install: yes + search: + install: yes + db: + install: no +``` + +- `ece-install.conf` equivalent: `fai_editor_install` +- `ece-install.conf` equivalent: `fai_search_install` +- `ece-install.conf` equivalent: `fai_db_install` + + +## db +``` +profiles: + db: + install: yes +``` + +- `ece-install.conf` equivalent: `fai_db_install` + +## cache + +``` +profiles: + cache: + install: yes + port: ${cache_port} + conf_dir: ${cache_conf_dir} + backends: + - ${cache_be1} + - ${cache_be2} +``` + +- `ece-install.conf` equivalent: `fai_cache_install` +- `ece-install.conf` equivalent: `fai_cache_backends` +- `ece-install.conf` equivalent: `fai_cache_conf_dir` +- `ece-install.conf` equivalent: `fai_cache_port` + +--- +Reference guide generated: Mon Mar 13 11:47:16 CET 2017 diff --git a/usr/share/doc/escenic/ece-install-conf-reference.org b/usr/share/doc/escenic/ece-install-conf-reference.org new file mode 100644 index 00000000..3f884e6e --- /dev/null +++ b/usr/share/doc/escenic/ece-install-conf-reference.org @@ -0,0 +1,606 @@ +Overview of =ece-install='s configuration options. Generated from the +configuration file parser's unit tests @ Thu Aug 17 16:02:10 CST 2017. + +*** environment + +#+begin_src yaml +environment: + type: ${environment_type} + java_home: ${foo_java_home} + java_download_url: ${foo_java_download_url} + java_oracle_licence_accepted: true + skip_password_checks: true + conf_url: ${conf_url} + apt: + escenic: + pool: ${apt_pool} + deb: + escenic: + use_deb_not_apt: true + base_url: ${deb_base_url} + rpm: + escenic: + base_url: ${rpm_base_url} + + maven: + repositories: + - ${mvn_repo1} + - ${mvn_repo2} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +java_home= +fai_environment= +fai_java_download_url= +fai_java_oracle_licence_accepted= +fai_maven_repositories= +fai_conf_url= +fai_package_rpm_base_url= +fai_package_deb_not_apt= + +#+end_src + +*** editor + +#+begin_src yaml +profiles: + editor: + install: yes + port: ${editor_port} + host: ${editor_host} + name: ${editor_name} + redirect: ${editor_redirect} + shutdown: ${editor_shutdown} + ear: ${editor_ear} + deploy_white_list: ${editor_deploy_white_list} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +fai_editor_install= +fai_editor_port= +fai_editor_shutdown= +fai_editor_redirect= +fai_editor_name= +fai_editor_ear= +fai_editor_deploy_white_list= + +#+end_src + +*** presentation + +#+begin_src yaml +profiles: + presentation: + install: yes + ear: ${presentation_ear} + environment: ${presentation_environment} + host: ${presentation_host} + name: ${presentation_name} + port: ${presentation_port} + redirect: ${presentation_redirect} + shutdown: ${presentation_shutdown} + deploy_white_list: ${presentation_deploy_white_list} + search_indexer_ws_uri: ${presentation_search_indexer_ws_uri} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +fai_presentation_ear= +fai_presentation_environment= +fai_presentation_install= +fai_presentation_name= +fai_presentation_port= +fai_presentation_redirect= +fai_presentation_shutdown= +fai_presentation_deploy_white_list= +fai_presentation_search_indexer_ws_uri= + +#+end_src + +*** search + +#+begin_src yaml +profiles: + search: + install: yes + legacy: yes + ear: ${search_ear} + for_editor: true + indexer_ws_uri: ${search_indexer_ws_uri} + port: ${search_port} + host: ${search_host} + name: ${search_name} + redirect: ${search_redirect} + shutdown: ${search_shutdown} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +fai_search_install= +fai_search_host= +fai_search_port= +fai_search_shutdown= +fai_search_redirect= +fai_search_name= +fai_search_legacy= +fai_search_for_editor= +fai_search_ear= +fai_search_indexer_ws_uri= + +#+end_src + +*** db + +#+begin_src yaml +profiles: + db: + install: ${db_install} + master: true + user: ${db_user} + ear: ${db_ear} + password: ${db_password} + schema: ${db_schema} + host: ${db_host} + port: ${db_port} + drop_old_db_first: yes + replication: yes + vendor: ${_db_vendor} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +fai_db_install= +fai_db_master= +fai_db_user= +fai_db_password= +fai_db_schema= +fai_db_host= +fai_db_port= +fai_db_ear= +fai_db_drop_old_db_first= +fai_db_replication= +db_vendor= + +#+end_src + +*** cache + +#+begin_src yaml +profiles: + cache: + install: yes + conf_dir: ${conf_dir} + port: ${port} + backends: + - ${be1} + - ${be2} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +fai_cache_install= +fai_cache_backends= +fai_cache_port= +fai_cache_conf_dir= + +#+end_src + +*** cue + +#+begin_src yaml +profiles: + cue: + install: yes + backend_ece: ${cue_backend_ece} + backend_ece_local: ${cue_backend_ece_local} + backend_ng: ${cue_backend_ng} + backend_bridge: ${cue_backend_bridge} + cors_origins: + - ${cue_cors_origin1} + - ${cue_cors_origin2} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +fai_cue_install= +fai_cue_backend_ece= +fai_cue_backend_ece_local= +fai_cue_backend_ng= +fai_cue_backend_bridge= +fai_cue_cors_origins= + +#+end_src + +*** sse_proxy + +#+begin_src yaml +profiles: + sse_proxy: + install: yes + exposed_host: ${exposed_host} + exposed_port: ${exposed_port} + ece_port: ${sse_proxy_ece_port} + ece_redirect: ${sse_proxy_ece_redirect} + backends: + - uri: ${sse_proxy_backend1_uri} + user: ${sse_proxy_backend1_user} + password: ${sse_proxy_backend1_password} + - uri: ${sse_proxy_backend2_uri} + user: ${sse_proxy_backend2_user} + password: ${sse_proxy_backend2_password} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +fai_sse_proxy_backends= +fai_sse_proxy_ece_port= +fai_sse_proxy_ece_redirect= +fai_sse_proxy_exposed_host= +fai_sse_proxy_exposed_port= +fai_sse_proxy_install= + + sse_proxy_backends="${sse_proxy_backend2_uri} ${sse_proxy_backend2_user} ${sse_proxy_backend2_password} +${sse_proxy_backend1_uri} ${sse_proxy_backend1_user} ${sse_proxy_backend1_password} +" + +#+end_src + +*** nfs_server + +#+begin_src yaml +profiles: + nfs_server: + install: yes + server_address: ${nfs_server_address} + allowed_client_network: ${nfs_allowed_client_network} + export_list: ${nfs_export_list} + client_mount_point_parent: ${nfs_client_mount_point_parent} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +fai_nfs_export_list= +fai_nfs_server_address= +fai_nfs_server_install= +fai_nfs_allowed_client_network= +fai_nfs_client_mount_point_parent= + +#+end_src + +*** nfs_client + +#+begin_src yaml +profiles: + nfs_client: + install: yes + server_address: ${nfs_server_address} + allowed_client_network: ${nfs_allowed_client_network} + export_list: ${nfs_export_list} + client_mount_point_parent: ${nfs_client_mount_point_parent} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +fai_nfs_export_list= +fai_nfs_server_address= +fai_nfs_client_install= +fai_nfs_allowed_client_network= +fai_nfs_client_mount_point_parent= + +#+end_src + +*** assembly_tool +#+begin_src yaml +profiles: + assembly_tool: + install: yes +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +fai_assembly_tool_install= +#+end_src + +*** credentials + +#+begin_src yaml +credentials: + - site: maven.escenic.com + user: ${escenic_download_user} + password: ${escenic_download_password} + - site: builder + user: ${builder_download_user} + password: ${builder_download_password} + - site: unstable.yum.escenic.com + user: ${unstable_yum_user} + password: ${unstable_yum_password} + - site: unstable.apt.escenic.com + user: ${unstable_apt_user} + password: ${unstable_apt_password} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +technet_user= +technet_password= +fai_package_rpm_user= +fai_package_rpm_password= +fai_package_apt_user= +fai_package_apt_password= +fai_builder_http_user= +fai_builder_http_password= +fai_conf_builder_http_user= +fai_conf_builder_http_password= + +#+end_src + +*** credentials_stable_yum + +#+begin_src yaml +credentials: + - site: yum.escenic.com + user: ${stable_yum_user} + password: ${stable_yum_password} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +fai_package_rpm_user= +fai_package_rpm_password= + +#+end_src + +*** credentials_stable_apt + +#+begin_src yaml +credentials: + - site: apt.escenic.com + user: ${stable_apt_user} + password: ${stable_apt_password} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +fai_package_apt_user= +fai_package_apt_password= + +#+end_src + +*** create_publication + + +#+begin_src yaml +profiles: + publications: + - name: ${publication1_name} + create: true + update_app_server_conf: true + update_ece_conf: true + update_nursery_conf: true + war: ${publication1_war} + war_remove_list: + - ${publication1_remove_file1} + - ${publication1_remove_file2} + webapps: + - ${publication_webapp1} + - ${publication_webapp2} + domain: ${publication1_domain} + ear: ${publication_ear} + environment: ${publication1_environment} + aliases: + - ${publication1_alias1} + - ${publication1_alias2} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +fai_publication_domain_mapping_list= +fai_publication_ear= +fai_publication_update_app_server_conf= +fai_publication_update_ece_conf= +fai_publication_update_nursery_conf= +fai_publication_war_remove_file_list= +fai_publication_environment= +fai_publication_webapps= +fai_publications_webapps # arg, the plural= + +#+end_src + +*** publication + + +#+begin_src yaml +profiles: + publications: + - name: ${publication1_name} + war: ${publication1_war} + domain: ${publication1_domain} + aliases: + - ${publication1_alias1} + - ${publication1_alias2} + - name: ${publication2_name} + war: ${publication2_war} + domain: ${publication2_domain} + aliases: + - ${publication2_alias1} + - ${publication2_alias2} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +fai_publication_domain_mapping_list= +#+end_src + +*** packages + +#+begin_src yaml +packages: + - name: ${package_name} + version: ${package_version} + arch: ${package_arch} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text +fai_package_map= +fai_package_arch_map= + declare -A fai_package_map + declare -A fai_package_arch_map +#+end_src + +*** packages_multiple + + +#+begin_src yaml +packages: + - name: ${package_name} + version: ${package_version} + - name: ${package_name_without_version} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text +fai_package_map= + declare -A fai_package_map +#+end_src + +*** analysis + + +#+begin_src yaml +profiles: + analysis: + install: yes + name: ${analysis_name} + port: ${analysis_port} + host: ${analysis_host} + shutdown: ${analysis_shutdown} + redirect: ${analysis_redirect} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +fai_analysis_install= +fai_analysis_name= +fai_analysis_port= +fai_analysis_host= +fai_analysis_shutdown= +fai_analysis_redirect= +#+end_src + +*** analysis_db + + +#+begin_src yaml +profiles: + analysis_db: + install: yes + user: ${analysis_db_user} + password: ${analysis_db_password} + schema: ${analysis_db_schema} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +fai_analysis_db_install= +fai_analysis_db_user= +fai_analysis_db_password= +fai_analysis_db_schema= +#+end_src +_ +*** use_escenic_packages + +#+begin_src yaml +packages: + foo: 1 +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +fai_package_enabled= +#+end_src + +*** restore + +#+begin_src yaml +profiles: + restore: + pre_wipe_solr: true + pre_wipe_all: true + pre_wipe_logs: true + pre_wipe_cache: true + pre_wipe_crash: true + from_backup: true + data_files: true + software_binaries: true + db: true + configuration: true + from_file: ${restore_from_file} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text +fai_restore_pre_wipe_solr= +fai_restore_pre_wipe_all= +fai_restore_pre_wipe_logs= +fai_restore_pre_wipe_cache= +fai_restore_pre_wipe_crash= +fai_restore_from_backup= +fai_restore_data_files= +fai_restore_software_binaries= +fai_restore_db= +fai_restore_configuration= +fai_restore_from_file= +#+end_src + +*** editor_install_multi_profiles +#+begin_src yaml +profiles: + editor: + install: yes + search: + install: yes + db: + install: no +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +fai_editor_install= +fai_search_install= +fai_db_install= + +#+end_src + +*** cache + +#+begin_src yaml +profiles: + cache: + install: yes + port: ${cache_port} + conf_dir: ${cache_conf_dir} + backends: + - ${cache_be1} + - ${cache_be2} +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +fai_cache_install= +fai_cache_backends= +fai_cache_conf_dir= +fai_cache_port= + +#+end_src + +*** monitoring +#+begin_src yaml +profiles: + monitoring: + install: yes +#+end_src +=ece-install.conf= equivalent: +#+begin_src: text + +fai_monitoring_install= +#+end_src diff --git a/usr/share/doc/escenic/ece-install-guide.html b/usr/share/doc/escenic/ece-install-guide.html deleted file mode 100644 index 659f7c18..00000000 --- a/usr/share/doc/escenic/ece-install-guide.html +++ /dev/null @@ -1,853 +0,0 @@ - - - - -Welcome to the /usr/sbin/ece-install Guide - - - - - - - - - - - - -
    - - -

    Welcome to the /usr/sbin/ece-install Guide

    - - - -
    -

    1 Supported operating systems

    -
    - -

    For the best experience, run this script on Debian Squeeze (or newer), -the latest Ubuntu LTS or most other Debian based operating systems, -but it should work without any problems on any Unix like system that -has a recent version of BASH. -

    - -
    - -
    -

    1.1 Debian based operating systems

    -
    - -

    If running on a Debian based operating system, ece-install will take -care of installing all required third party software, such as database -& caching server. -

    -
    - -
    - -
    -

    1.2 Other GNU/Linux and Unix systems

    -
    - -

    If you're installing on another Linux or Unix system, you must first -make sure that these 3rd party components are installed. ece-install -will tell you which ones it cannot find and tell you to install these, -so you can simply run it to find out which ones you need. -

    -

    -You may also look inside the ece-install script for all calls to the -'assert_pre_prequesite' method and you'll get a list of the binaries -ece-install assumes are present. -

    -
    -
    - -
    - -
    -

    2 A note on running ece-install on non-GNU/Linux systems

    -
    - -

    Please note, it's assumed that the OS uses GNU's tools, i.e. find, cp -& tar. If you're running a system which has a different set of core -tools, such as any of the BSDs (including Mac OS X) or Solaris, make -sure that GNU's tools take precedence in the command PATH offered to -ece-install. -

    -
    - -
    - -
    -

    3 Downloading the ece-install script

    -
    - -

    The ece-install script can be downloaded from: -https://raw.github.com/skybert/ece-scripts/master/usr/sbin/ece-install -or be downloaded together with the other ece-scripts using git: -

    - - -
    $ git clone git@github.com:skybert/ece-scripts.git
-
-
    - - - - -
    - -
    - -
    -

    4 Running the ece-install script

    -
    - -

    You must run the script as the root user. If you start the script as -a regular user, it will complain: -

    - - -
    [ece-install] You must be root when running ece-install
-
-
    - - - -

    -To become root on Ubuntu based systems and Mac OS X, do: -

    - - -
    $ sudo su
-
-
    - - - -

    -On all other Unix like system, do: -

    - - -
    $ su
-
-
    - - - -
    - -
    - -
    -

    5 Available server profiles

    -
    - -

    When starting the script, it will ask you which server profile you -want to install: -

    - - -
    Hi, which server profile do you wish to install?
-
-Select 1-7 and press ENTER
-
-  1 - The full stack on one host. Suitable for development and
-      test environments (cache, ECE, assembly host & database).
-  2 - Editorial (publication) server (ECE, assembly host).
-  3 - Presentation server (ECE, memcached).
-  4 - Database server.
-  5 - Cache server (cache & web server).
-  6 - RMI hub.
-  7 - Standalone search instance (solr + indexer-webapp).
-  8 - Install Widget Framework. This will also set up a WF based 
-      publication for you (repo.escenic.com user/pass required).
-
-
    - - - - -
    - -
    -

    5.1 Common for editorial, presentation and search instances

    -
    - -
      -
    • Sun Java 6 -
      • -
    • Apache Tomcat application server: -
        -
      • Compression of textual content (previously, this was typically set - up with Apache and its mod_deflate module). -
        • -
      • pooling set up tweaked for high read/write performance. -
        • -
      • proper logging configuration directing solr messages to its own log. -
        • -
      • routing information in the cookies -
        • -
      • access log - -
        • -
      - -
      • -
    • APR, native library for optimal Tomcat I/O performance -
      • -
    • Escenic Assembly environment -
      • -
    • Database driver - -
      • -
    • Compression of content (what was previously accomplished with - Apache/mod_deflate) -
      • -
    - - -
    - -
    - -
    -

    5.2 1 - Full stack on one host

    -
    - -

    This profile is suitable for developers and admins wanting to set up a -test environment. It will install the full stack including caching -server, application server, ECE, assembly host, database, Widget -Framework, as well as creating a publication for you. -

    -

    -For further details on each of the different server components, see -the different profile descriptions bellow. -

    - -
    - -
    - -
    -

    5.3 2 - Editorial server (publication server)

    -
    - - - -
    - -
    - -
    -

    5.4 3 - Presentation server

    -
    - -

    This will set up a typical presentation server: -

      -
    • Memcached, distributed memory cache -
      • -
    • Escenic Assembly environment (which may be removed after the - installation). -
      • -
    • Deployment setup to only deploy escenic-admin and the - publication(s). -
      • -
    - - -
    - -
    - -
    -

    5.5 4 - Database server

    -
    - -

    If ece-install is run on a support version of Debian or Ubuntu, this -will install the excellent Percona distribution of MySQL with their -set of high performance patches. -

    -

    -If not, MySQL or Percona must be installed in advance. -

    -

    -Given that the mysqld is install, this profile will download all the -Escenic components and install the ECE database schema based from the -SQL files contained inside the distribution bundles. To accomplish -this, the script will make a call to drop-and-create-ecedb in the same -directory as the ece-intall script itself. -

    -

    -If you wish to change the DB's host, user or password, you must update -the drop-and-create-ecedb script prior to running ece-install. -

    -
    - -
    - -
    -

    5.6 5 - Cache server

    -
    - -

    If ece-install is run on a support version of Debian or Ubuntu, it -will install the latest Varnish 3.0 caching server from the Varnish -APT repository. -

    -

    -If ece-install is run on a different platform, the admin must install -Varnish 3.x prior to running ece-install. -

    -

    -The script will configure Varnish for a typical Escenic site: -

      -
    • will set up an access control lists of IPs which may access the - privileged web applications such as /escenic-admin, /escenic and - /webservice. - -

      - ece-install will also add the host from which you connect, making - sure that if you've SSH-ed into the server to conduct the install, - you'll automatically be included in the "staff" ACL and can access - all the web applications without editing these ACLs (or disabling - security as many does). -

      -
      • -
    • will set up sticky sessions/session binding -
      • -
    • will set up a backend cluster and allow the user to enter the - different backend servers that will serve the web site. -
      • -
    • will set up configuration to strip away cookies from static - resources, such as CSS, JS and pictures. -
      • -
    - - -

    -TBD: -

      -
    • If run on a Linux platform, the script will tweak the kernel - parameters for optimal TCP handling for a web facing server. -
      • -
    • Will install the nginx web server for serving static content and - will configure Varnish accordingly. This will be very useful for - Adactus servers wanting to pull content from your ECEs. -
      • -
    - - -
    - -
    - -
    -

    5.7 8 - Install Widget Framework

    -
    - -

    You'll need a user name and password for accessing the -repo.escenic.com Maven repository. You should get these credentials -when you bought Widget Framework. If you for some reason do not have -these, please contact support@escenic.com. -

    -

    -If you don't have these ready in your .escenicrc, ece-install will -complain: -

    - - -
    [ece-install] Be sure to set wf_user and wf_password in /root/.escenicrc
-[ece-install] If you don't have these, please contact support@escenic.com
-
-
    - - - -
    - -
    - -
    -

    5.8 9 - Create Publication

    -
    - -

    This profile will create a publication for you, only asking you the -name of the publication and which ECE instance to use for its -creation. -

    -

    -This installation profile will base the publication on the Widget -Framework if its present on the system, if not, ECE's clean demo WAR -is used as a basis. -

    -
    - -
    - -
    -

    6 Full Automatic Install (FAI)

    -
    - -

    The ece-install script has support for doing a full automatic install -(FAI). -

    -

    -The ece-install script understands for the following settings in the -$HOME/.ece-install.conf file of the root user: -

    - - -- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    ParameterDefaultDescription
    fai_all_install0Install all components on your server.
    fai_cache_install0Install cache server profile
    fai_cache_backends${HOSTNAME}:8080Space separated, e.g. "app1:8080 app2:8080"
    fai_db_install0Install db profile
    fai_db_host$HOSTNAMEUseful for editor & presentation profiles
    fai_db_port3306Useful for editor & presentation profiles
    fai_editor_install0Install the editorial profile
    fai_editor_nameeditor1Name of the editor instance
    fai_editor_port8080HTTP port of the editor instance
    fai_editor_shutdown8005Shutdown port of the editor instance
    fai_enabled0Whether or not to run ece-install in FAI mode
    fai_presentation_install0Install the presentation server profile
    fai_presentation_nameweb1Name of the presentation server instance
    fai_presentation_port8080HTTP port of the presentation server instance
    fai_presentation_shutdown8005Shutdown port of the presentation instance
    fai_publication_create0Create a new publication
    fai_publication_namemypubName of the publication
    fai_publication_useinstancedev1Name of local instance to use for creation
    fai_rmi_install0Install RMI hub profile
    fai_search_namesearch1Name of the search instance
    fai_search_port8080HTTP port of the search instance
    fai_search_shutdown8005Shutdown port of the search instance
    fai_wf_install0Install Widget Framework profile
    - - -

    -As you've probably have guessed, 0 means "false" and 1 means "true" :-) -

    -

    -To automatically install an editorial server and create a publication -called "jollygood", the minimal configuration in .ece-install.conf -would be: -

    - - - -
    fai_enabled=1
-fai_editor_install=1
-fai_publication_create=1
-fai_publication_name=jollygood
-
-
    - - - - -

    -When FAI is enabled, ece-install will report: -

    - - -
    [ece-install] Full Automatic Install (FAI) enabled.
-[ece-install] All user input will be read from /root/.ece-install.conf
-
-
    - - - - -
    - -
    - -
    -

    7 Running more than one installation process

    -
    - -

    If the script believes there's already an ece-intall process running, -it will abort: -

    - - -
    [ece-install] There's already one ece-install process running. If you believe
-[ece-install] this is wrong, e.g. if a previous run of ece-install was aborted
-[ece-install] before it completed, you may delete /var/run/ece-install.pid and
-[ece-install] run ece-install again.
-
-
    - - - - -
    - -
    - -
    -

    8 Re-running ece-install

    -
    - -

    Although the initial thought behind ece-install, is to run it on a -clean system to get up and running as soon as possible. However, you -may want to re-run ece-install on the same host, for instance to add -another instance of ECE, set up Widget Framework or create another -publication. -

    -

    -ece-install has a number of features which will try to minimise the -time it takes to run it on consecutive runs. If running on Debian -based systems, it will check if you already have installed -pre-requisite 3rd party libraries and only if any are missing will it -ask the package manager to fetch it. -

    -

    -Likewise, ece-install will see if the Escenic artifacts or application -server that you need are already present in the /tmp/ece-downloads -folder, and only download the missing ones (if any). -

    -

    -To get a list of the artifacts it'll pull from -http://technet.escenic.com and http://tomcat.apache.org search for the -following variables: -

      -
    • technet_download_list -
      • -
    • wf_download_list -
      • -
    • tomcat_download -
      • -
    - - -
    - -
    - -
    -

    9 Overview of file paths used by the ece-install script

    -
    - -

    There are of course other paths involved when setting up your system, -but these should be the most interesting ones. -

    - - -- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    PathExplanation
    /etc/apt/sources.list.d/escenic.list3rd party APT repositories added by ece-install *)
    /etc/default/eceThe configuration file for the ece init.d script
    /etc/escenic/ece-<instance>.confInstance specific settings for /usr/bin/ece
    /etc/escenic/ece.confCommon ece.conf file for /usr/bin/ece
    /etc/escenic/engine/commonCommon Nursery configuration layer
    /etc/escenic/engine/common/securityCommon security configuration for all ECE instances.
    /etc/escenic/engine/common/trace.propertiesLog4j configuration, produces instance specific log files.
    /etc/escenic/engine/instance/<instance>Instance specific Nursery configuration
    /etc/escenic/solrECE specific Solr configuration
    /etc/init.d/mysql[d]For starting and stopping MySQL/Percona
    /etc/init.d/varnishFor starting and stopping Varnish
    /etc/intit.d/eceThe init.d script managing all the ECE instances on your host.
    /etc/varnish/default.vclThe Varnish configuration
    /opt/escenicAll ECE components can be found here
    /opt/escenic/assemblytoolThe assembly tool
    /opt/escenic/assemblytool/pluginsContains symlinks to all plugins in /opt/escenic
    /opt/escenic/engineSymlink pointing to the current ECE
    /opt/tomcatSymlink pointing to the install Apache Tomcat (catalina_home)
    /opt/tomcat-<instance>Instance specific Tomcat files (catalina_base)
    /usr/bin/eceScript for operating all ECE instances + RMI hub and EAE
    /usr/sbin/drop-and-create-ecedbDB script used by the ece-install script
    /usr/sbin/ece-installThe installation script described in this guide
    /var/log/escenic/<type>-<instance>.logThe instance's log4j log
    /var/log/escenic/<type>-<instance>.outThe instance system out log
    /var/log/escenic/solr.<date>.logThe Solr log (not in standard out!)
    /var/run/escenic/<type>-<instance>.pidThe instance's PID file
    - - -

    -*) Applies only to Debian based systems. -

    -
    - -
    - -
    -

    10 Uninstalling everything that the ece-install set up

    -
    - -

    WARNING: this is potentially dangerous as some of these components may -be used by other pieces of software you have running on your -host. However, this may be useful if you're installing a clean -environment and want to e.g. undo your previous install to install a -different profile. -

    -

    -Open the ece-install script and look for the "un_install_ece" -function, it has copy and pastable commands for undoing most/all -things set up by the script. -

    - -
    - -
    - -
    -

    11 Example Output

    -
    - -

    Below is an example output of running ece-install, installing the -all-in-one profile. -

    -

    -CANNOT INCLUDE FILE /tmp/ece-install.out -

    -
    -
    -
    -
    -

    - Author: Torstein Krause Johansen (tkj@vizrt.com) - Date: 2011-08-25 22:02:19 CST -

    -
    - - diff --git a/usr/share/doc/escenic/ece-install-guide.org b/usr/share/doc/escenic/ece-install-guide.org index 9cd730a5..93d51de1 100644 --- a/usr/share/doc/escenic/ece-install-guide.org +++ b/usr/share/doc/escenic/ece-install-guide.org @@ -1,320 +1,1379 @@ -Welcome to the /usr/sbin/ece-install Guide - -* Supported operating systems -For the best experience, run this script on Debian Squeeze (or newer), -the latest Ubuntu LTS or most other Debian based operating systems, -but it should work without any problems on any Unix like system that -has a recent version of BASH. - -*** Debian based operating systems -If running on a Debian based operating system, ece-install will take -care of installing all required third party software, such as database -& caching server. - -*** Other GNU/Linux and Unix systems -If you're installing on another Linux or Unix system, you must first -make sure that these 3rd party components are installed. ece-install -will tell you which ones it cannot find and tell you to install these, -so you can simply run it to find out which ones you need. - -You may also look inside the ece-install script for all calls to the -'assert\_pre\_prequesite' method and you'll get a list of the binaries -ece-install assumes are present. - -* A note on running ece-install on non-GNU/Linux systems -Please note, it's assumed that the OS uses GNU's tools, i.e. find, cp -& tar. If you're running a system which has a different set of core -tools, such as any of the BSDs (including Mac OS X) or Solaris, make -sure that GNU's tools take precedence in the command PATH offered to -ece-install. - -* Downloading the ece-install script -The ece-install script can be downloaded from: -https://raw.github.com/skybert/ece-scripts/master/usr/sbin/ece-install -or be downloaded together with the other ece-scripts using git: -#+BEGIN_SRC sh -$ git clone git@github.com:skybert/ece-scripts.git -#+END_SRC -* Running the ece-install script -You must run the script as the root user. If you start the script as -a regular user, it will complain: -#+BEGIN_SRC sh +#+TITLE: Escenic Installer User Guide +#+AUTHOR: Escenic Cloud Team +#+OPTIONS: H:6 num:5 toc:2 + +* NAME +ece-install - command for installing any server in a single- or +multi-server environment, with or without Escenic software. + +* SYNOPSIS +ece-install [[[-f|--file conf-file]]] [[[-V|--version]]] [[[-v|--verbose]]] + +* OPTIONS +** -f|--file conf-file +Instructs *ece-install* to read configuration data from the specified +file. This can either be a `.conf` or `.yaml` file. + +** -v|--verbose +Requests verbose output from *ece-install*. + +** -V|--version +Prints the version of *ece-install* + + +* DESCRIPTION +The Escenic Content Engine Installer, *ece-install* is a command line +utility for installing the Escenic Content Engine. In addition to +installing the Content Engine itself, *ece-install* also automates the +installation and configuration of other commonly-used components of +Escenic-based sites, such as: + + - Java + - Tomcat application server + - MariaDB/MySQL/Percona database server + - Varnish cache server + - Icinga & Munin monitoring server + - NFS server and clients + - Memcached distributed memory cache + +You can use *ece-install* to install The Content Engine and related +components on a wide range of GNU Linux distributions and UNIX or +UNIX-like operating systems, but the level of support varies between +platforms. On a fully-supported operating system (for example Debian 6.0), +*ece-install* can both install and configure all required +components, so that the entire process is reduced to a simple sequence +of *ece-install* commands. + +On less well-supported platforms, *ece-install* may not be able to +automatically install some third-party components. You will then need to +manually install some components. However, the process is very +straightforward, since *ece-install* tells you which components you +need to install manually, and it still handles the configuration of +these components for you. + +** Supported Operating Systems +| Operating system | Status | +|-----------------------------+------------------------------| +| Debian 8.7 (jessie) | Everything is 100% automatic | +| Ubuntu LTS 16.04 | Everything is 100% automatic | +| RedHat Enterprise Linux 7.3 | Everything is 100% automatic | +| CentOS 7.3 | Everything is 100% automatic | + +Other Unix operating systems may work, given that the required +binaries (=curl=, =varnishd=, =mysqld=++) are already available in +PATH and that the configuration files exist in familiar +locations. However this is untested. YMMV. + +** Requirements +Whichever operating system(s) you will be installing on, you must ensure +that the following requirements are met: + + - Sufficient memory :: The precise amount of memory required by + *ece-install* depends on what components are to be installed. To + successfully install a complete software stack on one machine, + you need at least 3GB of RAM. If you are installing on a + virtual machine it is important to remember this requirement + when creating the virtual machine. + + - Internet connection :: *ece-install* needs an Internet + connection. Ideally it should be a fast connection, since it + will need to download several hundred megabytes of data. + + - Firewall rules :: Your firewall settings must allow *ece-install* + to make outgoing connections from the machine on + which it is running to the following ports: + - 80 (standard HTTP) + - 443 (HTTPS) + +****** On non-GNU/Linux systems +*ece-install* relies on the GNU versions of various tools such as *find*, *cp* +and *tar*. If you use *ece-install* on any system which provides different +versions of these tools by default, then you must make sure that: + +- The GNU tools are installed on the system +- That the GNU tools take precedence in the command PATH offered + to *ece-install*. + +Platforms on which you will need to do this include: + + - Any BSD-based system (including Mac OS X) + - Solaris + + +* Using ece-install +The various software components required to run an Escenic-powered web +site are usually installed on a cluster of servers, with different +components installed on each server according to the +function it will perform. The set of components installed on a +particular server is called its *profile*. + +The overall procedure for installing Escenic on a cluster of machines +is as follows: + + - Determine the profile of each server in the cluster. For + descriptions of all available profiles, see Server Profiles. + - For each server: + * Log on to the server. + * Install the *ece-install* command (see Get ece-install). + * Become the root user (see Become Root). + * Create an *ece-install* configuration file (see Create a + Configuration File). + * Run the *ece-install* command, specifying the appropriate + profile (see Run ece-install). + +** <<>> +On Debian based systems, such as Ubuntu, you can use the [[http://apt.escenic.com][Escenic APT +repository]] and install the package *escenic-content-engine-installer* +to get the *ece-install* command. + +On RedHat based systems, such as CentOS, you can use the RPMs +available from http://yum.escenic.com/rpm and install the RPM +*escenic-common-scripts* and *escenic-content-engine-installer* to +get the *ece-install* command. + +If you have any problems installing using the DEB or RPM packages, or +if you're system is neither Debian nor RedHat based, you can also +download a ZIP archive of all the *ece-scripts*, which *ece-install* +is a part of, here: +https://github.com/escenic/ece-scripts/zipball/master + +** <<>> +You must be the root user to run *ece-install*. If you try to run it as +an ordinary user, it will complain: +#+BEGIN_SRC text [ece-install] You must be root when running ece-install #+END_SRC -To become root on Ubuntu based systems and Mac OS X, do: -#+BEGIN_SRC sh - $ sudo su + +To become root on Ubuntu-based systems and Mac OS X, enter: +#+BEGIN_SRC text +$ sudo su #+END_SRC -On all other Unix like system, do: -#+BEGIN_SRC sh - $ su + +On all other UNIX-like system, enter: +#+BEGIN_SRC text +$ su - #+END_SRC -* Available server profiles -When starting the script, it will ask you which server profile you -want to install: -#+BEGIN_SRC sh -Hi, which server profile do you wish to install? - -Select 1-7 and press ENTER - - 1 - The full stack on one host. Suitable for development and - test environments (cache, ECE, assembly host & database). - 2 - Editorial (publication) server (ECE, assembly host). - 3 - Presentation server (ECE, memcached). - 4 - Database server. - 5 - Cache server (cache & web server). - 6 - RMI hub. - 7 - Standalone search instance (solr + indexer-webapp). - 8 - Install Widget Framework. This will also set up a WF based - publication for you (repo.escenic.com user/pass required). -#+END_SRC -** Common for editorial, presentation and search instances -- Sun Java 6 -- Apache Tomcat application server: - * Compression of textual content (previously, this was typically set - up with Apache and its mod\_deflate module). - * pooling set up tweaked for high read/write performance. - * proper logging configuration directing solr messages to its own log. - * routing information in the cookies - * access log - -- APR, native library for optimal Tomcat I/O performance -- Escenic Assembly environment -- Database driver - -- Compression of content (what was previously accomplished with - Apache/mod\_deflate) - -** 1 - Full stack on one host -This profile is suitable for developers and admins wanting to set up a -test environment. It will install the full stack including caching -server, application server, ECE, assembly host, database, Widget -Framework, as well as creating a publication for you. - -For further details on each of the different server components, see -the different profile descriptions bellow. - - -** 2 - Editorial server (publication server) - - -** 3 - Presentation server -This will set up a typical presentation server: -- Memcached, distributed memory cache -- Escenic Assembly environment (which may be removed after the - installation). -- Deployment setup to only deploy escenic-admin and the - publication(s). - -** 4 - Database server -If ece-install is run on a support version of Debian or Ubuntu, this -will install the excellent Percona distribution of MySQL with their -set of high performance patches. - -If not, MySQL or Percona must be installed in advance. - -Given that the mysqld is install, this profile will download all the -Escenic components and install the ECE database schema based from the -SQL files contained inside the distribution bundles. To accomplish -this, the script will make a call to drop-and-create-ecedb in the same -directory as the ece-intall script itself. - -If you wish to change the DB's host, user or password, you must update -the drop-and-create-ecedb script prior to running ece-install. - -** 5 - Cache server -If ece-install is run on a support version of Debian or Ubuntu, it -will install the latest Varnish 3.0 caching server from the Varnish -APT repository. - -If ece-install is run on a different platform, the admin must install -Varnish 3.x prior to running ece-install. - -The script will configure Varnish for a typical Escenic site: -- will set up an access control lists of IPs which may access the - privileged web applications such as /escenic-admin, /escenic and - /webservice. - - ece-install will also add the host from which you connect, making - sure that if you've SSH-ed into the server to conduct the install, - you'll automatically be included in the "staff" ACL and can access - all the web applications without editing these ACLs (or disabling - security as many does). - -- will set up sticky sessions/session binding -- will set up a backend cluster and allow the user to enter the - different backend servers that will serve the web site. -- will set up configuration to strip away cookies from static - resources, such as CSS, JS and pictures. +#+BEGIN_QUOTE +Note that using *sudo* to run *ece-install* (that is, entering "*sudo +ece-install*") may not work. You should actually become the root user +before running *ece-install*. +#+END_QUOTE + +** <<>> +*ece-install* can use configuration files in either =.conf= (old) or +*=.yaml= (new) format. This manual describes first and foremost the +*YAML format. + +*ece-install.yaml*. may contain a large number of configuration +parameters, but the minimum requirement is that it contains: + + * A user name and password for downloading software from Escenic. + * A profile to install + +For example: +#+BEGIN_SRC yaml +credentials: + - site: maven.escenic.com + user: foo + password: bar + - site: apt.escenic.com + user: foo + password: bar +profiles: + editor: + install: yes +#+END_SRC + +Depending on what components you are installing on the server, you may +need to include other configuration parameters in the file. In most +cases, however, parameters have default settings that enable +*ece-install* to complete installation with very few settings. + +** <<>> +To run *ece-install*, enter: +#+BEGIN_SRC text +# ece-install +#+END_SRC + +*ece-install* writes a log file located at +*/var/log/escenic/ece-install.log*. All output generated by all the commands +it executes is written to this file. You can use *tail* to keep an eye +on what is being written to the log. + +*ece-install* tries to "fail fast", exiting as soon as it detects an +error and reporting the failure. For example: +#+BEGIN_SRC text +[ece-install-1] The command [cd /not/there] FAILED, exiting :-( +[ece-install-1] See /var/log/escenic/ece-install.log for further details. +#+END_SRC + +If you run into problems and the log file does not provide enough +clues about what is going wrong, the best debugging method is to run +the BASH interpreter with the -x flag: +#+BEGIN_SRC text +# bash -x ece-install +#+END_SRC + +Doing this lets you see everything that BASH does while executing the +command - how wild card variables are expanded and so on. + +*ece-install* displays a series of progress messages during the +installation process. you can redirect standard output to a log file +for easy reading of these messages later: +#+BEGIN_SRC text +# bash ece-install > ece-install.out +#+END_SRC + +If you are logged into the host via SSH, you can make it possible to +log out and leave *ece-install* running in the background by adding +*nohup* at the start of the command and an ampersand at the end, as follows: +#+BEGIN_SRC text +# nohup bash ece-install > ece-install.out & +#+END_SRC + +****** After Installing +When installation is completed an information message is +displayed. This contains important information about what you +should do next, plus references to where you can find useful +information, so you should read it carefully. + +You should now set a password for the user *escenic* (which has been +created for you by *ece-install*). To do this enter: + +#+BEGIN_SRC text +# passwd escenic +#+END_SRC + +The *escenic* user is the user you will need to use for most +escenic-related purposes. + +** Preventing accidental execution of ece-install +You can ensure that *ece-install* is not executed accidentally by +creating a *lock file*. Simply create a file with this path: +#+BEGIN_SRC text +/var/run/escenic/ece-install.lock +#+END_SRC + +If this file is present then *ece-install* will fail fast as follows: +#+BEGIN_SRC text +/var/run/escenic/ece-install.lock exists, I'll exit. +#+END_SRC + +The lock file does not need to contain anything, it just needs to exist. + +* <<>> + +*ece-install* installs *profiles*. A profile is a pre-defined set of +software components that enables a host computer to play a specific +role in an Escenic installation. + +The profile to be installed by *ece-install* is determined by setting +one of the following parameters in *ece-install.conf*: + +- *profiles: editor* :: Installs all the components that need to + be installed to create an Escenic *editorial server*. +- *profiles: presentation* :: Installs all the components that + need to be installed to create an Escenic *presentation server*. +- *profiles: db* :: Installs a Database Server. +- *profiles: cache* :: Installs a Cache Server. +- *profiles: search* :: Installs a Search Server. +- *profiles: publication* :: Creates an Escenic publication. +- *profiles: all* :: Installs all of the above on one host + machine. +- *profiles: restorebackup* :: Restores a backup created with *ece*. +- *profiles: analysis* :: Installs and configures an Escenic Analysis Engine. +- *profiles: nfs* :: Installs an NFS server. +- *profiles: nfs* :: Installs an NFS client. +- *profiles: vip* :: Installs virtual IP (VIP) providers. + +There are a number of common components that are included in all or +most of the profiles. These are described below. + +** <<>> +To use this profile add: +#+begin_src yaml +profiles: + editor: + create: true +#+end_src + +to your *ece-install.yaml* file. + +This profile contains all the components that need to be installed to +create an Escenic *editorial server*. An editorial server (sometimes +also called a publication server) hosts a Content Engine used for editorial +purposes (primarily Content Studio sessions). + +** <<>> +To use this profile, add: +#+begin_src yaml +profiles: + presentation: + create: true +#+end_src + +to your *ece-install.yaml* file. + +This profile contains all the components that need to be installed to +create an Escenic *presentation server*. A presentation server hosts a +Content Engine used for serving publications to web site +users. Differences between this and an editorial server include: + + * Only the Escenic administration web-app *escenic-admin* and publications are + deployed. Other editorial web-apps such as Web Studio are not + required. + * Memcached, the distributed memory cache is installed. + +** <<>> +To use this profile, add: + +#+begin_src yaml +profiles: + db: + create: true +#+end_src + +to your *ece-install.yaml* file. + +When this profile is used on a supported version of Debian, Ubuntu, +CentOS or RedHat, *ece-install* installs will install all the +components needed to create an Escenic database, based on the MariaDB +fork of MySQL or the Percona distribution of MySQL. + +The default is to install MariaDB. + +Furthermore, this profile contains all the Escenic components needed +on a database server plus the correct database schema for the Content +Engine version and plug-in versions you are installing. + +If an Escenic database has already been installed on the machine, then +*ece-install* will fail and display an information message: + +#+BEGIN_SRC text +[ece-install] Setting up the ECE database schema ... + ERROR 1007 (HY000) at line 1: Can't create database 'ece5db'; + database exists + ERROR 1050 (42S01) at line 2: Table 'DBChangeLog' already + exists +[ece-install] running tables FAILED, exiting :-( +#+END_SRC + +If you actually want to re-install the database you can do so by +adding this setting to *ece-install.yaml* before you run *ece-install*: + +#+BEGIN_SRC yaml +profiles: + db: + install: true + drop_old_db_first: true +#+END_SRC + +Given that *mysqld* is installed, this profile will download all the +Escenic components that you have defined in the configuration file and +install the ECE database schema based from the SQL files contained +inside the individual packages. + +*** Master & slave setup +You can easily use *ece-install* to set up master and slave databases +on different hosts. + +First create the master database using these *ece-install.yaml* settings: +#+BEGIN_SRC yaml +profiles: + db: + install: true + master: true + replication: true +#+END_SRC + +When you run *ece-install* with these settings, the output log +messages will include information that you need for creating the slave +database: +#+BEGIN_SRC text +[ece-install-35] - DB is now set up on localhost:3306 +[ece-install-35] - ece-install.conf for slave: + fai_db_master_log_file=mysql-bin.000013 +[ece-install-35] - ece-install.conf for slave: + fai_db_master_log_position=106 +#+END_SRC + +On the slave database host you can then use these values in your +*ece-install.yaml* file as follows: +#+BEGIN_SRC yaml +profiles: + db: + install: true + master: false + replication: true + master_host: "my-db-master" + master_log_file: "mysql-bin.000013" + master_log_position: "106" +#+END_SRC + +*ece-install* uses internal defaults to create a replication user and +credentials. You can override these defaults by setting additional +parameters in *ece-install.conf*. For details, see [[Overview of All FAI +Parameters]]. + +** <<>> +To use this profile, add: + +#+begin_src yaml +profiles: + cache: + install: true +#+end_src + + +to your *ece-install.yaml* file. + +When this profile is used on a supported version of Debian or Ubuntu, +*ece-install* installs the latest Varnish caching server from the +official distro repositories. + +Once Varnish is installed, *ece-install* configures it to suit the +typical requirements of an Escenic site: + + * Sets up the cache server on port 80 + * Creates an access control list (ACL) of IP addresses allowed to access + privileged web applications such as */escenic-admin*, */escenic* and + */webservice*. If you are running *ece-install* in an SSH session, + then it includes the IP address from which you connected in the ACL + so that you can access these applications without needing to + manually edit the ACL or disable security. + * Sets up sticky sessions/session binding + * Sets up a back-end cluster for balancing web site requests to the + cache server. + * Sets up configuration that strips cookies from static resources, + such as CSS files, JS files and images. + * Installs the *nginx* web server for serving static content and + configures Varnish accordingly. This is particularly useful for + installations where VME Online servers need to access content. + +#+BEGIN_COMMENT TBD: -- If run on a Linux platform, the script will tweak the kernel +- If run on a Linux platform, the command will tweak the kernel parameters for optimal TCP handling for a web facing server. -- Will install the nginx web server for serving static content and - will configure Varnish accordingly. This will be very useful for - Adactus servers wanting to pull content from your ECEs. - -** 8 - Install Widget Framework -You'll need a user name and password for accessing the -repo.escenic.com Maven repository. You should get these credentials -when you bought Widget Framework. If you for some reason do not have -these, please contact support@escenic.com. - -If you don't have these ready in your .escenicrc, ece-install will -complain: +- let the /munin run through on port 80, requiring the connecting IPs + to be in the staff network ACL, defined in the Varnish + configuration. +#+END_COMMENT + +** <<>> +To use this profile, add + +#+begin_src yaml +profiles: + search: + install: true +#+end_src + + +to your *ece-install.yaml* file. + +This profile installs a standalone Solr instance and an app server +with the Escenic *indexer* web app. + +** <<>> +To use this profile, add + +#+begin_src yaml +profiles: + publications: + - name: mypub + create: true +#+end_src + + +to your *ece-install.yaml* file. + +This profile creates a publication for you. It should be used on a +machine where you have already installed either an editorial or +presentation profile. + +*ece-install* will create publications of all the publication WAR +files in your EAR file if you also +define domain (and optionally, aliases): +#+BEGIN_SRC yaml +profiles: + publications: + - name: stoppok + domain: stoppok.example.com + aliases: + - s.exampel.com + - o.exampel.com + ear: /tmp/stoppok-rev6195-2012-12-04_1134.ear + - name: helden + domain: helden.example.com + ear: /tmp/stoppok-rev6195-2012-12-04_1134.ear +#+END_SRC + +See [[Overview of All Configuration Parameters]] for further details on the format +of the domain mapping list. + +** <<>> +To use this profile, add: + +#+begin_src yaml +profiles: + restore: + from_file: /var/backups/backup.tar.gz +#+end_src + +to your *ece-install.yaml* file. + +Unlike all the other profiles, this profile does not install anything +or create anything new. Instead, it restores a backup you have +previously created using the *ece* command - like this, for example: +#+BEGIN_SRC text +$ ece -i backup +#+END_SRC + +Exactly what such a backup contains depends on: + + * What was present on the host machine where the backup was created + * What options were specified when the backup was created. + +It may, however, contain: + + * The Escenic software components (Content Engine etc.) installed on the host. + * Content Engine, cache and web server configuration data. + * A database dump. + * An Escenic multimedia archive (images, video files and so on). + +You can use this profile in two ways: + + * To restore a host to an earlier state. + * To install a copy of some other installation on a "clean" host. + +In order to use this profile you have to set some additional +parameters in *ece-install.conf* in order to specify the location of +the backup file you want to restore and the specific items you want to +restore from the file. + +The parameters you can use together with *profiles: restore* +to specify what you want to restore are: + +#+begin_src yaml +profiles: + restore: + # The *.tar* file produced by *ece -i backup*. Default is "". + from_file: + from_backup: true + + # Restore the Solr and Content Engine data files. Default is false. + data_files: true + + # Restore the Escenic and Apache Tomcat software. Default is false. + software_binaries: true + + # Install the database server and restore its. Default false + db: true + + # Restore the Solr and content Engine configuration files. Default is false + configuration: true +#+end_src + + +*ece-install* can also remove unwanted files from an existing +installation prior to restoring from a backup. You can specify the +files you would like to remove using the following parameters: + +#+begin_src yaml +profiles: + restore: + # Remove the solr data/state files. Default is false. + pre_wipe_solr: true + # Remove all log files. Default is false. + pre_wipe_logs: true + # Remove the cache files. Default is false. + pre_wipe_cache: true + # Remove the crash files. Default is false. + pre_wipe_crash: true + + # Remove all of the above + pre_wipe_all: true +#+end_src + +*** Data security +You must be careful when restoring backups that you don't +inadvertently restore the backup over a system that actually contains +valuable data. *ece-install* incorporates some safeguards, but +ultimately cannot prevent you from making such mistakes. + +If you try to restore the DB and the ECE schema already exists, the +restore will fail as follows: +#+BEGIN_SRC text +[ece-install-8] Restoring the database contents on ubiquitous ... +[ece-install-24] Selecting the most recent database dump + ece5db-2011-10-10.sql.gz + ERROR 1007 (HY000) at line 1: Can't create database 'ece5db'; + database exists + ERROR 1050 (42S01) at line 25: Table + '`ece5db`.`AccessControlList`' already exists +[ece-install-24] The command [restoring from + /var/backups/escenic/ece5db-2011-10-10.sql.gz] FAILED, + exiting :-( +[ece-install-24] See /var/log/escenic/ece-install.log for further + details. +#+END_SRC + +** <<>> +To use this profile, add: + +#+begin_src yaml +profiles: + analysis: + create: true +#+end_src + +to your *ece-install.yaml* file. + +This profile installs the Escenic Analysis Engine, and configures it +for production use with a sensible set of defaults. + +The Analysis Engine uses a database to store statistics. You must not +use the same database as is used by the Content Engine for storing +publication contents. + +** <<>> +To use this profile, add *fai\_nfs\_server\_install=1* to your +*ece-install.conf* file. + +This profile installs an NFS server. + +This feature is not yet supported with YAML configuration files. + +** <<>> +To use this profile, add *fai\_nfs\_client\_install=1* to your +*ece-install.conf* file. + +This profile installs an NFS client, creates the client mount points +and mounts them on the host. Per default, all network drives are +mount under */mnt*. + +The following example shows the ece-install.conf settings required to +mount the Escenic multimedia archive on the NFS server: +#+BEGIN_SRC conf +fai_nfs_client_install=1 +fai_nfs_server_address=192.168.1.200 +fai_nfs_export_list="/var/exports/multimedia" +#+END_SRC + +This feature is not yet supported with YAML configuration files. + +** <<>> +[[file:images/nfs-vip.png]] + +To use this profile, add *fai\_vip\_install=1* to your +*ece-install.conf* file. + +This profile is usually used in combination with one of the other +*ece-install* profiles. It makes a host capable of providing the +services it offers on specified virtual IP addresses (VIPs). This +makes it possible to provide fail-over for all single points of +failure ([[http://en.wikipedia.org/wiki/Single_point_of_failure][SPOFs)]] in your installation, such as the file server or +database. + +You might, for example, in order to provide a robust file +server, install both an NFS server and a VIP provider on two hosts: + + * The primary NFS server that provides the normal service on one host + * The secondary NFS server that takes over if the primary one fails + on the other. + +Installing VIP providers with the *fai\_vip\_install* profile +allows both servers to be accessed via the same virtual IP address, so +that a fail-over is invisible to users of the service. + +The following *ece-install.conf* settings installs an NFS server and +configures two VIP providers: + + * The primary node (this host) at 192.168.1.112 + * The secondary node at 192.168.1.111 + +Both providers are configured to expose the NFS service on the VIP +192.168.1.200. + +#+BEGIN_SRC conf +# install the NFS server +fai_nfs_server_install=1 +fai_nfs_export_list="/var/exports/multimedia" +fai_nfs_allowed_client_network="192.168.1.0/255.255.255.0" + +# install the VIP provider, primary node +fai_vip_install=1 +fai_vip_service_list="nfs-kernel-server" +fai_vip_primary_node_name=ubiquitous +fai_vip_primary_node_ip=192.168.1.112 +fai_vip_primary_node_auth_key=d41d8cd98f00b204e9800998ecf8427e +fai_vip_secondary_node_name=ubiquitous-lts +fai_vip_secondary_node_ip=192.168.1.111 +fai_vip_address=192.168.1.200 +fai_vip_sibling_ip=$fai_vip_secondary_node_ip +#+END_SRC + +The setting *fai\_vip\_sibling\_ip*$fai\_vip\_secondary\_node\_ip* says that +the secondary node is this node's sibling, and therefore implicitly +defines this node as the primary node. + +The secondary node can therefore be defined using an almost identical +configuration - only *fai\_vip\_sibling\_ip* needs to be set differently: +#+BEGIN_SRC conf +fai_vip_sibling_ip=$fai_vip_primary_node_ip +#+END_SRC + +The *fai\_vip\_primary\_node\_auth\_key* setting is optional. If you do not +set it, ece-install will generate it for you. However, you will then have to +add the generated key to *ece-install.conf* when installing the secondary +node. + +You can generate the key as follows: #+BEGIN_SRC sh -[ece-install] Be sure to set wf_user and wf_password in /root/.escenicrc -[ece-install] If you don't have these, please contact support@escenic.com -#+END_SRC -** 9 - Create Publication -This profile will create a publication for you, only asking you the -name of the publication and which ECE instance to use for its -creation. - -This installation profile will base the publication on the Widget -Framework if its present on the system, if not, ECE's clean demo WAR -is used as a basis. -* Full Automatic Install (FAI) -The ece-install script has support for doing a full automatic install -(FAI). - -The ece-install script understands for the following settings in the -$HOME/.ece-install.conf file of the root user: - -|--------------------------------+------------------+-----------------------------------------------| -| Parameter | Default | Description | -|--------------------------------+------------------+-----------------------------------------------| -| fai\_all\_install | 0 | Install all components on your server. | -| fai\_cache\_install | 0 | Install cache server profile | -| fai\_cache\_backends | ${HOSTNAME}:8080 | Space separated, e.g. "app1:8080 app2:8080" | -| fai\_db\_install | 0 | Install db profile | -| fai\_db\_host | $HOSTNAME | Useful for editor & presentation profiles | -| fai\_db\_port | 3306 | Useful for editor & presentation profiles | -| fai\_editor\_install | 0 | Install the editorial profile | -| fai\_editor\_name | editor1 | Name of the editor instance | -| fai\_editor\_port | 8080 | HTTP port of the editor instance | -| fai\_editor\_shutdown | 8005 | Shutdown port of the editor instance | -| fai\_enabled | 0 | Whether or not to run ece-install in FAI mode | -| fai\_presentation\_install | 0 | Install the presentation server profile | -| fai\_presentation\_name | web1 | Name of the presentation server instance | -| fai\_presentation\_port | 8080 | HTTP port of the presentation server instance | -| fai\_presentation\_shutdown | 8005 | Shutdown port of the presentation instance | -| fai\_publication\_create | 0 | Create a new publication | -| fai\_publication\_name | mypub | Name of the publication | -| fai\_publication\_use_instance | dev1 | Name of local instance to use for creation | -| fai\_rmi\_install | 0 | Install RMI hub profile | -| fai\_search\_name | search1 | Name of the search instance | -| fai\_search\_port | 8080 | HTTP port of the search instance | -| fai\_search\_shutdown | 8005 | Shutdown port of the search instance | -| fai\_wf\_install | 0 | Install Widget Framework profile | -|--------------------------------+------------------+-----------------------------------------------| - -As you've probably have guessed, 0 means "false" and 1 means "true" :-) +$ dd if=/dev/urandom bs=512 count=1 | \\ + openssl md5 | \\ + cut -d' ' -f2 +#+END_SRC + +** Installing from EARs instead of Binaries +It is possible to get *ece-install* to use a supplied EAR and +configuration archive instead of using the files provided with the +Escenic Content Engine and plugins. + +The EAR to provide is the one you generate with: +#+BEGIN_SRC text +$ ece -i assemble +#+END_SRC +Normally, the EAR will then be available in: +#+BEGIN_SRC conf +/var/cache/escenic/engine.ear +#+END_SRC + +The configuration bundle must contain: +#+BEGIN_SRC text +engine/security +engine/siteconfig/bootstrap-skeleton +engine/siteconfig/config-skeleton +assemblytool/plugins//siteconfig +#+END_SRC + +and optionally also: +#+BEGIN_SRC text +engine/solr/conf +#+END_SRC + +A simple way to create this bundle, is to use a server which has the +assembly environment set up and then do: + +#+BEGIN_SRC text +$ cd /opt/escenic +$ tar czf /tmp/config-skeleton.tar.gz \\ + engine/security \\ + engine/siteconfig/config-skeleton \\ + engine/solr/conf \\ + engine/siteconfig/bootstrap-skeleton +#+END_SRC + +*/tmp/nursery-skeleton-solr-and-security.tar.gz* should now have everything +you need. You can now configure your FAI installation to use these by, +e.g.: + +#+BEGIN_SRC conf +fai_presentation_ear=/tmp/engine.ear +fai_presentation_conf_archive=/tmp/config-skeleton.tar.gz +#+END_SRC + +Corresponding configuration options are available for the other server +profiles, see the table below. + +The inclusion of the engine/solr directory makes it easy for users to +provide their own, optimised Solr configuration. In this context, also +note that a post install hook, *set\_up\_solr.postinst*, is available. +If you want the EAR file to dictate the production environment, set up +the build environment to create a file META-INF/distributions.txt in +your EAR, and put the maven coordinates needed by the run-time in it. +This will tell ece-install what files to download. + +#+BEGIN_SRC text +com.escenic:engine-dist:zip:bin:5.5.2.123456 +com.escenic.plugins.someplugin:someplugin-dist:zip:1.2.134 +#+END_SRC + +In your ece-install.conf file, specify *fai_maven_repositories* as a +whitespace separated list of repositories that may hold these artifacts, +and ece-install will try each repository in order to download the +arifacts. + +#+BEGIN_SRC conf +fai_maven_repositories=" + http://example.org/nexus/content/groups/trunk + http://builder:secretpassword@mybuildserver.internal/trunk +" +#+END_SRC + +If you wish to provide Nursery configuration for the plugins, you +simply put them in engine/siteconfig/config-skeleton inside your +tarball, together with the other Nursery configuration files. + +The *fai\_presentation\_conf\_archive* and *fai\_presentation\_ear* variables +both accept the following types of value (here using the value of +*fai\_presentation\_ear* as an example): +- *http://build.server/stable/engine-mysite.com-1.2.3.ear* +- *https://build.server/stable/engine-mysite.com-1.2.3.ear* +- *file:///var/lib/build/stable/engine-mysite.com-1.2.3.ear* +- */var/lib/build/stable/engine-mysite.com-1.2.3.ear* + +** Setting up virtual hosts +Setting up virtual host definitions in the application server makes a +some things easier, such as ECE plugins which set cookies based on +information they get from the app server. + +ece-install can set up the virtual hosts configuration for Tomcat +application servers if the profile is *editor*, *all* or +*presentation*. + +To use this feature, you must define one domain for each publication +in the following FAI parameter: +#+BEGIN_SRC yaml +profiles: + publications: + - name: firepub + domain: fire.escenic.com + - name: ildpub + domain: ild.escenic.com + aliases: + - feuer.example.com + - fuego.example.com +#+END_SRC + +This will produce the following stanzas in *server.xml*: + +#+BEGIN_SRC nxml + + + + + feuer.escenic.com + fuego.escenic.com + + +#+END_SRC +If you wish to provide Nursery configuration for the plugins, you +simply put them in engine/siteconfig/config-skeleton inside your +tarball, together with the other Nursery configuration files. + +The *fai\_presentation\_conf\_archive* and *fai\_presentation\_ear* variables +both accept the following types of value (here using the value of +*fai\_presentation\_ear* as an example): +- *http://build.server/stable/engine-mysite.com-1.2.3.ear* +- *https://build.server/stable/engine-mysite.com-1.2.3.ear* +- *file:///var/lib/build/stable/engine-mysite.com-1.2.3.ear* +- */var/lib/build/stable/engine-mysite.com-1.2.3.ear* + +** Setting up virtual hosts +Setting up virtual host definitions in the application server makes a +some things easier, such as ECE plugins which set cookies based on +information they get from the app server. + +ece-install can set up the virtual hosts configuration for Tomcat +application servers if the profile is *editor*, *all* or +*presentation*. + +To use this feature, you must define one domain for each publication +in the following FAI parameter: +#+BEGIN_SRC yaml +profiles: + publications: + - name: firepub + domain: fire.escenic.com + - name: ildpub + domain: ild.escenic.com + aliases: + - feuer.example.com + - fuego.example.com +#+END_SRC + +This will produce the following stanzas in *server.xml*: + +#+BEGIN_SRC nxml + + + + + feuer.escenic.com + fuego.escenic.com + + +#+END_SRC + +*** Having WAR files with a different name than the publication +Nine out of ten times (probably a lot more too), the WAR file matches +the name of the publication, e.g. if your publication is called +*sports*, your WAR is called *sports.war*. + +However, if you for some reason have a WAR with a different name, you +can add this as an addition to the first element of the +*fai\_publication\_domain\_mapping\_list* entry: +#+BEGIN_SRC text +fai_publication_domain_mapping_list=" + firepub,fire.war#fire.escenic.com +" +#+END_SRC + +Now, the *appBase* and *docBase* values will become *webapps-fire* and +*fire* respectively. + +*** Host name aliases +As you can see, there's a third optional option to the +*fai\_publication\_domain\_mapping\_list* which can be specified as a +comma separated list of host aliases to be added to the app server host +configuration. + +Furthermore, if these host names are not resolvable to your local +host (neither localhost or the IP of your $HOSTNAME), ece-install will +add entries for these domains to the machine's */etc/hosts* file: +#+BEGIN_SRC conf +# added by ece-install @ Wed Feb 8 19:21:49 CST 2012 +127.0.1.1 fire.escenic.com + +# added by ece-install @ Wed Feb 8 19:21:51 CST 2012 +127.0.1.1 ild.escenic.com +#+END_SRC + +If you do not want ece-install to touch your */etc/hosts*, you can set +*fai\_keep\_off\_etc\_hosts=1* in your *ece-install.conf*. + +** Overview of All Configuration Parameters +The *ece-install* command understands the following settings in +the *ece-install.yaml*: + +#+include: "ece-install-conf-reference.org" + +** Example Configurations +Here are a few simple example configurations, you can find more under +*/usr/share/doc/escenic/examples* if you've got the +*escenic-content-engine-installer* package installed. + +****** Install an Editorial Server and Create a Publication To automatically install an editorial server and create a publication -called "jollygood", the minimal configuration in .ece-install.conf -would be: +called "jollygood", run *ece-install* with the following settings: -#+BEGIN_SRC sh -fai_enabled=1 -fai_editor_install=1 -fai_publication_create=1 -fai_publication_name=jollygood +#+BEGIN_SRC yaml +profiles: + editor: + install: true + publications: + - name: jollygood + create: true #+END_SRC -When FAI is enabled, ece-install will report: -#+BEGIN_SRC sh -[ece-install] Full Automatic Install (FAI) enabled. -[ece-install] All user input will be read from /root/.ece-install.conf +****** Install Two Presentation Servers On a Single Host +To install two presentation servers called *engine1* and +*engine2* on the same host, first run *ece-install* with the following settings: +#+BEGIN_SRC yaml +profiles: + presentation: + install: true +#+END_SRC + +Then run it a second time with the following settings: +#+BEGIN_SRC yaml +profiles: + presentation: + install: true + name: engine2 + port: 8081 + shutdown: 8105 #+END_SRC +More parameters are required the second time. On the first run, +defaults could be used, but the second time you need to override the +defaults to ensure that the second server gets different values. -* Running more than one installation process -If the script believes there's already an ece-intall process running, +* Running More Than One Installation Process +If the command believes there's already an ece-intall process running, it will abort: -#+BEGIN_SRC sh -[ece-install] There's already one ece-install process running. If you believe -[ece-install] this is wrong, e.g. if a previous run of ece-install was aborted -[ece-install] before it completed, you may delete /var/run/ece-install.pid and -[ece-install] run ece-install again. +#+BEGIN_SRC text +There's already one ece-install process running. If you believe +this is wrong, e.g. if a previous run of ece-install was aborted +before it completed, you may delete /var/run/ece-install.pid and +run ece-install again. #+END_SRC -* Re-running ece-install -Although the initial thought behind ece-install, is to run it on a -clean system to get up and running as soon as possible. However, you -may want to re-run ece-install on the same host, for instance to add -another instance of ECE, set up Widget Framework or create another +* Re-running ece-install (and How To Speed It Up) +The initial thought behind ece-install, is to run it on a clean system +to get up and running as soon as possible. However, you may want to +re-run ece-install on the same host, for instance to add another +instance of ECE, set up Widget Framework or create another publication. -ece-install has a number of features which will try to minimise the -time it takes to run it on consecutive runs. If running on Debian -based systems, it will check if you already have installed -pre-requisite 3rd party libraries and only if any are missing will it -ask the package manager to fetch it. +*ece-install* has a number of features which will try to minimise the +time it takes to run it on consecutive runs. For instance, it will +check if you already have installed pre-requisite 3rd party libraries +and only if any are missing will it ask the package manager to fetch +it. -Likewise, ece-install will see if the Escenic artifacts or application -server that you need are already present in the /tmp/ece-downloads -folder, and only download the missing ones (if any). +Likewise, *ece-install* will see if the Escenic artifacts or +application server that you need are already present in +the */var/cache/escenic/ece-install* folder, and only download the +missing ones (if any). To get a list of the artifacts it'll pull from http://technet.escenic.com and http://tomcat.apache.org search for the -following variables: -- technet\_download\_list -- wf\_download\_list -- tomcat\_download +following variables inside */usr/sbin/ece-install*: +- *technet\_download\_list* +- *wf\_download\_list* +- *tomcat\_download* + +Two other ways of speeding up the installation is (of course) to use +the backup/restore feature or install from a EAR and configuration +bundle, see the FAI section. + +* Using a Custom Configuration File for ece-install +You can specify a different configuration by using the -f parameter: +#+BEGIN_SRC text +# ece-install -f ece-install-presentation-server.yaml +#+END_SRC -* Overview of file paths used by the ece-install script +* Overview of File Paths Used by the ece-install command There are of course other paths involved when setting up your system, but these should be the most interesting ones. -|---------------------------------------------+------------------------------------------------------------------| -| Path | Explanation | -|---------------------------------------------+------------------------------------------------------------------| -| /etc/apt/sources.list.d/escenic.list | 3rd party APT repositories added by ece-install *) | -| /etc/default/ece | The configuration file for the ece init.d script | -| /etc/escenic/ece-.conf | Instance specific settings for /usr/bin/ece | -| /etc/escenic/ece.conf | Common ece.conf file for /usr/bin/ece | -| /etc/escenic/engine/common | Common Nursery configuration layer | -| /etc/escenic/engine/common/security | Common security configuration for all ECE instances. | -| /etc/escenic/engine/common/trace.properties | Log4j configuration, produces instance specific log files. | -| /etc/escenic/engine/instance/ | Instance specific Nursery configuration | -| /etc/escenic/solr | ECE specific Solr configuration | -| /etc/init.d/mysql[d] | For starting and stopping MySQL/Percona | -| /etc/init.d/varnish | For starting and stopping Varnish | -| /etc/intit.d/ece | The init.d script managing _all_ the ECE instances on your host. | -| /etc/varnish/default.vcl | The Varnish configuration | -| /opt/escenic | All ECE components can be found here | -| /opt/escenic/assemblytool | The assembly tool | -| /opt/escenic/assemblytool/plugins | Contains symlinks to all plugins in /opt/escenic | -| /opt/escenic/engine | Symlink pointing to the current ECE | -| /opt/tomcat | Symlink pointing to the install Apache Tomcat (catalina\_home) | -| /opt/tomcat- | Instance specific Tomcat files (catalina\_base) | -| /usr/bin/ece | Script for operating all ECE instances + RMI hub and EAE | -| /usr/sbin/drop-and-create-ecedb | DB script used by the ece-install script | -| /usr/sbin/ece-install | The installation script described in this guide | -| /var/log/escenic/-.log | The instance's log4j log | -| /var/log/escenic/-.out | The instance system out log | -| /var/log/escenic/solr..log | The Solr log (not in standard out!) | -| /var/run/escenic/-.pid | The instance's PID file | -|---------------------------------------------+------------------------------------------------------------------| +- */etc/apt/sources.list.d/escenic.list* :: 3rd party APT repositories + added by ece-install *) + +- */etc/default/ece* :: The configuration file for the ece init.d + script. + +- */etc/escenic/ece-.conf* :: Instance specific settings + for */usr/bin/ece* + +- */etc/escenic/ece.conf* :: Common ece.conf file for */usr/bin/ece* + +- */etc/escenic/engine/common* :: Common Nursery configuration layer + +- */etc/escenic/engine/common/security* :: Common security + configuration for all ECE instances. + +- */etc/escenic/engine/common/trace.properties* :: Log4j + configuration, produces instance specific log files. + +- */etc/escenic/engine/instance/* :: Instance specific + Nursery configuration + +- */etc/escenic/solr* :: ECE specific Solr configuration + +- */etc/init.d/mysql[d]* :: For starting and stopping MariaDB/MySQL/Percona + +- */etc/init.d/varnish* :: For starting and stopping Varnish + +- */etc/intit.d/ece* :: The init.d script managing \_all\_ the ECE + instances on your host. + +- */etc/varnish/default.vcl* :: The Varnish configuration + +- */opt/escenic* :: All ECE components can be found here + +- */opt/escenic/assemblytool* :: The assembly tool + +- */opt/escenic/assemblytool/plugins* :: Contains symlinks to all + plugins in */opt/escenic* + +- */opt/escenic/engine* :: Symlink pointing to the current ECE + +- */opt/tomcat* :: Symlink pointing to the install Apache Tomcat + (*catalina\_home*) + +- */opt/tomcat-* :: Instance specific Tomcat files + (*catalina\_base*) + +- */usr/bin/ece* :: Command for operating all ECE instances + RMI hub + and EAE + +- */usr/sbin/ece-install* :: The installation command described in this + guide + +- */var/log/escenic/-.log* :: The instance's log4j log + +- */var/log/escenic/-.out* :: The instance system out + log + +- */var/log/escenic/solr..log* :: The Solr log (not in standard + out!) + +- */var/run/escenic/-.pid* :: The instance's PID file *) Applies only to Debian based systems. -* Uninstalling everything that the ece-install set up -WARNING: this is potentially dangerous as some of these components may -be used by other pieces of software you have running on your -host. However, this may be useful if you're installing a clean -environment and want to e.g. undo your previous install to install a -different profile. +* Overriding the Escenic directories +All of the Escenic specific directories may be overwritten in +=ece-install.conf=. Here's an example of changing all the paths possible +with the same suffix. + +#+BEGIN_SRC conf +dir_suffix=escenic-parallel +escenic_root_dir=/opt/${dir_suffix} +escenic_conf_dir=/etc/${dir_suffix} +escenic_log_dir=/var/log/${dir_suffix} +escenic_data_dir=/var/lib/${dir_suffix} +escenic_run_dir=/var/run/${dir_suffix} +escenic_backups_dir=/var/backups/${dir_suffix} +escenic_spool_dir=/var/spool/${dir_suffix} +escenic_cache_dir=/var/cache/${dir_suffix} +escenic_crash_dir=/var/crash/${dir_suffix} +appserver_parent_dir=/opt +#+END_SRC + +Note, this is only needed if you are running two completely separate +environments on the same host. A use case is if you're setting up a +test environment and want to separate stacks of Escenic Content Engine +and plugins. Another usecase is if you want to test out a new minor +version of ECE, e.g. you're currently running 5.4, but want to try +out 5.5 in parallel on the same boxes. + +This feature is currently only available through the use of +=.conf= configuration files. + +* Extending ece-install by Writing Hooks +ece-install has a number of hooks on which you can hook on your own +scripts. The scripts are to reside in $HOME/ece-conf.d/ and have names +inspired by Debian's package scripts: + +#+BEGIN_SRC text +. +#+END_SRC + +e.g.: + +#+BEGIN_SRC text +install_analysis_server.preinst +#+END_SRC + +Will be run before the body of the hook, just the corresponding +*.postinst* hook will be run after. + +** Accessing ece-install variables +Before running the hook, ece-install will make all its local variables +available in */var/run/escenic/ece-install.env*, which can then be +used by the hook scripts. + +** Example hook +Here is an example hook which will be run after the EAE is installed. + +#+BEGIN_SRC sh +# Put this in your +# $HOME/ece-install.d/install_analysis_server.postinst + +# read ece-install's current variables +source /var/run/escenic/ece-install.env + +# do something useful +echo "Hello from $0, EAE is installed in ${tomcat_base}" \\ + > /tmp/hello.txt +#+END_SRC + +** Available hooks +Currently, the following hooks are available: + +#+BEGIN_SRC text +install_analysis_server.preinst +install_analysis_server.postinst +set_up_solr.preinst +set_up_solr.postinst +#+END_SRC + +* GOALS & DESIGN CHOICES +If you are a power user of *ece-install*, if you're just curious or +indeed wish to extend it, you will find it interesting to read this +section explaining some of its goals and design choices. + +** As few prerequisites as possible +The only pre-requisite should be having a Debian or RedHat based +machine with a good internet connection set up, as well +as *ece-install* itself and an *ece-install.conf* configuration file +of course. No other prerequisite is needed. No Java, no app server, no +database, no nothing. + +** Run on the latest stable releases of Debian, Ubuntu and RedHat +It should run on the latest stable release of Debian, Ubuntu LTS and +RedHat. + +This applies for instance not features only found in BASH 4.x as this +wasn't available in the latest RedHat/CentOS version when the bulk +of *ece-install* was written. + +Another implications, is the *ece-install* cannot depend on (newer) +software packages that only exist in e.g. the regular Ubuntu release, +Fedora or Debian testing. + +** Make it possible to run ece-install on other UNIX systems too +Whenever possible, do not assume the system to be Linux, making it +possible to use it on other unices like FreeBSD, Solaris and OSX too. + +** Require as little input as possible +An important design goal is to install a system with so many best +practises applied as possible. Hence, *ece-install* goes to great +lengths to provide sane defaults that will work for most +users. + +However, there will always be cases were the defaults are not +sufficient and this is where the myriad of *fai_* configuration +options have arisen. + +** Production ready +A lot of choices in *ece-install* are made to ensure that +an *ece-install*-ed machine is production ready, or at the very least +production "like" (for instance if the machine has too little memory, +there's no way it'll be production "ready", although it will still be +set up "like" production). + +This means for instance that the application server configuration is +way more advanced than what you'll find on a regular developer's or +test system, the same goes for the cache configuration, the JVM +parameters, the logging configuration, the monitoring setup and many, +many more things. They are all set up to be ready to go into +production. + +This drive has also another purpose, namely to educate everyone +involved with the site development, including template & Java +developers, of the different components that a production environment +are comprised of. We believe that by having components like monitoring +and caching servers in place from day one on all systems (also the +developer's machine and the test servers), the finished web site will +reach a successful production state faster and more reliably. + +** Fail as fast as possible +Great heed has been taken to detect errors as fast as possible and +bail out with a detailed stack trace of where the error occured. + +One of the implications of this, is that all commands are executed +within a wrapper (called *run*), which checks the return code of the +called command and makes *ece-install* fail if any of its called +commands failed. Both standard out and error are logged +to */var/log/escenic/ece-install.log* and a stack trace is provided to +aid the error hunting. + +** Take care of the user +Take care of the user: speak clear English in full sentences when +giving feedback. Write technical details to the log and keep the user +feedback in standard out as "pleasant" as possible. + +** Safe to re-run +All *ece-install* modules have been written to ensure that it's safe +to re-run *ece-install* with the same *ece-install.conf* +file. Anything which may harm your data (that being two things: the +database data files and the ECE multimedia archive (the Solr index can +be re-generated, so that's not protected in the same way)) will not +be run with a second run of *ece-install*. + +For the cases were the user really wants this (like wiping out a +database when re-running an integration test), options are provided +to override the default behaviour. + +** Modularisation +The command consists of around 20 modules, 5 common libraries and the +main executable: +- /usr/sbin/ece-install :: Responsible for parsing of arguments and + configuration and sending the request off to the + right sub module(s). +- /usr/share/escenic/ece-scripts/ece-install.d/* :: Roughly speaking, + there exists one module per *ece-install* profile, but more + precisely, this division follows along the lines of the + technology involved. For instance, all database related code + resides in *database.sh*, even though there are three profiles + that deal with databases (fai_all_install=1, + fai_restore_from_backup=1 and fai_db_install) +- /usr/share/escenic/ece-scripts/common-*.sh :: common libraries with + code shared between the different ece-scripts commands (such + as *ece*, *system-info*, *ece-build* & *vosa*). + +** Interactive and automatic installations +*ece-install* started out as a purely interactive command, where the +user was prompted for all the choices on what to install (most of +which he/she could just hit ENTER to complete). Once this the +interactive command was shown to customers and Escenic employees, the +need for a fully automatic version became apparent and *ece-install* +got the *fai_enabled* parameter to make the switch. After the a number +of years, we saw however, that pretty much everyone ran *ece-install* +in its fully automatic mode, so in March 2017, we removed the +interactive mode, making the *fai_enabled* parameter obsolete. + +For this reason, the *ece-install.conf* options which have a *fai_* +prefix only applies when running *ece-install* in non-interactive +mode, whereas the ones without any such prefix (like *technet\_user* +and *technet\_download\_list*) applies to both ways of +running *ece-install*. + +As the complexity of *ece-install* and its profiles have grown, some +of the profiles have not been implemented for the interactive mode. + +The term "fully automatic install" (FAI) is inspired by +http://fai-project.org + + +** Possible to run ece-install from anywhere +For easy testing and flexible deployment, it should be possible to +run *ece-install* from a checked out or extracted copy of the source +code. This means that no absolute paths to the *ece-install* and +common modules should be assumed when extending the command. + +* HISTORY +Installing a full Escenic production environment has historically +taken at least a week, with another few to tune all the settings for +high performance. In the middle of 2011, Torstein Krause Johansen set +out to remedy this and created *ece-install* as a personal side +project. + +In February 2012, he joined the Escenic Online SaaS team and was allowed +to work on *ece-install* full time. The command has since then been +used to install several full scale production environments, as well as +being used for full scale automatic integration tests, development +machines, test servers and staging environments. *ece-install* is now +the preferred way of installing new Escenic systems. + +From February 2013, it has been maintained by Erik Mogensen. -Open the ece-install script and look for the "un\_install\_ece" -function, it has copy and pastable commands for undoing most/all -things set up by the script. +* BUGS +Report any bugs found on https://github.com/escenic/ece-scripts/issues. +and be sure to check +https://github.com/escenic/ece-scripts/known-issues.org before +using *ece-install*. +* SEE ALSO +[[varnishd]], [[vcl]], [[nginx]], [[memcached]], [[java]], [[lighthttpd]], [[munin]], [[icinga]], +[[heartbeat]], [[haproxy]], [[mysqld]] -* Example Output -Below is an example output of running ece-install, installing the -all-in-one profile. +* AUTHOR +Torstein Krause Johansen, Erik Mogsensen -#+INCLUDE: "/tmp/ece-install.out" src text +* COPYRIGHT +Copyright 2011-2017 Escenic +Licensed under the Apache License, Version 2.0, see +https://github.com/escenic/ece-scripts/COPYING for further details. diff --git a/usr/share/doc/escenic/ece-install-guide.txt b/usr/share/doc/escenic/ece-install-guide.txt deleted file mode 100644 index 830ef88c..00000000 --- a/usr/share/doc/escenic/ece-install-guide.txt +++ /dev/null @@ -1,395 +0,0 @@ - Welcome to the /usr/sbin/ece-install Guide - ========================================== - -Author: Torstein Krause Johansen -Date: 2011-08-25 22:02:21 CST - - -Table of Contents -================= -1 Supported operating systems - 1.1 Debian based operating systems - 1.2 Other GNU/Linux and Unix systems -2 A note on running ece-install on non-GNU/Linux systems -3 Downloading the ece-install script -4 Running the ece-install script -5 Available server profiles - 5.1 Common for editorial, presentation and search instances - 5.2 1 - Full stack on one host - 5.3 2 - Editorial server (publication server) - 5.4 3 - Presentation server - 5.5 4 - Database server - 5.6 5 - Cache server - 5.7 8 - Install Widget Framework - 5.8 9 - Create Publication -6 Full Automatic Install (FAI) -7 Running more than one installation process -8 Re-running ece-install -9 Overview of file paths used by the ece-install script -10 Uninstalling everything that the ece-install set up -11 Example Output - - -1 Supported operating systems ------------------------------- -For the best experience, run this script on Debian Squeeze (or newer), -the latest Ubuntu LTS or most other Debian based operating systems, -but it should work without any problems on any Unix like system that -has a recent version of BASH. - -1.1 Debian based operating systems -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -If running on a Debian based operating system, ece-install will take -care of installing all required third party software, such as database -& caching server. - -1.2 Other GNU/Linux and Unix systems -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -If you're installing on another Linux or Unix system, you must first -make sure that these 3rd party components are installed. ece-install -will tell you which ones it cannot find and tell you to install these, -so you can simply run it to find out which ones you need. - -You may also look inside the ece-install script for all calls to the -'assert\_pre\_prequesite' method and you'll get a list of the binaries -ece-install assumes are present. - -2 A note on running ece-install on non-GNU/Linux systems ---------------------------------------------------------- -Please note, it's assumed that the OS uses GNU's tools, i.e. find, cp -& tar. If you're running a system which has a different set of core -tools, such as any of the BSDs (including Mac OS X) or Solaris, make -sure that GNU's tools take precedence in the command PATH offered to -ece-install. - -3 Downloading the ece-install script -------------------------------------- -The ece-install script can be downloaded from: -[https://raw.github.com/skybert/ece-scripts/master/usr/sbin/ece-install] -or be downloaded together with the other ece-scripts using git: - - - $ git clone git@github.com:skybert/ece-scripts.git - - - - -4 Running the ece-install script ---------------------------------- -You must run the script as the root user. If you start the script as -a regular user, it will complain: - - - [ece-install] You must be root when running ece-install - - - -To become root on Ubuntu based systems and Mac OS X, do: - - - $ sudo su - - - -On all other Unix like system, do: - - - $ su - - - -5 Available server profiles ----------------------------- -When starting the script, it will ask you which server profile you -want to install: - - - Hi, which server profile do you wish to install? - - Select 1-7 and press ENTER - - 1 - The full stack on one host. Suitable for development and - test environments (cache, ECE, assembly host & database). - 2 - Editorial (publication) server (ECE, assembly host). - 3 - Presentation server (ECE, memcached). - 4 - Database server. - 5 - Cache server (cache & web server). - 6 - RMI hub. - 7 - Standalone search instance (solr + indexer-webapp). - 8 - Install Widget Framework. This will also set up a WF based - publication for you (repo.escenic.com user/pass required). - - - -5.1 Common for editorial, presentation and search instances -============================================================ -- Sun Java 6 -- Apache Tomcat application server: - * Compression of textual content (previously, this was typically set - up with Apache and its mod\_deflate module). - * pooling set up tweaked for high read/write performance. - * proper logging configuration directing solr messages to its own log. - * routing information in the cookies - * access log - -- APR, native library for optimal Tomcat I/O performance -- Escenic Assembly environment -- Database driver - -- Compression of content (what was previously accomplished with - Apache/mod\_deflate) - -5.2 1 - Full stack on one host -=============================== -This profile is suitable for developers and admins wanting to set up a -test environment. It will install the full stack including caching -server, application server, ECE, assembly host, database, Widget -Framework, as well as creating a publication for you. - -For further details on each of the different server components, see -the different profile descriptions bellow. - - -5.3 2 - Editorial server (publication server) -============================================== - - -5.4 3 - Presentation server -============================ -This will set up a typical presentation server: -- Memcached, distributed memory cache -- Escenic Assembly environment (which may be removed after the - installation). -- Deployment setup to only deploy escenic-admin and the - publication(s). - -5.5 4 - Database server -======================== -If ece-install is run on a support version of Debian or Ubuntu, this -will install the excellent Percona distribution of MySQL with their -set of high performance patches. - -If not, MySQL or Percona must be installed in advance. - -Given that the mysqld is install, this profile will download all the -Escenic components and install the ECE database schema based from the -SQL files contained inside the distribution bundles. To accomplish -this, the script will make a call to drop-and-create-ecedb in the same -directory as the ece-intall script itself. - -If you wish to change the DB's host, user or password, you must update -the drop-and-create-ecedb script prior to running ece-install. - -5.6 5 - Cache server -===================== -If ece-install is run on a support version of Debian or Ubuntu, it -will install the latest Varnish 3.0 caching server from the Varnish -APT repository. - -If ece-install is run on a different platform, the admin must install -Varnish 3.x prior to running ece-install. - -The script will configure Varnish for a typical Escenic site: -- will set up an access control lists of IPs which may access the - privileged web applications such as /escenic-admin, /escenic and - /webservice. - - ece-install will also add the host from which you connect, making - sure that if you've SSH-ed into the server to conduct the install, - you'll automatically be included in the "staff" ACL and can access - all the web applications without editing these ACLs (or disabling - security as many does). - -- will set up sticky sessions/session binding -- will set up a backend cluster and allow the user to enter the - different backend servers that will serve the web site. -- will set up configuration to strip away cookies from static - resources, such as CSS, JS and pictures. - -TBD: -- If run on a Linux platform, the script will tweak the kernel - parameters for optimal TCP handling for a web facing server. -- Will install the nginx web server for serving static content and - will configure Varnish accordingly. This will be very useful for - Adactus servers wanting to pull content from your ECEs. - -5.7 8 - Install Widget Framework -================================= -You'll need a user name and password for accessing the -repo.escenic.com Maven repository. You should get these credentials -when you bought Widget Framework. If you for some reason do not have -these, please contact support@escenic.com. - -If you don't have these ready in your .escenicrc, ece-install will -complain: - - - [ece-install] Be sure to set wf_user and wf_password in /root/.escenicrc - [ece-install] If you don't have these, please contact support@escenic.com - - - -5.8 9 - Create Publication -=========================== -This profile will create a publication for you, only asking you the -name of the publication and which ECE instance to use for its -creation. - -This installation profile will base the publication on the Widget -Framework if its present on the system, if not, ECE's clean demo WAR -is used as a basis. - -6 Full Automatic Install (FAI) -------------------------------- -The ece-install script has support for doing a full automatic install -(FAI). - -The ece-install script understands for the following settings in the -$HOME/.ece-install.conf file of the root user: - - Parameter Default Description - --------------------------------+------------------+----------------------------------------------- - fai\_all\_install 0 Install all components on your server. - fai\_cache\_install 0 Install cache server profile - fai\_cache\_backends ${HOSTNAME}:8080 Space separated, e.g. "app1:8080 app2:8080" - fai\_db\_install 0 Install db profile - fai\_db\_host $HOSTNAME Useful for editor & presentation profiles - fai\_db\_port 3306 Useful for editor & presentation profiles - fai\_editor\_install 0 Install the editorial profile - fai\_editor\_name editor1 Name of the editor instance - fai\_editor\_port 8080 HTTP port of the editor instance - fai\_editor\_shutdown 8005 Shutdown port of the editor instance - fai\_enabled 0 Whether or not to run ece-install in FAI mode - fai\_presentation\_install 0 Install the presentation server profile - fai\_presentation\_name web1 Name of the presentation server instance - fai\_presentation\_port 8080 HTTP port of the presentation server instance - fai\_presentation\_shutdown 8005 Shutdown port of the presentation instance - fai\_publication\_create 0 Create a new publication - fai\_publication\_name mypub Name of the publication - fai\_publication\_use_instance dev1 Name of local instance to use for creation - fai\_rmi\_install 0 Install RMI hub profile - fai\_search\_name search1 Name of the search instance - fai\_search\_port 8080 HTTP port of the search instance - fai\_search\_shutdown 8005 Shutdown port of the search instance - fai\_wf\_install 0 Install Widget Framework profile - -As you've probably have guessed, 0 means "false" and 1 means "true" :-) - -To automatically install an editorial server and create a publication -called "jollygood", the minimal configuration in .ece-install.conf -would be: - - - - fai_enabled=1 - fai_editor_install=1 - fai_publication_create=1 - fai_publication_name=jollygood - - - - -When FAI is enabled, ece-install will report: - - - [ece-install] Full Automatic Install (FAI) enabled. - [ece-install] All user input will be read from /root/.ece-install.conf - - - - -7 Running more than one installation process ---------------------------------------------- -If the script believes there's already an ece-intall process running, -it will abort: - - - [ece-install] There's already one ece-install process running. If you believe - [ece-install] this is wrong, e.g. if a previous run of ece-install was aborted - [ece-install] before it completed, you may delete /var/run/ece-install.pid and - [ece-install] run ece-install again. - - - - -8 Re-running ece-install -------------------------- -Although the initial thought behind ece-install, is to run it on a -clean system to get up and running as soon as possible. However, you -may want to re-run ece-install on the same host, for instance to add -another instance of ECE, set up Widget Framework or create another -publication. - -ece-install has a number of features which will try to minimise the -time it takes to run it on consecutive runs. If running on Debian -based systems, it will check if you already have installed -pre-requisite 3rd party libraries and only if any are missing will it -ask the package manager to fetch it. - -Likewise, ece-install will see if the Escenic artifacts or application -server that you need are already present in the /tmp/ece-downloads -folder, and only download the missing ones (if any). - -To get a list of the artifacts it'll pull from -[http://technet.escenic.com] and http://tomcat.apache.org search for the -following variables: -- technet\_download\_list -- wf\_download\_list -- tomcat\_download - -9 Overview of file paths used by the ece-install script --------------------------------------------------------- -There are of course other paths involved when setting up your system, -but these should be the most interesting ones. - - Path Explanation - ---------------------------------------------+------------------------------------------------------------------ - /etc/apt/sources.list.d/escenic.list 3rd party APT repositories added by ece-install *) - /etc/default/ece The configuration file for the ece init.d script - /etc/escenic/ece-.conf Instance specific settings for /usr/bin/ece - /etc/escenic/ece.conf Common ece.conf file for /usr/bin/ece - /etc/escenic/engine/common Common Nursery configuration layer - /etc/escenic/engine/common/security Common security configuration for all ECE instances. - /etc/escenic/engine/common/trace.properties Log4j configuration, produces instance specific log files. - /etc/escenic/engine/instance/ Instance specific Nursery configuration - /etc/escenic/solr ECE specific Solr configuration - /etc/init.d/mysql[d] For starting and stopping MySQL/Percona - /etc/init.d/varnish For starting and stopping Varnish - /etc/intit.d/ece The init.d script managing _all_ the ECE instances on your host. - /etc/varnish/default.vcl The Varnish configuration - /opt/escenic All ECE components can be found here - /opt/escenic/assemblytool The assembly tool - /opt/escenic/assemblytool/plugins Contains symlinks to all plugins in /opt/escenic - /opt/escenic/engine Symlink pointing to the current ECE - /opt/tomcat Symlink pointing to the install Apache Tomcat (catalina\_home) - /opt/tomcat- Instance specific Tomcat files (catalina\_base) - /usr/bin/ece Script for operating all ECE instances + RMI hub and EAE - /usr/sbin/drop-and-create-ecedb DB script used by the ece-install script - /usr/sbin/ece-install The installation script described in this guide - /var/log/escenic/-.log The instance's log4j log - /var/log/escenic/-.out The instance system out log - /var/log/escenic/solr..log The Solr log (not in standard out!) - /var/run/escenic/-.pid The instance's PID file - -*) Applies only to Debian based systems. - -10 Uninstalling everything that the ece-install set up -------------------------------------------------------- -WARNING: this is potentially dangerous as some of these components may -be used by other pieces of software you have running on your -host. However, this may be useful if you're installing a clean -environment and want to e.g. undo your previous install to install a -different profile. - -Open the ece-install script and look for the "un\_install\_ece" -function, it has copy and pastable commands for undoing most/all -things set up by the script. - - -11 Example Output ------------------- -Below is an example output of running ece-install, installing the -all-in-one profile. - -CANNOT INCLUDE FILE /tmp/ece-install.out - diff --git a/usr/share/doc/escenic/ece-install-hooks.org b/usr/share/doc/escenic/ece-install-hooks.org new file mode 100644 index 00000000..1665ae0b --- /dev/null +++ b/usr/share/doc/escenic/ece-install-hooks.org @@ -0,0 +1,3 @@ + +- install_analysis_server.preinst +- install_analysis_server.postinst diff --git a/usr/share/doc/escenic/ece-local-builder-guide.org b/usr/share/doc/escenic/ece-local-builder-guide.org new file mode 100644 index 00000000..d1792b84 --- /dev/null +++ b/usr/share/doc/escenic/ece-local-builder-guide.org @@ -0,0 +1,109 @@ +#+TITLE: The ece-local-builder Guide +#+AUTHOR: Escenic Cloud Team + +* NAME +ece-local-builder + +* SYNOPSIS +ece-local-builder [[[-c builder-conf-file]]] [[[-s]]] [[[-a]]] [[[-b]]] [[[-V]]] + +* DESCRIPTION +ece-local-builder is a command to create a single project builder in a standalone machine or +setting up local builder with in a Escenic Installation. This command requires an configuration +file to setup the builder. + +** -c builder-conf-file +Setup builder under customer_name user provided in the builder.conf file . +The user creation can be skipped using -s option. + +** -a +Add builder under all the existing user. + +** -s +Skip creation of the user. + +** -b +Executes a ece-build operation on given branch in builder.conf file + +** -V +Shows the version of ece-local-builder + +** Initialize builder configuration +To setup builder requires a builder.conf file. A sample builder.conf file can be generated +under /etc/escenic directory by running the following command +#+BEGIN_SRC sh +$ ece-local-builder initialize +#+END_SRC + +** Setting up builder +Setting up builder is running the ece-local-builder command with the configuration file. +The sample /etc/escenic/builder.conf file looks like following +#+BEGIN_SRC sh +customer_name=customer-name +technet_user=technet-username +technet_password=technet-password +maven_user=maven-username +maven_password=maven-password +git_url=ssh://git@git-ip-address/mycloud.git +assemblytool_url=http://technet.escenic.com/downloads/release/57/assemblytool-2.0.7.zip +local_mode=false +auto_deploy=false +ear_base_url=http://builder-ip-address:81 +package_for_machine=all +branch_name=master +maven_download_url=http://www.eu.apache.org/dist/maven/maven-3/3.3.3/binaries/apache-maven-3.3.3-bin.zip +#+END_SRC + +The properties of builder.conf file are described here +* customer_name +The name of the customer +* technet_user +Username for escenic technet to download the distribution +* technet_password +Password for the technet user +*maven_user +Username for Escenic maven repository +* maven_password +Password for Escenic maven repository +* git_url +Customer git project url +* assemblytool_url +Download url for Escenic Assembly Tool +* local_mode +Sets the builder in local mode if value is set to true. Defaule is false +* auto_deploy +Sets the builder to automatic deploy of the created ear and configuration package if set to true. +Default value is false +* ear_base_url +The base url for the ear and configuration packages to be exposed to external systems. +* package_for_machine +Machine name, for which the configuration will be created and deployed if auto_deploy is set to ture. +* branch_name +Customer git project branch name from where the build will be made +* maven_download_url +Maven download url. Default value is set to +http://www.eu.apache.org/dist/maven/maven-3/3.3.3/binaries/apache-maven-3.3.3-bin.zip + +Now after preparing the configuration file with valid values for the properties. We can run the following commands + +* Examples +* To setup a builder under the customer_name user +#+BEGIN_SRC sh +$ ece-local-builder -c /etc/escenic/builder.conf +#+END_SRC + +* To skip the user creation but setup builder under the existing user + +#+BEGIN_SRC sh +$ ece-local-builder -c /etc/escenic/builder.conf -s +#+END_SRC + + +* COPYRIGHT +Copyright 2011-2015 Escenic + +Licensed under the Apache License, Version 2.0, see +https://github.com/escenic/ece-scripts/COPYING for further details. + +* AUTHOR +Sk Mohd Anwarul Islam diff --git a/usr/share/doc/escenic/examples/ece-install-all-in-one.conf b/usr/share/doc/escenic/examples/ece-install-all-in-one.conf new file mode 100644 index 00000000..e336f329 --- /dev/null +++ b/usr/share/doc/escenic/examples/ece-install-all-in-one.conf @@ -0,0 +1,43 @@ +# example ece-install.conf for installing an all in one environment, +# for instance for a developer's PC or a test server. + +## Your credentials for accessing the Escenic downloads & WF Maven +## repository. +technet_user= +technet_password= +wf_user= +wf_password= + +fai_all_install=1 + +## This setting is useful if you're constantly re-running ece-install +## to wipe out an environment where the data are not +## important. Developers and QA personell will find this setting +## useful. +# fai_db_drop_old_db_first=1 + + +## If you want to use ECE 5.4 instead of ECE 5.3, uncomment these +## download lists (ece-install has built-in download lists which +## currently default to ECE 5.3 archives): +# technet_download_list=" +# http://technet.escenic.com/downloads/assemblytool-2.0.2.zip +# http://technet.escenic.com/downloads/community-engine/community-engine-3.7.0.123608.zip +# http://technet.escenic.com/downloads/release/54/analysis-engine-2.4.0.127591.zip +# http://technet.escenic.com/downloads/release/54/dashboard-1.3.0.123618.zip +# http://technet.escenic.com/downloads/release/54/engine-5.4.1.126472.zip +# http://technet.escenic.com/downloads/release/54/forum-3.1.0.123741.zip +# http://technet.escenic.com/downloads/release/54/geocode-dist-2.4.0.123726.zip +# http://technet.escenic.com/downloads/release/54/inpage-2.1.1.126618.zip +# http://technet.escenic.com/downloads/release/54/lucy-dist-4.2.0.0.zip +# http://technet.escenic.com/downloads/release/54/menu-editor-dist-2.1.0.123205.zip +# http://technet.escenic.com/downloads/release/54/poll-2.2.0.123043.zip +# http://technet.escenic.com/downloads/release/54/section-feed-dist-2.1.0.122795.zip +# http://technet.escenic.com/downloads/release/54/xml-editor-dist-2.2.0.0.zip +# " + +# wf_download_list=" +# http://download.escenic.com/release/release/54/widget-framework-core-2.0.1.126521.zip +# http://download.escenic.com/release/release/54/widget-framework-community-2.0.1.126521.zip +# " + diff --git a/usr/share/doc/escenic/examples/ece-install-analysis-engine.conf b/usr/share/doc/escenic/examples/ece-install-analysis-engine.conf new file mode 100644 index 00000000..d4e46683 --- /dev/null +++ b/usr/share/doc/escenic/examples/ece-install-analysis-engine.conf @@ -0,0 +1,15 @@ +# Example ece-install.conf for installing the Escenic Analysis Engine + +## Your credentials for accessing the Escenic downloads. +technet_user= +technet_password= + +fai_analysis_install=1 +fai_analysis_db_install=1 +fai_analysis_db_schema=analysisdb + +## EAE only needs on download (ece-install will per default download a +## full stack of ECE & plugins): +technet_download_list=" + http://technet.escenic.com/downloads/release/54/analysis-engine-2.4.0.127591.zip +" diff --git a/usr/share/doc/escenic/examples/ece-install-cache-and-content-engine.conf b/usr/share/doc/escenic/examples/ece-install-cache-and-content-engine.conf new file mode 100644 index 00000000..317f8468 --- /dev/null +++ b/usr/share/doc/escenic/examples/ece-install-cache-and-content-engine.conf @@ -0,0 +1,19 @@ +## Example ece-install.conf for installing the Escenic Content Engine +## and a caching server on the same host. +## +## This will set up: varnish:80 -> tomcat:8080 on the same host. +## +## See the installation guide for further explanation of the +## parameters. + +## Your credentials for accessing the Escenic downloads & WF Maven +## repository. +technet_user= +technet_password= +wf_user= +wf_password= + +fai_cache_install=1 +fai_presentation_install=1 +fai_presentation_ear=http://my.buildserver.com/mypub-trunk-rev3568-2012-05-28_1213.ear +fai_presentation_deploy_white_list='mypub escenic-admin' diff --git a/usr/share/doc/escenic/examples/ece-install-cache.conf b/usr/share/doc/escenic/examples/ece-install-cache.conf new file mode 100644 index 00000000..62358150 --- /dev/null +++ b/usr/share/doc/escenic/examples/ece-install-cache.conf @@ -0,0 +1,15 @@ +## Example ece-install.conf for installing a stand alone cache server +## with the ECE backends on other machines. +## +## See the installation guide for further explanation of the +## parameters. + +## Your credentials for accessing the Escenic downloads & WF Maven +## repository. +technet_user= +technet_password= +wf_user= +wf_password= + +fai_cache_install=1 +fai_cache_backends="app1:8080 app2:8080 app3:8080" diff --git a/usr/share/doc/escenic/examples/ece-install-db-master.conf b/usr/share/doc/escenic/examples/ece-install-db-master.conf new file mode 100644 index 00000000..e274b02c --- /dev/null +++ b/usr/share/doc/escenic/examples/ece-install-db-master.conf @@ -0,0 +1,15 @@ +## Example ece-install.conf for installing the Escenic Analysis Engine +## See the installation guide for further explanation of the +## parameters. + +## Your credentials for accessing the Escenic downloads & WF Maven +## repository. +technet_user= +technet_password= +wf_user= +wf_password= + +fai_db_install=1 +fai_db_master=1 +fai_db_replication=1 + diff --git a/usr/share/doc/escenic/examples/ece-install-db-slave.conf b/usr/share/doc/escenic/examples/ece-install-db-slave.conf new file mode 100644 index 00000000..ea60c956 --- /dev/null +++ b/usr/share/doc/escenic/examples/ece-install-db-slave.conf @@ -0,0 +1,18 @@ +## Example ece-install.conf for installing the Escenic Analysis Engine +## See the installation guide for further explanation of the +## parameters. + +## Your credentials for accessing the Escenic downloads & WF Maven +## repository. +technet_user= +technet_password= +wf_user= +wf_password= + +fai_db_install=1 +fai_db_master=0 +fai_db_replication=1 +fai_db_master_host=my-db-master +fai_db_master_log_file=mysql-bin.000013 +fai_db_master_log_position=106 + diff --git a/usr/share/doc/escenic/examples/ece-install-dev-environment-from-ear.conf b/usr/share/doc/escenic/examples/ece-install-dev-environment-from-ear.conf new file mode 100644 index 00000000..80c73c8b --- /dev/null +++ b/usr/share/doc/escenic/examples/ece-install-dev-environment-from-ear.conf @@ -0,0 +1,51 @@ +# credentials +technet_user= +technet_password= +wf_user= +wf_password= + +# Java +fai_server_java_version=1.6 + +# Download list, as a minimum, you need assemblytool and engine. +technet_download_list=" + http://technet.escenic.com/downloads/assemblytool-2.0.2.zip + http://technet.escenic.com/downloads/community-engine/community-engine-3.7.0.123608.zip + http://technet.escenic.com/downloads/release/54/analysis-engine-2.4.0.127591.zip + http://technet.escenic.com/downloads/release/54/dashboard-1.3.0.123618.zip + http://technet.escenic.com/downloads/release/54/engine-5.4.1.126472.zip + http://technet.escenic.com/downloads/release/54/forum-3.1.0.123741.zip + http://technet.escenic.com/downloads/release/54/geocode-dist-2.4.0.123726.zip + http://technet.escenic.com/downloads/release/54/inpage-2.1.1.126618.zip + http://technet.escenic.com/downloads/release/54/lucy-dist-4.2.0.0.zip + http://technet.escenic.com/downloads/release/54/menu-editor-dist-2.1.0.123205.zip + http://technet.escenic.com/downloads/release/54/poll-2.2.0.123043.zip + http://technet.escenic.com/downloads/release/54/section-feed-dist-2.1.0.122795.zip + http://technet.escenic.com/downloads/release/54/xml-editor-dist-2.2.0.0.zip +" + +# if you don't need WF, you can set wf_download_list to an empty +# string. +wf_download_list=" + http://download.escenic.com/release/release/54/widget-framework-core-2.0.1.126521.zip + http://download.escenic.com/release/release/54/widget-framework-community-2.0.1.126521.zip +" + +# Install a seperate search instance making us look cool, just like a +# production environment. +fai_search_install=1 +fai_search_port=8081 +fai_search_shutdown=8105 +fai_search_name=search1 + +# Install the database +fai_db_install=1 + +# Presentation server +fai_presentation_install=1 +fai_presentation_ear=http://my.buildserver.com/mypub-trunk-rev3568-2012-05-28_1213.ear +fai_presentation_name=engine1 +fai_presentation_deploy_white_list='mypub escenic-admin indexer-webservice webservice escenic studio' + +# Publication +fai_publication_domain_mapping_list="mypub#example.com" diff --git a/usr/share/doc/escenic/examples/ece-install-editorial.conf b/usr/share/doc/escenic/examples/ece-install-editorial.conf new file mode 100644 index 00000000..bab26c78 --- /dev/null +++ b/usr/share/doc/escenic/examples/ece-install-editorial.conf @@ -0,0 +1,28 @@ +# this will install both an editorial server and a separate search +# instance tailored to an editorial ECE: +# +# content-engnine:8080 -> search:8081 + +# credentials +technet_user= +technet_password= +wf_user= +wf_password= + +# Java +fai_server_java_version=1.6 + +# Install a seperate search instance making us look cool, just like a +# production environment. +fai_search_install=1 +fai_search_port=8081 +fai_search_shutdown=8105 +fai_search_name=search1 +fai_search_for_editor=1 +fai_search_ear=http://my.buildserver.com/mypub-trunk-rev3568-2012-05-28_1213.ear + +# Presentation server +fai_editor_install=1 +fai_editor_ear=http://my.buildserver.com/mypub-trunk-rev3568-2012-05-28_1213.ear +fai_editor_name=engine1 +fai_editor_deploy_white_list='escenic-admin indexer-webservice webservice escenic studio' diff --git a/usr/share/doc/escenic/examples/ece-install-nfs-client.conf b/usr/share/doc/escenic/examples/ece-install-nfs-client.conf new file mode 100644 index 00000000..849e6e9f --- /dev/null +++ b/usr/share/doc/escenic/examples/ece-install-nfs-client.conf @@ -0,0 +1,17 @@ +## Example ece-install.conf for installing an NFS client on a given +## machine. All the ECE hosts, e.g., will typically have this +## ece-isntall profile as a part of their installation procedure. +## +## See the installation guide for further explanation of the +## parameters. + +## Your credentials for accessing the Escenic downloads & WF Maven +## repository. +technet_user= +technet_password= +wf_user= +wf_password= + +fai_nfs_client_install=1 +fai_nfs_server_address=192.168.1.200 +fai_nfs_export_list="/var/exports/multimedia" diff --git a/usr/share/doc/escenic/examples/ece-install-nfs-server.conf b/usr/share/doc/escenic/examples/ece-install-nfs-server.conf new file mode 100644 index 00000000..630e98f7 --- /dev/null +++ b/usr/share/doc/escenic/examples/ece-install-nfs-server.conf @@ -0,0 +1,15 @@ +## Example ece-install.conf for installing an NFS server. +## +## See the installation guide for further explanation of the +## parameters. + +## Your credentials for accessing the Escenic downloads & WF Maven +## repository. +technet_user= +technet_password= +wf_user= +wf_password= + +fai_nfs_server_install=1 +fai_nfs_export_list="/var/exports/multimedia" +fai_nfs_allowed_client_network="192.168.1.0/255.255.255.0" diff --git a/usr/share/doc/escenic/examples/ece-install-presentation-server-with-assembly-tool.conf b/usr/share/doc/escenic/examples/ece-install-presentation-server-with-assembly-tool.conf new file mode 100644 index 00000000..d08fab99 --- /dev/null +++ b/usr/share/doc/escenic/examples/ece-install-presentation-server-with-assembly-tool.conf @@ -0,0 +1,19 @@ +technet_user= +technet_password= +wf_user= +wf_password= + +# This will set up a presentation server, downloading ECE & plugins + +# Assembly Tool from Technet. +fai_presentation_install=1 + +# This will make the above presentation profile use a provided WAR +# file for creating a publication definition in the abovementioned +# assemblytool, which again will be deployed on the new presentation +# server. +fai_publication_name=mypub +fai_publication_war=/tmp/mypub.war + +# This will make sure the above presentation server gets the correct +# host context defintion for your publication. +fai_publication_domain_mapping_list="mypub#mypub.example.com" diff --git a/usr/share/doc/escenic/generate-changelog-guide.org b/usr/share/doc/escenic/generate-changelog-guide.org new file mode 100644 index 00000000..0002aaee --- /dev/null +++ b/usr/share/doc/escenic/generate-changelog-guide.org @@ -0,0 +1,81 @@ +#+TITLE: User Guide for the generate-changelog Command +#+AUTHOR: Escenic Cloud Team +#+OPTIONS: H:6 num:5 toc:2 + +* Introduction +The =generate-changelog= command will help you with the following tasks: + +- [[Getting an overview of all the JIRA issues]] that have been worked on + in the list of commits. + +- [[Getting a full code diff]] of all changes in the current build. + +- [[Getting a risk assessment]] score for this build of the source code. + +* Configuration +The =generate-changelog= command requires a configuration file in the +user's home directory: =.generate-changelog.conf=: +#+BEGIN_SRC text +jira_base_url=https://jira.mycompany.com +svn_base_url=${jira_base_url}/svn + +# optional parameters, may be passed on the command line +project_code=my-jira-project +user=my-jira-user-name +password=my-jira-password +#+END_SRC + +* Running generate-changelog +Go into the directory where you've got the trunk or branch of your +project checked out and issue =generate-changelog= with the =--from= +and optional =--to== parameters to specify which revision you want to +create a change log for: + +#+BEGIN_SRC sh +$ cd ~/src/my-project/branches/5.5 +$ generate-changelog --from 2342 +Full diff of all 7 changes: +/home/torstein/.generate-changelog/my-project/branches/5.5/from-2342-to-COMMITTED.diff +Report: /home/torstein/.generate-changelog/my-project/branches/5.5/from-2342-to-COMMITTED.report +#+END_SRC + +If you don't specify =--to=, you will get the report up until the +latest committed change. + +* Getting a risk assessment +[[Getting an overview of all the JIRA][The report]] also contains a risk assessment score. This score is +calculated on the number of lines in the code diff. This means the +number of lines changed + the diff context. Hence, a one line change +in a two line file will have a lower risk score than a one line +change in a 100 line big file. + +The risk assessment score is just an indicator, it doesn't (as of +yet) have more logic than what's mentioned above, but in future, it +might have some logic that counts some code changes less (or more) +risky than others. + +* Getting an overview of all the JIRA issues +Each run of =generate-changelog= will create a full code diff +available in: +#+BEGIN_SRC text +~/.generate-changelog//trunk/from--to-.diff +#+END_SRC +For instance, if you run =generate-changelog= as this: +#+BEGIN_SRC text +$ generate-changelog --from 5401 --to 5856 --project toy +#+END_SRC + +you will find an overview of all the JIRA issues that have been +mentioned in commit messages. The issue titles have been pulled from +JIRA, so is reporter information and some other bits and pieces. + +#+BEGIN_SRC text +~/.generate-changelog/myproject/trunk/from-5401-to-5856.report +#+END_SRC + +* Getting a full code diff +If you've run =generate-changelog= like what's described in [[Gettingan overview of all the JIRA][getting +the JIRA overview]], you will see your diff in: +#+BEGIN_SRC text +~/.generate-changelog/myproject/trunk/from-5401-to-5856.diff +#+END_SRC diff --git a/usr/share/doc/escenic/help-us-help-you.md b/usr/share/doc/escenic/help-us-help-you.md new file mode 100644 index 00000000..90bceed0 --- /dev/null +++ b/usr/share/doc/escenic/help-us-help-you.md @@ -0,0 +1,90 @@ +# Help Us Help You + +When you have problems with `/usr/bin/ece`, `/usr/sbin/ece-install` or +any of the other commands on this repository, please provide us with a +couple of things so that we can help you in the best way possible: + +## 1 — The versions of ece and ece-install + +``` +$ grep ^ece_scripts_version= /usr/{sbin/ece-install,bin/ece} +``` + +## 2 — The version of your operating system + +Just run these three commands and give us the output: +``` +$ lsb_release -a +$ cat /etc/debian_version +$ cat /etc/redhat-release +``` + +## 3 — For ece-install problems +3.1) Your `ece-install.conf` or `ece-install.yaml` file. + +3.2) The contents of your sources.list (if you're on Debian/Ubuntu): +``` +$ grep -r escenic /etc/apt/sources.list* +``` + +3.3) The log files of both `ece-install` and the instances you're setting up +(here, `engine1` is the instance): + +``` +/var/log/escenic/ece-install.log + +/var/log/escenic/engine1.out +/var/log/escenic/engine1-catalina.out +/var/log/escenic/engine1-tomcat +/var/log/escenic/engine1-messages +``` + +## 4 - For `/usr/bin/ece` or `/etc/init.d/ece` problems +4.1 — The configuration files for your ece command +``` +/etc/default/ece +/ece/escenic/ece.conf +/etc/escenic/ece-engine1.conf +``` + +4.2 The log files of the instance you're working with (`engine1` is the +default): + +``` +/var/log/escenic/engine1.out +/var/log/escenic/engine1-catalina.out +/var/log/escenic/engine1-tomcat +/var/log/escenic/engine1-messages +``` + +## I don't want to think! + +> "I just want to copy and paste a bunch of commands and send them to +> you" + +``` +# Versions +grep ^ece_scripts_version= /usr/{sbin/ece-install,bin/ece} +lsb_release -a +cat /etc/debian_version +cat /etc/redhat-release + +# Sources list(s): +grep -r escenic /etc/apt/sources.list* + +# Conf and log files: +tar czf /tmp/$(date --iso)-debug-files.tar.gz \ + /var/log/escenic/ece-install.log \ + /var/log/escenic/engine1.out \ + /var/log/escenic/engine1-catalina.out \ + /var/log/escenic/engine1-tomcat \ + /var/log/escenic/engine1-messages \ + /etc/default/ece \ + /ece/escenic/ece.conf \ + /etc/escenic/ece-engine1.conf \ + /root/*.conf \ + /root/*.yaml + +echo "Now, send this archive to us:" /tmp/$(date --iso)-debug-files.tar.gz + +``` diff --git a/usr/share/doc/escenic/images/ece-architecture.blockdiag b/usr/share/doc/escenic/images/ece-architecture.blockdiag new file mode 100644 index 00000000..bc7cceb4 --- /dev/null +++ b/usr/share/doc/escenic/images/ece-architecture.blockdiag @@ -0,0 +1,42 @@ +# a typical ECE production environment +blockdiag { + orientation = portrait; + + internet [ shape = "cloud" ]; + cache1 [ color = "green" ]; + cache2 [ color = "green" ]; + pres1 [ color = "orange" ]; + pres2 [ color = "orange" ]; + pres3 [ color = "orange" ]; + db1 [ shape = "flowchart.database" ]; + db2 [ shape = "flowchart.database" ]; + editor1 [ color = "orange" ]; + editor2 [ color = "orange" ]; + journalist [ shape = "actor" ]; + + group { + pres1; + pres2; + pres3; + color = "white"; + }; + # group { + # cache1; + # cache2; + # color = "green"; + # } + + internet -> lb -> cache1, cache2; + cache1 -> pres1, pres2, pres3; + cache2 -> pres1, pres2, pres3; + pres1 -> db-vip, nfs-vip, search1; + pres2 -> db-vip, nfs-vip, search1; + pres3 -> db-vip, nfs-vip, search1; + + journalist -> editor1 -> db-vip, nfs-vip, search2; + import -> editor2 -> db-vip, nfs-vip, search2; + + db-vip -> db1, db2; + db1 <-> db2; + nfs-vip -> nfs1, nfs2; +} diff --git a/usr/share/doc/escenic/images/ece-architecture.png b/usr/share/doc/escenic/images/ece-architecture.png new file mode 100644 index 00000000..eeac468a Binary files /dev/null and b/usr/share/doc/escenic/images/ece-architecture.png differ diff --git a/usr/share/doc/escenic/images/nfs-vip.nwdiag b/usr/share/doc/escenic/images/nfs-vip.nwdiag new file mode 100644 index 00000000..68a4cd88 --- /dev/null +++ b/usr/share/doc/escenic/images/nfs-vip.nwdiag @@ -0,0 +1,13 @@ +nwdiag { + group { + color = "orange"; + nfs1; + nfs2; + } + + network nfs-vip { + address = "192.168.1.200"; + nfs1 [ address = "192.168.1.122", description = "NFS master" ]; + nfs2 [ address = "192.168.1.123", description = "NFS slave" ]; + } +} diff --git a/usr/share/doc/escenic/images/nfs-vip.png b/usr/share/doc/escenic/images/nfs-vip.png new file mode 100644 index 00000000..283e0f2c Binary files /dev/null and b/usr/share/doc/escenic/images/nfs-vip.png differ diff --git a/usr/share/doc/escenic/images/puppet-build-server-ece-deploy-and-ece-info-workflow.blockdiag b/usr/share/doc/escenic/images/puppet-build-server-ece-deploy-and-ece-info-workflow.blockdiag new file mode 100644 index 00000000..f88bcc11 --- /dev/null +++ b/usr/share/doc/escenic/images/puppet-build-server-ece-deploy-and-ece-info-workflow.blockdiag @@ -0,0 +1,19 @@ +blockdiag { + A [label = "build-server"]; + B [label = "build-server: http://build/engine-1.2.3.ear"]; + C [label = "pres1: puppet-agent" ]; + D [label = "pres1: ece deploy", color = "orange" ]; + E [label = "pres1: app server" ]; + F [label = "pres1: /var/lib/escenic/pres1.state"]; + G [label = "pres1: ece info" ]; + + A -> B [label = "builds"]; + C -> B [label = "reads"]; + C -> D [folded, label = "calls"]; + + D -> B [label = "gets"]; + D -> E [label = "deploys"]; + D -> F [label = "writes"]; + + G -> F [label = "reads"]; +} \ No newline at end of file diff --git a/usr/share/doc/escenic/images/puppet-build-server-ece-deploy-and-ece-info-workflow.png b/usr/share/doc/escenic/images/puppet-build-server-ece-deploy-and-ece-info-workflow.png new file mode 100644 index 00000000..b4c82a15 Binary files /dev/null and b/usr/share/doc/escenic/images/puppet-build-server-ece-deploy-and-ece-info-workflow.png differ diff --git a/usr/share/doc/escenic/images/vosa_eae_machine.png b/usr/share/doc/escenic/images/vosa_eae_machine.png new file mode 100644 index 00000000..3098bba6 Binary files /dev/null and b/usr/share/doc/escenic/images/vosa_eae_machine.png differ diff --git a/usr/share/doc/escenic/images/vosa_eae_machine.rackdiag b/usr/share/doc/escenic/images/vosa_eae_machine.rackdiag new file mode 100644 index 00000000..52f0d31a --- /dev/null +++ b/usr/share/doc/escenic/images/vosa_eae_machine.rackdiag @@ -0,0 +1,9 @@ +rackdiag { + rack { + ascending; + + 2U; + 1: analysis-engine:8080 + 2: db:3306 + } +} diff --git a/usr/share/doc/escenic/images/vosa_ece_machine.png b/usr/share/doc/escenic/images/vosa_ece_machine.png new file mode 100644 index 00000000..13c42caf Binary files /dev/null and b/usr/share/doc/escenic/images/vosa_ece_machine.png differ diff --git a/usr/share/doc/escenic/images/vosa_ece_machine.rackdiag b/usr/share/doc/escenic/images/vosa_ece_machine.rackdiag new file mode 100644 index 00000000..a5bb8a00 --- /dev/null +++ b/usr/share/doc/escenic/images/vosa_ece_machine.rackdiag @@ -0,0 +1,11 @@ +rackdiag { + rack { + ascending; + + 4U; + 1: cache-server:80 + 2: content-engine:8080 + 3: memory-cache:11211 + 4: search-server:8180 + } +} diff --git a/usr/share/doc/escenic/images/vosa_http_caches.blockdiag b/usr/share/doc/escenic/images/vosa_http_caches.blockdiag new file mode 100644 index 00000000..d2c196e9 --- /dev/null +++ b/usr/share/doc/escenic/images/vosa_http_caches.blockdiag @@ -0,0 +1,14 @@ +blockdiag { + orientation = portrait; + + pres1-apt [ label ="pres1: apt-get", color = "orange" ]; + control-apt [ label ="control: apt-cacher", color = "green" ]; + pres1-get [ label ="pres1: http_proxy=control:3128", color = "orange" ]; + control-get [ label ="control: squid:3128", color = "green" ]; + apt [ label ="vosa apt repo", color = "brown" ]; + + pres1-apt -> control-apt -> apt, official-apt-repos, 3rd-party-apt-repos; + pres1-get -> control-get -> technet, ftp.vizrt.com [ label = "GETs" ]; + + +} diff --git a/usr/share/doc/escenic/images/vosa_http_caches.png b/usr/share/doc/escenic/images/vosa_http_caches.png new file mode 100644 index 00000000..f5fa5b5a Binary files /dev/null and b/usr/share/doc/escenic/images/vosa_http_caches.png differ diff --git a/usr/share/doc/escenic/presentations/ece-install-introduction/2012-03-30-times-online.png b/usr/share/doc/escenic/presentations/ece-install-introduction/2012-03-30-times-online.png new file mode 100644 index 00000000..a84b6590 Binary files /dev/null and b/usr/share/doc/escenic/presentations/ece-install-introduction/2012-03-30-times-online.png differ diff --git a/usr/share/doc/escenic/presentations/ece-install-introduction/presentation.org b/usr/share/doc/escenic/presentations/ece-install-introduction/presentation.org new file mode 100644 index 00000000..779ecc09 --- /dev/null +++ b/usr/share/doc/escenic/presentations/ece-install-introduction/presentation.org @@ -0,0 +1,129 @@ +#+TITLE: BASH all you want: ece-install +#+AUTHOR: Torstein Krause Johansen +#+EMAIL: tkj@escenic.com +#+DATE: 2012-03-30 + +* Agenda +- The Challenge +- What? (is ece-install) +- Why? (should anyone want to use ece-install) +- What Does It Need? (to run) +- For Whom? (is ece-install intended for) +- Show Me! (how to use ece-install) +- How? (can I get started using ece-install) + +* The Challenge +[[file:2012-03-30-times-online.png]] + +* The Challenge - II +[[file:ece-architecture.png][ece-architecture.png]] + +* What? +ece-install is a a shell command which gives you full automatic +installation of: + +- Escenic Content Engine & all plugins +- Escenic Analysis Engine +- Standalone search instances (Solr + indexer) +- Database +- NFS + +and ... + +* What? - II + +Full automatic installation of: +- Cache server +- High availability services (HA) +- Monitoring clients +- Monitoring server +- Host self reporting +- and yes, even the RMI Hub + +* What? - III + +Does all the hard bits for you, like: +- Cache server with cookie stripping, load balancing, + session binding, locked down security. +- Master/slave database setup with HA services and virtual IPs. +- Analysis Engine (stats) with production ready settings. + +and there's more ... + +* What? - IV + +- Separate search instances, +- Tweaked JVM parameters, GC strategies that's known to work +- DB pool settings that's been tested for high load +- Proper logging (yes, there are endless systems were this is not set + up correctly). + +* Why? +- Instead of one week, you'll spend one day, perhaps even less. +- You will get production quality settings from day one. +- Benefit from best practices from experienced Escenic engineers, + customers & partners. + +* Why? - II +- Web site will function better from day one +- Web site will perform better from day one +- Reduces installation costs +- Reduces maintenance costs + +* Why? - III +Not only for production environments, using ece-install, you'll also +find it's: + +- Easy to set up proper staging environments +- Easy to set up proper testing environments +- Easy to set up development environments + +* What Does It Need? +- The latest stable Ubuntu, Debian, CentOS or RedHat Linux + distribution installed. +- A network connection + +Yes, that's it. + +* For Whom? + +- Professional Services +- QA +- R & D +- SaaS / VOSA +- Escenic partners +- Experienced customers + +* Show Me! + +- Install two presentation servers +- Install a cache server with load balancing +- Check out its new self reporting module + + +* How? + +- git clone https://github.com/escenic/ece-scripts/ (or zip download) +- Daily builds of DEB and RPM packages from + http://hudson.dev.escenic.com + +* How - II +- Documentation: + /usr/share/doc/escenic/ece-install-guide.org +- Screencasts (instruction videos): http://www.screenr.com/user/skybert + +* To Sum It All Up +- Full automatic install of all components of an ECE production + environment +- Production ready settings out of the box +- Up to date, generated documentation & monitoring included +- You only needs a Linux machine with a network connection +- Daily DEB & RPM packages available + +* 問題? + +- Questions to torstein@escenic.com +- Or ask on the SaaS mailing list: saas@lists.escenic.com +- Or pop by the SaaS chat room: saas@conference.ardendo.se + +* Xièxiè! diff --git a/usr/share/doc/escenic/system-info-guide.org b/usr/share/doc/escenic/system-info-guide.org new file mode 100644 index 00000000..7938fa21 --- /dev/null +++ b/usr/share/doc/escenic/system-info-guide.org @@ -0,0 +1,60 @@ +#+TITLE: The system-info Guide +#+AUTHOR: Escenic Cloud Team + +* NAME +system-info + +* SYNOPSIS +system-info [[[-o output-file]]] [[[-u user]]] [[[-t]]] [[[-m]]] [[[-s]]] [[[-v]]] [[[-V]]] [[[-V][--version]]] + +* DESCRIPTION +system-info is a command that creates an overview of your system, +specially crafted for Escenic installations, but also of great value +for any UNIX or GNU/Linux system. + +** -t +Report an import job overview + +** -m +Output one report file per module. + +** -o output-file +The file to output the report t + +** -u user +The user to run the ece commands. This is required when running +system-info as root. + +** -s +Do not show temporaries, i.e. values that change often, like dates +and uptime. + +** -v +Verbose + +** -V +Shows the version of system-info + +* Running as a regular user +Running as a regular user, creating an HTML report: +#+BEGIN_SRC sh +$ system-info -f html > report.html +#+END_SRC + +* Running as the root user +When running system-info as the root user, more OS information +will be printed. When running as root, you must specify the user to +run the queries to ece as: +#+BEGIN_SRC sh +# system-info -u escenic -f html \\ + > /var/www/system-info/report.html +#+END_SRC + +* COPYRIGHT +Copyright 2011-2015 Escenic + +Licensed under the Apache License, Version 2.0, see +https://github.com/escenic/ece-scripts/COPYING for further details. + +* AUTHOR +Torstein Krause Johansen diff --git a/usr/share/doc/vizrt/README.org b/usr/share/doc/vizrt/README.org new file mode 100644 index 00000000..6614d27c --- /dev/null +++ b/usr/share/doc/vizrt/README.org @@ -0,0 +1,3 @@ +* Depricated + +This code is now maintained in a private repository diff --git a/usr/share/doc/vizrt/vosa-guide.org b/usr/share/doc/vizrt/vosa-guide.org new file mode 100644 index 00000000..0b9f681d --- /dev/null +++ b/usr/share/doc/vizrt/vosa-guide.org @@ -0,0 +1,380 @@ +#+TITLE: The VOSA Guide +#+AUTHOR: Vizrt Online / The SaaS Team +#+OPTIONS: H:6 num:5 toc:2 + +* NAME +vosa - command to manage KVM and Amazon instances that typically form +a cluster of servers serving one or more Escenic based web sites. + +* SYNOPSIS +vosa [-i instance-dir] [-v uec-version] [-V] [-d] [-h] COMMAND + +* OPTIONS +** -i instance-dir +Most [[COMMANDS]] require the *-i* parameter to be set. You pass it the +path to your VM's configuration like this: +#+BEGIN_SRC sh +# vosa -i /etc/vizrt/vosa/available.d/vm03 enable +#+END_SRC +** -v uec-version +Let's you specify the UEC version to use +** -d +Make *vosa* more verbose. The more *-d* switches you specify, the +more verbose it gets. +** -V +Print the *vosa* version and exit +** -h +Print usage help and exit. + +* COMMANDS +** enable +Enable a specific VM. +** disable +Disable a specific VM. +** install +Creates a new disk image etc. +** uninstall +Removes the disk image etc. +** start +Starts (boots) the VM. +** status +Tells you about the VM, if it's enabled, running, alive, its uptime. +** make +Makes a vmdk image of the vm +** ova +Creates a ova file from the vmdk image. + +* What is VOSA? + +Vosa is short for Vizrt Online System Administration, a set of tools +created by Vizrt to ease various system administration chores. + +It facilitates the set up and installation of virtual machines, +booting them up and installing software on them. It also includes +steps to set up a puppet master with a generic client certificate, +and tools to install puppets that do what that puppet master says. + +All of this is possible manually by apt-get'ing and configuring +manualy; the value of these scripts is that all of this is +made possible without intervention.. + +There are various post-installation hooks available which perform +various tasks, for simple things like creating an /etc/motd file, +to more advanced things like setting up a puppet master or +installing a production ready Escenic Content Engine using +ece-install. + +** Who is it for? + +It is aimed at anyone who needs to install lots of virtual machines, +and particularly at those who agree that it is valuable to be able +to completely reinstall a virtual machine from scratch without +intervention, and then put that virtual machine in production. + +** Why should I use it? + +The normal way of virtualizing is to sit someone in front of a +fresh VM and start to install software on it. The software is +configured, and more software piles on until it's production ready. +Some final tweaks are made and then it might go into production. +All of this installation, configuration and tweaking is /valuable/ +in the sense that doing it all over again actually costs money since +someone needs to sit down and do it. Most of the time, the people +who did the original installation are no longer around, so doing it +all from scratch would also revert some tweaks. + +A different way of looking at virtualization is to ascribe /no +value/ to the virtual machine itself, or at least to the the disk +image that constitutes the virtual machine. In order to do so, you +need to make sure that it is possible to create a fully functional +virtual machine at any time. One that is production ready, with all +the last-minute-tweaks in place. + +The Vosa scripts do just that. + +A short definition file tells vosa what base image to start with; +this is an Ubuntu Enterprise Cloud image, but in theory other image +types could be supported. The file also provides networking +parameters and so on. Most importantly, these files also tell vosa +what to do with the virtual machine once it has booted, so-called +"post install hooks". Each of these are executed when the "first +boot" has completed. These hooks run only the first time the +virtual machine has booted, and should be written in such a way that +they end up with a production ready system that can go straight into +production. + +** Amazon or Physical? + +Vosa supports two flavours of virtualization, namely one based on +Amazon's excellent Elastic Compute Cloud (EC2), and one based on the +excellent Open Source Kernel-based Virtual Machine (KVM). Vosa aims +to blur the boundaries of these two types of virtualization, making +it possible to use the same technologies for installing software. +The possibility of using local hardware makes it possible to run a +cheap stack of virtual machines using the same method of installing +software as the production environment. Developers can also have a +local "stack" of virtual machines, and not have to rely on AWS for +virtualization. + +In the absence of EC2 metadata service that's available to real +cloud instances, the kvm flavour of virtualization allows some +additional perks, like vnc support, and it generates "throw-away" +SSH keys which are used for a single kvm instance. + +** What can I do with this? + +Probably the most advanced thing you could do with this is to +automate the setup of a complete data center, with database, nfs, +a virtualized server park, and various virtual machines, tomcats, +varnishes and so on, all from a single set of configuration files, +without intervention. + +But you could also use it to just easily re-install a VM using the +latest and greatest Ubuntu release, instead of (as is usual) not even +daring to run apt-get upgrade. + +* Installation of VOSA package + +This is a terse description of what it takes to get a system to run +vosa. + +** Requirements: + +In order to benefit from running vosa with kvm, you need the following + +- A physical machine that supports virtualization +- kvm or qemu-kvm +- genisoimage +- nc +- tunctl +- sudo access to kvm (or the more usual, which is to run vosa as root...) +- a bridged network + +In theory, it should be possible to script this too, or to get e.g. +puppet to do this for us. Watch this space! + +In order to benefit from running vosa with ec2, you need the following + +- Amazon account +- ec2-api-tools +- A named SSH keypair +- Signing key + +** Setting up vosa on EC2 + +Commonly, setting up vosa in the cloud entails first making an EC2 +instance that will act as "the control server". vosa will typically +be installed there. + + : apt-get install vosa + +The vosa command should now work (to a certain extent) + + : vosa help + : vosa commands + : vosa longhelp | less + +To make vosa usable, you need to initialize it: + + : sudo vosa init + +This will create the /etc/vizrt/vosa directory structure, and a +skeleton of a virtual machine definition. + +#+BEGIN_SRC sh +$ sudo cp -r /etc/vizrt/vosa/skeleton-amazon \\ + /etc/vizrt/vosa/available.d/my-first-vosa +#+END_SRC + +Configure the my-first-vosa/amazon.conf file as you see fit. This would include: + +- Generate (or re-use) an SSH private key, and uploading the public key to Amazon +- Generate (or re-use) an EC2 API signing key, and uploading the certificate to Amazon +- Create a VPC and subnet (if you want to deploy the instances to a VPC subnet) + +Details on these steps are outlined in the sample amazon.conf file. + +** Setting up vosa on bare metal (kvm) + +Setting up vosa entails a few manual installation steps. Among other +things it will +- provide the "vosa" command and its required libraries +- download an image of an Ubuntu Enterprise Cloud +- create 10 tap interfaces (an arbitrary number, really, see below + for an explanation). +- create the same number of "tap*.availablenetwork" files in + */var/run/vizrt/vosa/* each one signifying the names of the tap + interfaces that can be used. + +So without further ado, let's get started. First of all, we need to +onstall the vosa command itself and its required libraries: + + : apt-get install vosa + +The vosa command should now work (to a certain extent) + + : vosa help + : vosa commands + : vosa longhelp | less + +To make vosa usable, you need to initialize it: + + : vosa init + +This will create the /etc/vizrt/vosa directory structure, and a +skeleton of a virtual machine definition. + +Let's download an Ubuntu Enterprise Cloud (UEC) image to use as the +base OS. vosa does this for you: + + : vosa -v oneiric download + +** Networking + +For this to be useful, your machines need to be accessible directly +on the local network. The scripts have only been tested on a bridged +network. So make a bridge, call it br0 or something. How this is +done is, however outside the scope of this document. + +Once you have a bridged network, you need to create tap interfaces +for each of your virtual machines. Let's make 10 to start with. +The reason these need to be pre-allocated is that we've seen that +doing this temporarily (ca 10 seconds) makes the network go +completely dark, and so shouldn't happen whenever any virtual +machines are running. + +#+BEGIN_SRC sh +mkdir -p /var/run/vizrt/vosa +br=br0 +for i in $(seq 1 10) ; do + tap=$(tunctl -b) + echo $br > /var/run/vizrt/vosa/$tap.availablenetwork + brctl addif $br $tap + ifconfig $tap up 0.0.0.0 +done +#+END_SRC + +If your machine has multiple network interfaces, talking with different +physical networks, you need to make several bridges, and create tap +devices for each bridge, indicating which bridge the tap interface is +bound to. The above script uses the *br0* bridge. + +This needs to happen every time you boot, so the script above should be +copied to e.g. /etc/rc.local or some other system wide boot location. + +*tunctl* creates the tap interface, and we create a file with the name +of the created tap interface in a directory. This little snippet +needs to run every time the host machine boots. + +** Defining a virtual machine + +Defining a virtual machine is a bit different than when using virsh +or VMware of VirtualBox. Vosa exploits the fact that the UEC images +are pre-seeded with cloud-init, and so have a hook to execute code +during the first boot. This means we don't need to make any changes +to the image file itself, but can boot the unmodified UEC image. + +A big benefit of this is that the exact same UEC images are available +in Amazon EC2, and also in a Eucalyptus private cloud. This means +that vosa will be able to control Amazon EC2 images in the same way. + +Defining a virtual machine means creating two files (boot.conf and +install.conf) in a directory. *vosa init* has already created a +documented skeleton which you can customize as you see fit. + +#+BEGIN_SRC sh +mkdir /etc/vizrt/vosa/available.d/my-first-vm && \\ +cp /etc/vizrt/vosa/skeleton-kvm/* \\ + /etc/vizrt/vosa/available.d/my-first-vm/ +vi /etc/vizrt/vosa/available.d/my-first-vm/* +#+END_SRC + +Note that the name you choose ("my-first-vm") must be a valid +internet host name with no domain part. I.e. only lowercase +alphanumerics and hyphens. The name you choose will become the +virtual machine's host name. + +When you're happy with them you should of course track these in a +version control system, so you don't lose them. Over time, these +will become more valuable than the virtual machine images themselves. + +Make sure your IP and MAC addresses are unique, or make a script to +randomize them. + +Now, enable your virtual machine: + + : vosa -i my-first-vm enable + +This creates a symlink from available.d/my-first-vm to enabled.d, it +serves no other purpose than to differentiate between a possibly long +list of virtual machine definitions (in available.d), and the ones +you have decided to actually run on this machine. + +To install the machine, just issue the "install" command: + + : vosa -i my-first-vm install + +This will copy the disk image +to */var/lib/vizrt/vosa/images/my-first-vm/* and put some more files +in there (like the SSH private key), and finally it will boot up the +image and use the UEC's cloud-init support to prime the image and +execute any post-installation hooks you defined. + +The host name (as the machine sees it, at least) will be the same as +the name of the virtual machine; in this case "my-first-vm" + +When it's done you can SSH into the system: + + : ssh -F /var/lib/vizrt/vosa/my-first-vm/ssh.conf guest + +Not that you should need to do that, of course. + +** Making a development image + +Making a development image from the virtual machine is also possible using +vosa. The main purpose of the development image is to boost the development +time with in an production like environment. We can create VMDK (Virtual Machine Disk) +files from the UEC images used to create the virtual machine. It is also possible +to create a OVA (Open Virtual Archive) file from the generated VMDK image to use +it out of the box in VirtualBox. + +To make a VMDK image we have to run +#+BEGIN_SRC sh +vosa -i my-first-vm make +#+END_SRC + +We can create a VMDK and ova file with a single command like +#+BEGIN_SRC sh +vosa -i my-first-vm make ova +#+END_SRC + +* Puppet Master + +Setting up a puppet master is also an important piece of vosa. + +To make this possible, vosa supplies a post-install hook. This hook: + +- installs the puppet master from the apt repositories, +- configures the puppet master to use hostnames instead of its DNS + name for certificates +- configures a self signed certificate for all guests (mainly to avoid + having to sign or auto-sign the puppets, since that is problematic + when a machine is re-installed) +- creates vosa post-installation hook to set up a pre-authenticated + puppet in */etc/vizrt/vosa/puppet/-client.sh* + +This makes it possible to define more virtual machines that +automatically dance to the puppet master's tune. + +Making this useful of course means pushing your puppet configuration +into the puppet master, but that's outside the scope of this +document. + +* COPYRIGHT +Copyright 2011-2013 Vizrt + +Licensed under the Apache License, Version 2.0, see +https://github.com/vizrt/ece-scripts/COPYING for further details. + +* AUTHOR +Erik Mogsensen diff --git a/usr/share/doc/vizrt/vosa-handbook/admin-pac.org b/usr/share/doc/vizrt/vosa-handbook/admin-pac.org new file mode 100644 index 00000000..29eb7527 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/admin-pac.org @@ -0,0 +1,7 @@ +* Proxy auto-config for admin pages +Users access our servers on well known ports like 80 or 443. These ports are usually shared with the Internet without any filter. +We also use webbrowsers to view admin userinterfaces on strange ports like port 5678. To make this possible in a reasonably secure way we make use of a proxy. +Vizrt maintains an "Automatic proxy confuguration URL". As you can only have one such URL in your browser, Vizrt maintains a "PAC" resource on the Internet that contains valid settings for many of her customers. This pac URL is http://vizrtsaas.com/gk.pac. +The proxy for <%=trail_customer_shortname%> requires a password that you can see here <%=trail_secret_adminproxy%> + +See [[http://en.wikipedia.org/wiki/Proxy_auto-config][WikipediA: Proxy auto-config]] for more information on PAC. diff --git a/usr/share/doc/vizrt/vosa-handbook/amazon.org b/usr/share/doc/vizrt/vosa-handbook/amazon.org new file mode 100644 index 00000000..5c3dfbdf --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/amazon.org @@ -0,0 +1,42 @@ +* Amazon Instance +(Ignore this chapter for if you have KVM implementation) + +This chapter tells you about dealing with amazon instances in your installation. + +** Resize disk (EBS Volume) in an Amazon instance + +To resize an EBS Volume using the AWS GUI console + +1. Shut down the machine/instance. +2. Detach it from the instance, while detaching NOTE DOWN the exact device name where the volume had been attached +3. Take a snapshot of the EBS Volume +4. Go to your snapshots listing and detect your newly created snapshot and create a new volume of your desired size from the snapshot +5. You have now successfully re-sized your volume. +6. Now attach it to your instances, while attaching use the exact device name you noted while detaching, this is important while you are resizing the root device +7. Start the machine and associate elastic ip (if it had one before) +8. You will still see you volume with the previous size. You have to run resize2fs on the device to get the new size effective + #+BEGIN_SRC text + # resize2fs /dev/xvda1 + #+END_SRC + + +** Add a new EBS volume in a Amazon machine instance: + +From Amazon EC2 Management Console +1. Create the volume of desired size +2. Attach the volume as device with the desired instance (same zone as the volume) + +From the ec2 instance +1. ssh into it as root +2. format the device + #+BEGIN_SRC text + # mkfs.ext4 /dev/xvdf + #+END_SRC +3. Put the mount entry in /etc/fstab + #+BEGIN_SRC text + /dev/xvdf /var/exports auto defaults,nobootwait,comment=cloudconfig 0 2 + #+END_SRC +4. Then mount it + #+BEGIN_SRC text + # mount -a + #+END_SRC diff --git a/usr/share/doc/vizrt/vosa-handbook/backups-db.org b/usr/share/doc/vizrt/vosa-handbook/backups-db.org new file mode 100644 index 00000000..81f58ca6 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/backups-db.org @@ -0,0 +1,11 @@ +** DB backups +Daily backups of the DB is made on <%= trail_db_daily_backup_host %> +by the way of the cron script: +#+BEGIN_SRC text +/etc/cron.daily/<%= trail_db_schema %>-backup +#+END_SRC + +All the DB backups are put in: +#+BEGIN_SRC text +<%= trail_db_daily_backup_host %>:<%= trail_db_backup_dir %> +#+END_SRC diff --git a/usr/share/doc/vizrt/vosa-handbook/backups.org b/usr/share/doc/vizrt/vosa-handbook/backups.org new file mode 100644 index 00000000..9e39b1ff --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/backups.org @@ -0,0 +1,17 @@ +* Backups + +** Multimedia archive +The mutlimedia archive lives on <%= trail_nfs_master_host %> (with a +mirror on <%= trail_nfs_slave_host %> ) from which it exports its +shares to the various hosts on the network through an NFS interface. A +backup of the =/var/exports= directory on this server should suffice +as all NFS shares reside in this directory: + +#+BEGIN_SRC sh +<%= trail_nfs_master_host %>:/var/exports +#+END_SRC + +It's recommended that the backup isn't done using the NFS mount on one +of the other hosts using this NFS server, but instead target the +<%= trail_nfs_master_host %> =/var/exports= directory directly as the +backup tools would access any server directory. diff --git a/usr/share/doc/vizrt/vosa-handbook/cache-server.org b/usr/share/doc/vizrt/vosa-handbook/cache-server.org new file mode 100644 index 00000000..f220e7b9 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/cache-server.org @@ -0,0 +1,41 @@ +* Cache server + +The habitat uses Varnish caching servers. Caching servers are +sometimes referred to as reverse proxies (as opposed to normal proxies +such as Squid, which cache an office's web site traffic Squid) + +The Varnish caches are set up to do several tasks to make your web +site really fast and secure. + +** Caching +This is the obvious task of any caching server and per default, the +Varnish caches will cache static content like pictures, flash, CSS and +JS for you. + +** Security +The Varnish caches are set up to protect the following services in +your habitat: + +|----------------+------------------------------------| +| URI | What | +|----------------+------------------------------------| +| /escenic | Web Studio | +| /escenic-admin | Escenic Admin | +| /studio | Content Studio | +| /webservice | Escenic Content Engine Web service | +| /munin | The Munin overview | +|----------------+------------------------------------| + +** Useful features for debugging +Varnish in your habitat will add the following HTTP headers to add you +in your debugging of the web site: + +|------------------------+--------------------------------------------------------------------------------------| +| Example | Explanation | +|------------------------+--------------------------------------------------------------------------------------| +| X-Cache-Backend: pres2 | Which backend the cache used | +| X-Cache: HIT #68/80s | If the requested URI's content was found in cache, and how old it has been in cache. | +|------------------------+--------------------------------------------------------------------------------------| + + +#+INCLUDE: "generated-cache-server.org" diff --git a/usr/share/doc/vizrt/vosa-handbook/content-engine.org b/usr/share/doc/vizrt/vosa-handbook/content-engine.org new file mode 100644 index 00000000..b8c6e450 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/content-engine.org @@ -0,0 +1,22 @@ +* Content Engine +We differentiate between Escenic Content Engines that are presentation +servers and the ones that are publication servers. + +Architecturally, these look the same (strictly speaking, the +publication servers don't need the memory cache, but for simplicity of +uniformity, we install memcached on the publication servers too): + +[[file:./graphics/content-engine.svg][./graphics/content-engine.svg]] + +** Presentation Server +The presentation is an ECE which has the publication(s) templates and +typically doesn't have the Content Studio and Web Studio web +applications (~studio~ and ~escenic~ respectively). + +** Publication Server +A publication is an ECE which doesn't have the publication(s) +templates and serves the Content Studio and Web Studio web +applications to the editorial staff. + +The publication server is often referred to as "the editorial server" +and sometimes "the studio server". diff --git a/usr/share/doc/vizrt/vosa-handbook/create-handbook.sh b/usr/share/doc/vizrt/vosa-handbook/create-handbook.sh new file mode 100755 index 00000000..ae146fae --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/create-handbook.sh @@ -0,0 +1,807 @@ +#! /usr/bin/env bash + +# by torstein@escenic.com + +handbook_org=vosa-handbook.org +target_dir=$HOME/tmp/$(basename $0 .sh)-$(date --iso) +host_name_list_outside_network=" + amazonaws.com +" +$(which blockdiag >/dev/null) || { + cat <~${value}~g" -e "s~^'~~" ${f} + done + done +} + +function set_up_build_directory() { + if [[ -n "$target_dir" && -d $target_dir ]]; then + rm -rf $target_dir + fi + run mkdir -p $target_dir/graphics + run cp $(dirname $0)/*.{org,el} $target_dir + + # customer chapters & overrides + if [[ -n $customer_doc_dir && -d $customer_doc_dir ]]; then + run mkdir -p $target_dir/customer + + if [ $(ls $customer_doc_dir/*.org 2>/dev/null | wc -l) -gt 0 ]; then + echo "Copying chapter overides from $customer_doc_dir ..." + run cp $customer_doc_dir/*.org $target_dir/customer/ + fi + local dir=$customer_doc_dir/extra-chapters + if [ -d $dir ]; then + echo "Copying extra customer chapters from $dir" + run cp -r $dir $target_dir/customer/extra-chapters/ + fi + fi +} + +function get_blockdiag_defs() { + if [ -n "$trail_db_master_host" ]; then + echo " " $trail_db_master_host '[shape = "flowchart.database" ];' + fi + if [ -n "$trail_db_slave_host" ]; then + echo " " $trail_db_slave_host '[shape = "flowchart.database" ];' + fi + if [ -n "$trail_presentation_host_list" ]; then + for el in $trail_presentation_host_list; do + echo " " $el '[color = "orange" ];' + done + fi + if [ -n "$trail_editor_host" ]; then + echo " " $trail_editor_host '[color = "orange" ];' + fi + if [ -n "$trail_import_host" ]; then + echo " " $trail_import_host '[color = "orange" ];' + fi + if [ -n "$trail_analysis_host" ]; then + echo " " $trail_analysis_host '[color = "yellow" ];' + fi + if [ -n "$trail_db_vip_host" ]; then + echo " " $trail_db_vip_host '[shape = roundedbox];' + fi + if [ -n "$trail_nfs_vip_host" ]; then + echo " " $trail_nfs_vip_host '[shape = roundedbox];' + fi + if [ -n "$trail_lb_host" ]; then + echo " " $trail_lb_host '[shape = roundedbox];' + fi +} + +function get_blockdiag_groups() { + # serving presentation + echo " group {" + echo " internet;" + if [ -n "$trail_lb_host" ]; then + echo " $trail_lb_host;" + fi + if [ -n "$trail_presentation_host_list" ]; then + for el in $trail_presentation_host_list; do + echo " $el;" + done + fi + if [ -n "$trail_analysis_host" ]; then + echo " $trail_analysis_host;" + fi + echo ' color = "white";' + echo " }" +} + +function get_blockdiag_call_flow() { + # from the internet and to the ECEs + local lb_call_flow="internet ->" + if [ -n "$trail_lb_host" ]; then + lb_call_flow="$lb_call_flow $trail_lb_host ->" + fi + for el in $trail_presentation_host_list; do + lb_call_flow="$lb_call_flow ${el}," + done + echo " " "$(echo ${lb_call_flow} | sed 's/,$//g');" + + if [ -n "$trail_editor_host" ]; then + echo " journalist -> $trail_editor_host" \ + '[ label = "writes" ];' + fi + if [ -n "$trail_import_host" ]; then + echo " xml-feeds -> $trail_import_host" \ + '[ label = "imports" ];' + fi + + # ECEs + for el in \ + $trail_presentation_host_list \ + $trail_editor_host \ + $trail_import_host; do + local one_flow="$el ->" + + if [ -n "$trail_db_vip_host" ]; then + one_flow="${one_flow} $trail_db_vip_host," + elif [ -n "$trail_db_master_host" ]; then + one_flow="${one_flow} $trail_db_master_host," + fi + if [ -n "$trail_nfs_vip_host" ]; then + one_flow="${one_flow} $trail_nfs_vip_host," + elif [ -n "$trail_nfs_master_host" ]; then + one_flow="${one_flow} $trail_nfs_master_host," + fi + echo " " ${one_flow} | sed 's/,$/;/g' + done + + for el in $trail_presentation_host_list; do + if [ -n "$trail_analysis_host" ]; then + echo " $el -> $trail_analysis_host;" + fi + done + + # DB + if [ -n "$trail_db_vip_host" -a \ + -n "$trail_db_master_host" -a \ + -n "$trail_db_slave_host" ]; then + echo " " $trail_db_vip_host "->" \ + $trail_db_master_host"," \ + $trail_db_slave_host";" + fi + if [ -n "$trail_db_master_host" -a \ + -n "$trail_db_slave_host" ]; then + echo " " $trail_db_master_host "<->" $trail_db_slave_host \ + '[ label = "syncs" ];' + fi + + # NFS + if [ -n "$trail_nfs_vip_host" -a \ + -n "$trail_nfs_master_host" -a \ + -n "$trail_nfs_slave_host" ]; then + echo " " $trail_nfs_vip_host "->" \ + $trail_nfs_master_host"," \ + $trail_nfs_slave_host";" + fi + if [ -n "$trail_nfs_master_host" -a \ + -n "$trail_nfs_slave_host" ]; then + echo " " $trail_nfs_master_host "<->" $trail_nfs_slave_host \ + '[ label = "syncs" ];' + fi +} + +function generate_architecture_diagram() { + local file=$target_dir/graphics/architecture.blockdiag + cat > $file < /dev/null + echo "$target_dir/$(basename $handbook_org .org).html is now ready" +} + +function get_network_name() { + if [ -n "$trail_network_name" ]; then + echo ".${trail_network_name}" + else + echo "" + fi +} + +## $1 :; the host (not FQDN) +function get_fqdn() { + for el in $host_name_list_outside_network; do + if [ $(echo $1 | grep $el | wc -l) -gt 0 ]; then + echo "network outside: $el" "$1" + return + fi + done + + echo "${1}$(get_network_name)" +} + +## $1 :; the host (not FQDN) +function get_link() { + echo "http://$(get_fqdn $1)" +} + +function get_generated_overview() { + cat < $file < $file <> $file < "${appserver_host}:${appserver_port}"; +EOF + done + else + cat >> $file < "tomcat:8080"; +EOF + fi + echo "}" >> $file + +} + +## included from cache-server.org +function generate_cache_server_org() { + # if the trail_cache_host is not set, we assume the cache is running + # on (at least one o) the presentation servers. + if [ -z "$trail_cache_host" ]; then + trail_cache_host=$trail_presentation_host + fi + + generate_cache_server_diagram + local file=$target_dir/generated-cache-server.org + local svg_file=./graphics/${trail_cache_host}-cache.svg + cat > $file < $file < "${trail_db_vip_host-${trail_db_master_host-${trail_db_host-mydb}}}"; +EOF + if [ -n "$trail_db_master_host" ]; then + cat >> $file <> $file < "${trail_db_slave_host}" [ label = "syncs" ]; +EOF + fi + + if [ -n "$trail_db_vip_host" ]; then + cat >> $file < "${trail_db_master_host-${trail_db_host}}"; +EOF + fi + + echo "}" >> $file +} + +function generate_db_org() { + generate_db_diagram + + if [[ -n "${trail_db_master_host}" && -n "${trail_db_slave_host}" ]]; then + run cat $target_dir/database-changing-master.org >> $target_dir/database.org + fi + if [[ -n "${trail_db_vip_host}" && \ + -n "${trail_db_master_host}" && \ + -n "${trail_db_slave_host}" ]]; then + run cat $target_dir/database-changing-vip.org >> $target_dir/database.org + fi + if [ $(echo "${trail_db_vendor}" | \ + tr [A-Z] [a-z] | \ + grep ${DB_VENDOR_AMAZON_RDS} | \ + wc -l) -gt 0 ]; then + run cat $target_dir/database-rds.org >> $target_dir/database.org + fi +} + +function generate_nfs_org() { + if [[ -n "${trail_nfs_master_host}" && -n "${trail_nfs_slave_host}" ]]; then + local blockdiag_file=$target_dir/graphics/network-file-system-sync.blockdiag + cat > $blockdiag_file < "${trail_nfs_master_host}" [label = "reads"]; +} +EOF + local file=$target_dir/network-file-system-sync.org + local svg_file=./graphics/$(basename $blockdiag_file .blockdiag).svg + cat >> $file <> $target_dir/network-file-system.org + fi +} + +function generate_content_engine_diagram() { + local file=$target_dir/graphics/content-engine.blockdiag + local the_db="${trail_db_vendor-mysql}:${trail_db_master_port-${trail_db_port-3306}}" + + cat > $file < "memcached:11211"; + "content-engine" -> "${the_db}"; + "content-engine" -> "${trail_nfs_vip_host-${trail_nfs_master_host-${trail_nfs_host}}}:2049"; +EOF + + if [ -n "${trail_analysis_host}" ]; then + cat >> $file < "${trail_analysis_host}:${trail_analysis_port-8080}"; +EOF + fi + + if [ -n "$trail_search_host" ]; then + cat >> $file < "${trail_search_host}:${trail_search_port}/solr"; +EOF + fi + + + + cat >> $file <> $target_dir/backups.org + fi +} + +function add_customer_chapters() { + if [ $(ls $target_dir/customer 2>/dev/null | \ + grep .org$ | \ + wc -l) -gt 0 ]; then + echo "Applying customer overrides ..." + cp $target_dir/customer/*.org $target_dir + fi + + if [ $(ls $target_dir/customer/extra-chapters/ 2>/dev/null | \ + grep .org$ | \ + wc -l) -gt 0 ]; then + ( + local file=$target_dir/vosa-handbook.org + local title="Customer specific chapters" + if [ -n "$trail_network_name" ]; then + title="Special chapters for the $trail_network_name network" + fi + cat >> $file <> $file <> $file + done + fi + ) + fi +} + +function generate_virtualization_overview_org() { + if [ -z "$trail_virtualization_map" ]; then + return + fi + + local file=$target_dir/virtualization-overview-generated.org + cat > $file <> $file + for ele in $(echo "$guests" | sed 's/,/ /g'); do + echo -n " [[$(get_link $ele):5678][$(get_fqdn $ele)]] " >> $file + done + echo "|" >> $file + done + + cat >> $file <> $target_dir/overview-generated.org +} +function generate_aws_overview_org() { + if [ -z "$trail_aws_map" ]; then + return + fi + + local file=$target_dir/aws-overview-generated.org + cat > $file <> $file + for ele in $(echo "$instances" | sed 's/,/ /g'); do + echo -n " [[$(get_link $ele):5678][$(get_fqdn $ele)]] " >> $file + done + echo "|" >> $file + done + + cat >> $file <> $target_dir/overview-generated.org +} + +function get_user_input() { + local customer_doc_dir_is_next=0 + local customer_conf_is_next=0 + conf_file=$(basename $0 .sh).conf + + for el in $@; do + if [[ "$el" == "--doc-dir" || "$el" == "-i" ]]; then + customer_doc_dir_is_next=1 + elif [[ "$el" == "--conf-file" || "$el" == "-f" ]]; then + customer_conf_is_next=1 + elif [ $customer_conf_is_next -eq 1 ]; then + conf_file=$el + customer_conf_is_next=0 + elif [ $customer_doc_dir_is_next -eq 1 ]; then + customer_doc_dir=$el + customer_doc_dir_is_next=0 + fi + done +} + +echo "Building the $trail_customer_shortname VOSA Guide" +get_user_input $@ +set_up_build_directory +set_customer_specific_variables +add_customer_chapters +generate_architecture_diagram +generate_overview_org +generate_content_engine_org +generate_cache_server_org +generate_db_org +generate_nfs_org +generate_backup_org +generate_virtualization_overview_org +generate_aws_overview_org +generate_html_from_org +generate_svg_from_blockdiag +echo "done: http://start.vizrtsaas.com/${trail_customer_acronym}/" + diff --git a/usr/share/doc/vizrt/vosa-handbook/database-changing-master.org b/usr/share/doc/vizrt/vosa-handbook/database-changing-master.org new file mode 100644 index 00000000..9c17bdce --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/database-changing-master.org @@ -0,0 +1,92 @@ + +** Changing the Master DB +At install time, <%= trail_db_master_host %> was the master DB and +<%= trail_db_slave_host %> was the slave. + +If you for some reason need to change the master to the slave, for +instance if you need to take down the server for maintenance, you must +do following steps described in this section. + +*** Prepare <%= trail_db_slave_host %> to become the new master +#+BEGIN_SRC text +$ ssh root@<%= trail_db_slave_host %> +# mysql <%= trail_db_schema %> +#+END_SRC + +You must then stop the IO thread of the slave: +#+BEGIN_SRC text +mysql> stop slave io_thread; +#+END_SRC + +Next, you must wait for <%= trail_db_slave_host %> to chew through everything +that <%= trail_db_master_host %> has sent it. This information is stored in +<%= trail_db_slave_host %>'s process list. To see how it's doing, do: +#+BEGIN_SRC text +mysql> show processlist; +#+END_SRC + +What you're looking for, is for the slave to say: ~Slave has read all +relay log~. When ~show processlist~ says that, <%= trail_db_slave_host %> is +ready to become master. + +If there's no mention of the relay at all, it could mean that +<%= trail_db_slave_host %> has been replicating <%= trail_db_master_host %> at all, or +that it's slave status have been turned off some while ago. As the +administrator, you must be the judge of that. + +*** Promote <%= trail_db_slave_host %> to master +Still in <%= trail_db_slave_host %>'s mysql shell, do: +#+BEGIN_SRC text +mysql> stop slave; +mysql> reset master; +#+END_SRC + +*** Make sure the replication user is available on <%= trail_db_slave_host %> +This is the user the slave uses to replicate the master. Since +<%= trail_db_slave_host %> hasn't done any replication before, you might want +to ensure that the replication user is there. The user and password +of the replication user is typically different than that of the user +ECE uses to access the DB. + +#+BEGIN_SRC text +grant replication slave on *.* + to 'replicationuser'@'%' + identified by 'replicationpassword'; +flush privileges; +#+END_SRC + +*** Set <%= trail_db_master_host %> as slave of <%= trail_db_slave_host %> +The steps are as follows: get the binary log position from +<%= trail_db_slave_host %> and +set <%= trail_db_master_host %> to be slave of <%= trail_db_slave_host %>. + +First, get the binary log & position from <%= trail_db_slave_host %>: +#+BEGIN_SRC text +$ ssh root@<%= trail_db_slave_host %> +# mysql <%= trail_db_schema %> +mysql> show master status; +#+END_SRC + +The log file and the position are pretty much all that's outputted +from the mysql command above, so there shouldn't be anything to +confuse you. + +Next, log on to <%= trail_db_master_host %> and make it be a slave of +<%= trail_db_slave_host %>: +#+BEGIN_SRC sql +reset master; +slave stop; + +change master to + master_host='<%= trail_db_slave_host %>', + master_user='replicationuser', + master_password='replicationpassword', + master_log_file='<%= trail_db_slave_host %>-bin-file', + master_log_pos= +; + +slave start; +#+END_SRC + + + diff --git a/usr/share/doc/vizrt/vosa-handbook/database-changing-vip.org b/usr/share/doc/vizrt/vosa-handbook/database-changing-vip.org new file mode 100644 index 00000000..92b5a807 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/database-changing-vip.org @@ -0,0 +1,40 @@ +** Changing the DB Virtual IP + +In your habitat, DB clients, such as the ECEs, use the database using +the <%= trail_db_vip_host %> virtual IP (VIP). Whenever you change the master, +as described in [[Changing the Master DB]], you must also change the VIP +that the clients of the DB use. + +We have deliberately _not_ set this up as an automatic failover using +[[http://linux-ha.org][Heartbeat]] as we want DB failover to be a conscious matter. + +*** Take down the existing VIP +If <%= trail_db_master_host %> is still up, log on to it and take down its +VIP: +#+BEGIN_SRC text +$ ssh root@<%= trail_db_master_host %> +# ifconfig <%= trail_db_vip_interface %> down +#+END_SRC + +*** Set up the VIP interface on <%= trail_db_slave_host %> +You're now ready to set up the VIP on <%= trail_db_slave_host %>. You may want +to ensure that there's no one acclaiming the IP, do: +#+BEGIN_SRC text +$ ping <%= trail_db_vip_host %> +#+END_SRC + +If no one answers, you can go ahead and set it up: +#+BEGIN_SRC text +$ ssh root@<%= trail_db_slave_host %> +# ifconfig eth0:0 <%= trail_db_vip_ip %> up +#+END_SRC + +Pinging <%= trail_db_vip_host %> should now yield <%= trail_db_vip_ip %> again, +and the answer should come frmo <%= trail_db_slave_host %>: +#+BEGIN_SRC text +$ ping <%= trail_db_vip_host %> +PING <%= trail_db_vip_host %> (<%= trail_db_vip_ip %>) 56(84) bytes of data. +64 bytes from <%= trail_db_vip_ip %>: icmp_req=1 ttl=249 time=1.18 ms +#+END_SRC + + diff --git a/usr/share/doc/vizrt/vosa-handbook/database-failover.org b/usr/share/doc/vizrt/vosa-handbook/database-failover.org new file mode 100644 index 00000000..5e741bcc --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/database-failover.org @@ -0,0 +1,10 @@ +** Failing over the master DB +If <%= trail_db_master_host %> goes down and you don't get it up again, you +must change <%= trail_db_slave_host %> to become the master DB. + +*** Changes to the DB on <%= trail_db_slave_host %> + +*** Changing the VIP + +*** Changing the master back to <%= trail_db_master_host %> + diff --git a/usr/share/doc/vizrt/vosa-handbook/database-rds.org b/usr/share/doc/vizrt/vosa-handbook/database-rds.org new file mode 100644 index 00000000..e21a18c2 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/database-rds.org @@ -0,0 +1,75 @@ +<%= trail_customer_shortname %> uses Amazon RDS for MySQL for the production database. + +| Database hostname | <%= trail_db_master_host %> | +| Database instance class | <%= trail_rds_instance_class %> | +| Multi-AZ Deployment | <%= trail_rds_multiaz %> | +| Zone | <%= trail_rds_zone %> | +** General information about Amazon RDS +[[http://aws.amazon.com/rds/][RDS]] is the Amazon Web Services database as a service product. As of this writing they have the following database products available: +- [[http://aws.amazon.com/rds/mysql/][MySQL]] +- [[http://aws.amazon.com/rds/oracle/][Oracle]] +- [[http://aws.amazon.com/rds/mysql/][Microsoft SQL Server]] +As of writing we have only used MySQL for Escenic Content Engine and +Escenic Analysis Engine. +*** Amazon RDS for MySql flavors +The database sizes and pricing are comparable to equivalent EC2 instances. We have good experience running Escenic in the following RDS configuration: +- Extra Large MySQL DB Instance +- 15 Million PI +- 2 minutes full page TTL on Varnish +- 2 presentation engines, No memcached +- 20.000 presentation article cache. +We will share more experience as it becomes available. + +*** Amazon RDS for MySQL availability +RDS goes down like any other Amazon resource but it is a bit more resilient than EBS in our experience. + +Like any Amazon resource, redundancy over availability zones is needed. For this, Amazon provides the multi-az option. Our experience with RDS+Multi-az availability is good. Our experience with RDS without multi-az is horrible. Do your backups often and practice the restores because you will need it. + +*** Amazon RDS for MySQL limitations +- No shell access :: Has not been been a problem for Escenic so far. +- No triggers :: Minor upgrades of Escenic sometimes need triggers to run the database migrations. This is a minor problem as the database migration is never done in production anyway. Just remember to do the migration of databases in a restore on a normal MySQL installation in an OS that you _can_ login to. +- Different slow query log :: The slow query log is an important tool to understand database overload problems. You need to be able to investigate these problems in cases where template developers make inefficient templates or in cases where Escenic has bugs that cause it to overload the database. How to set up a slow query log on RDS/MySQL is explained later in this document. +- This is it so far :: More learnings should be added here. + +*** Restore and backup of Amazon RDS for MySql +Automated Backups are turned on by default, the automated backup feature of Amazon RDS enables point-in-time recovery for your DB Instance. Amazon RDS will backup your database and transaction logs and store both for a user-specified retention period. This allows you to restore your DB Instance to any second during your retention period, up to the last five minutes. Your automatic backup retention period can be configured to up to thirty five days. + +It is rec commended to set the retention period to at least 7 days. + +The retention period of the RDS instance at <%= trail_customer_shortname %> is <%= trail_rds_backup_retention_days %> days. + +*** Slow Query Logging on Amazon RDS for MySql - a HOWTO + +/This is borrowed text[fn:1] that still needs validation and verification/ + +[fn:1] Slow query log text from an [[http://www.memonic.com/user/chris/id/1pwgo][Article]] by [[http://www.memonic.com/user/chris/profile][Chris Hauzenberger]] on emonic.com + +Amazon RDS-MySQL instances have slow query logging disabled by default. Here is how to enable slow query logging and access the slow query data on your RDS instance using the RDS command-line tools: + +- Install the Amazon RDS command-line tools and configure your environment according to the documentation: [[http://aws.amazon.com/developertools/2928]] +- Create a parameter group: +#+BEGIN_SRC sh +$ rds-create-db-parameter-group sweet-parameter-group -f mysql5.1 -d "This is a totally sweet database parameter group" +#+END_SRC +- Place your DB instance in that parameter group: +#+BEGIN_SRC sh +$ rds-modify-db-instance my-instance \ + --db-parameter-group-name sweet-parameter-group \ + --apply-immediately +#+END_SRC +- Modify the parameter group to turn on slow query logging: +#+BEGIN_SRC sh +$ rds-modify-db-parameter-group sweet-parameter-group \ + --parameters "name=slow_query_log, value=ON, method=immediate" \ + --parameters "name=long_query_time, value=1, method=immediate" \ + --parameters "name=min_examined_row_limit, value=100, method=immediate" +#+END_SRC +- If you like, you can verify that your settings were made properly: +#+BEGIN_SRC sh +$ rds-describe-db-parameters sweet-parameter-group +#+END_SRC +- Reboot your DB instance: +#+BEGIN_SRC sh +$ rds-reboot-db-instance my-instance +#+END_SRC +After following the above steps, you will be able to access the slow query log by querying the =mysql.slow\_log= table. diff --git a/usr/share/doc/vizrt/vosa-handbook/database.org b/usr/share/doc/vizrt/vosa-handbook/database.org new file mode 100644 index 00000000..80846ed1 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/database.org @@ -0,0 +1,9 @@ +* Database + +A <%= trail_db_vendor %> DB has been installed in your habitat. + +[[file:./graphics/db.svg][./graphics/db.svg]] + + + + diff --git a/usr/share/doc/vizrt/vosa-handbook/deployment.org b/usr/share/doc/vizrt/vosa-handbook/deployment.org new file mode 100644 index 00000000..e1b7aa33 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/deployment.org @@ -0,0 +1,409 @@ +* Deployment +** Building a New EAR File and Configuration Packages +For a full deployment, two artifacts are needed: an EAR file and a +configuration package for each of the machines. The build server +provides you with both. + +*** Build from from your Hipchat room +You can start a build right from your chatroom! Go to the <%= trail_customer_shortname %> Hipchat room named: <%= trail_customer_hipchat_room_name %> or go to Hipchat in your [[https://vizrtcustomers.hipchat.com/chat][browser]]. + +When you are in the chatroom type: +#+BEGIN_SRC sh +guru: build trunk +#+END_SRC +(As of this writing it was not possible to build from a tag or a branch yet) + +The chatroom will report on the progress of your build but if you are impatient you can type: +#+BEGIN_SRC sh +guru: jobs +#+END_SRC + +Once the build is done, the URI of the finished EAR file is printed in +the chat room. This is the URI you use for the =ece deploy= command below. + +In future you will also be able to deploy an EAR to staging in the chatroom. + +*** Build from the command line on your build server +Log on to the build server as the user for the given habitat and run +the build script: + +#+BEGIN_SRC sh +$ ssh <%= trail_builder_user %>@<%= trail_builder_host %> +$ ece-build +#+END_SRC +This will build from trunk. + +If you wish to build from a tag in Subversion you do: +#+BEGIN_SRC sh +$ ece-build -t +#+END_SRC + +And to build a branch you can use the =-b= parameter instead: +#+BEGIN_SRC sh +$ ece-build -b +#+END_SRC + +Once the build is done, the URI of the finished EAR file and the +corresponding configuration packages for all the machines in your +environment are printed out in the shell: +#+BEGIN_SRC text +[ece-build-0] Starting release creation! @ <%= trail_today_date %> +[ece-build-0] Additional output can be found in /home/<%= trail_builder_user %>/ece-build.log +[ece-build-477] Generating change log for revision 6307 ... +[ece-build-515] Configuration packages available here: +[ece-build-515] http://<%= trail_builder_host %>/<%= trail_builder_user %>/releases/vosa-conf-${HOSTNAME}-1-<%= trail_builder_user %>-trunk-r6307.deb +[ece-build-515] Replace '${HOSTNAME}' with any of: [ <%= trail_presentation_host_list %> +[ece-build-515] <%= trail_editor_host %> <%= trail_staging_editor_host %> ] for the other machines' conf packages. +[ece-build-515] BUILD SUCCESSFUL! @ <%= trail_today_date %> +[ece-build-515] You'll find the release here: +[ece-build-515] http://<%= trail_builder_host %>/<%= trail_builder_user %>/releases/<%= trail_builder_user %>-trunk-rev6307-<%= trail_today_date %>_1225.ear +#+END_SRC + +The EAR and configuration package URIs can be used by the =ece deploy= command [[Making a full deployment][as described below]]. + +** Preparing to Deploy to Production +The first thing you need to do, is to [[Get the current version of the +EAR and configuration packages]] + +If the version of the EAR and vosa-conf package differ, you must find +out why (ask your colleagues) or if there are any notable changes that +has been skipped (this is normally the case, that someone has updated +the EAR without updating the conf package. However, this *should* +never happen if everyone always uses =ece-deploy=. + +*** Read Through the Change Log and Code Diff +Second, read through [[Build changelogs][all the change logs and code diffs]] for the new EAR +and conf packages. + +You will see a number of files e.g.: +#+BEGIN_SRC text +from-6100-to-6200.report +from-6100-to-6200.diff +from-6200-to-6250.report +from-6200-to-6250.diff +from-6250-to-6255.report +from-6250-to-6255.diff +#+END_SRC + +Now, if the new EAR you [[Build new EAR][built on the build server]] has revision =6255= and +the current deployed EAR has revision =6200=, you must look at the +files which contains the changes in between, namely: +=from-6200-to-6250.report=, +=from-6200-to-6250.diff=, +=from-6250-to-6255.report= and =from-6250-to-6255.diff=. + +It's important that you confirm that all JIRA issues found in the +report files are fixed. + +*** Deploy & test on staging +Log on to <%= trail_staging_editor_host %>, [[Making a full deployment][make a full deployment]] and +[[Seeing the status of all instances][ensure that your ECE, Search and EAE instances are running]] + +Once this is through, you then need to [[Performing a smoke test][perform a smoke test]] to see +that the site(s) are still basically working. + +Once this is done, ensure that all the listed JIRA issues in the +report files are tested on <%= trail_uat_url %> and closed. If you cannot +close one, make a new issue for the resulting work and close it +anyway. + +To make a link to this deployment into the CRM, copy and paste the +list of Jira issues in a support case (by email or otherwise) and make +sure that the subject contains the name of the EAR file. + +** Deploying to a production system +Deployments to production are only done by operators in the Support group of Vizrt Online in Dhaka or Oslo. + +*** Check list before you start +If you are getting ready to deploy to a production system you have to follow the next checklist: +0. Verify that you are not the same person who did the changes to the code :-) +1. Has the EAR & configuration package been properly _release_ tested? +1. Has the EAR & configuration package been properly _smoke_ tested? +2. Do the release notes match the changes made to the code and do they make sense? +3. Are the changes in the EAR causing you to feel that the service will fail after deploy? +5. Is someone you trust available to help you if you run into trouble rolling back? +6. Does the site on staging show the differences expected when reading the release notes? + +If any of these prerequisites is not in place you should refuse the deploy request and notify the user how they can convince you to perform the deploy. + +*** Steps to make the deployment +If, on the other hand, you can answer yes to all of the above, you can +go ahead and deploy on production. The steps are pretty much the same +as described in [[Deploy & test on staging]], with the exception that you +also must: + +1. First, [[Schedule downtime]] of each machine you Update +2. If the machine you're updating, you must remove it from the load + balancer (<%= trail_lb_host %>) that receives the incoming web + traffic. +3. Log on to the machine and [[Making a full deployment][make the deployment]] +4. Remove the scheduled downtime of the machine from + http://<%= trail_monitoring_host %>.<%= trail_network_name %>/icinga + +** Seeing the status of all instances +This will show the status of all ECEs, EAEs, search instances & +RMI-hub on the machine: + +#+BEGIN_SRC text +$ sudo /etc/init.d/ece status +[ece#engine-engine1] UP 0d 0h 1m 53s +[ece#search-search1] UP 0d 0h 1m 53s +[ece#analysis-analysis1] UP 0d 0h 1m 53s +#+END_SRC + +** Build changelogs +Each time <%= trail_builder_host %> builds a new EAR and configuration +packages, it also creates two files which describe all the changes +between the previous build and the current one. For each build, there +are two files: one report file with excerpts from all JIRA issues +mentioned in the commit messages and one diff file with all the code +changes. + +You can access these changelogs under http://<%= trail_builder_host %>/<%= trail_builder_user %>/changelogs +As you will see, it keeps changelogs for trunk and the different branches in separate +directories under. + +The report files contain a generated summary of the related JIRA +issues that have been worked on with this build, as well as a _risk +assessment score_. This score is calculated from the code diffs and +diff contexts. + +** Making a full deployment +Log on to the machine you want to make deployment on and use +=ece-deploy= to deploy everything: + +#+BEGIN_SRC text +$ sudo ece-deploy \ + --ear http://<%= trail_builder_host %>/<%= trail_builder_user %>/releases/<%= trail_builder_user %>-trunk-rev6307-<%= trail_today_date %>_1225.ear \ + --conf http://<%= trail_builder_host %>/<%= trail_builder_user %>/releases/vosa-conf-${HOSTNAME}-1-<%= trail_builder_user %>-trunk-r6307.deb \ + --update-publication-resources +#+END_SRC +The EAR and DEB file is what [[Building a New EAR File and Configuration Packages][you got from the build server]] + +This will deploy the new configuration, update all the publication +resources of all your publications and update all ECEs, search +instances and EAEs you have on your machine. =ece-deploy= looks in +=/etc/default/ece= to determine which instances to deploy to, just +like how =/etc/init.d/ece= decides which instances to start and stop. + +If anything goes wrong, you can just [[Rolling back to a previous version][roll back to a previous version]] + +** Rolling back to a previous version +You can roll back to any previous deployment you've done using +=ece-deploy=. To get a list of all previous deployments done with +=ece-deploy=, you do: +#+BEGIN_SRC sh +$ ssh <%= trail_presentation_host %> +$ sudo ece-deploy --list-deployments + - Deployment <%= trail_presentation_host %>-1354540403 was made @ Mon Dec 3 18:43:23 IST 2012 + - Deployment <%= trail_presentation_host %>-1354621048 was made @ Tue Dec 4 17:07:28 IST 2012 + - Deployment <%= trail_presentation_host %>-1355319440 was made @ Wed Dec 12 19:07:20 IST 2012 + - Deployment <%= trail_presentation_host %>-1355320868 was made @ Wed Dec 12 19:31:08 IST 2012 + - Deployment <%= trail_presentation_host %>-1355390454 was made @ Thu Dec 13 14:50:54 IST 2012 +#+END_SRC + +Normally, the previous one will be the right one to roll back to, but +if you've played a lot back and forth If you don't know which one to +choose, then pick the one that's fairly recent and has been running +for a long time, i.e., there's a long span between that deployment and +the next one. + +From the output above, we see that the one from the 4th of December +has been running the longest, so we roll back to that one with a +simple command: +#+BEGIN_SRC sh +$ sudo ece-deploy \ + --rollback <%= trail_presentation_host %>-1354621048 \ + --update-publication-resources +#+END_SRC + +The reason why =ece-deploy= has its own deployment ID and doesn't use +the version of the EAR & configuration package, is that it's possible +to make several deployments of the same EAR/configuration package, +even on the same host. Furthermore, =ece-deploy= deploys on several +instances, not only one. And lastly, it's even possible to choose +whether or not to update the publication resources. Hence, +=ece-deploy= has its own IDs and database of its deployments to make +everything reproduce-able. + +In this connection, it should also be noted that each of the ECE +instances also have their own [[Instance deployment log]] + +** Performing a smoke test +The command below will call the local ECE with +=Host= header set and output the amount of bytes returned. If this +number is less than a few thousand, you should immediately investigate +why. Also, we check that there's a == element returned from +the front page of each of the domains: +#+BEGIN_SRC text +$ for host in <%= trail_virtual_host_list %>; do \ + echo "${host}'s title:" + curl --silent --header "Host: $host" http://localhost:8080/ | grep -A 1 '<title>'; \ + echo "${host}'s front page bytes:"; \ + curl --silent --header "Host: $host" http://localhost:8080/ | wc -c; \ + done +#+END_SRC + +** Manually deploying a new EAR file to an ECE instance +We strongly recommend that you [[Making a full deployment][use ece-deploy to deploy a new EAR +file]]. If you only want to deploy the EAR and not the configuration +package, you can just call =ece-deploy= without the =--conf= +parameter. + +However, if you for some reason, perhaps you don't have root privileges on the machine, and want to deploy an EAR to a specific instance, you can use =ece deploy= (note that =ece-deploy= is different from =ece deploy=): + +#+BEGIN_SRC sh +$ ece -i engine1 \ + --uri http://<%= trail_builder_host %>/<%= trail_builder_user %>/releases/<%= trail_builder_user %>-trunk-rev4121-<%= trail_today_date %>_1524.ear \ + deploy \ + restart +#+END_SRC + +You can confirm that the instance came up again by querying =ece -i +engine1 info | grep -i EAR= or looking in the [[Instance deployment log][deployment log for the instance]] +to see that the new EAR has been deployed. + +** Manually deploying a new EAR file to a search instance +Again, we recommend you using =ece-deploy= for this, but if you really +want to do it explicitly for a search instance, this is the same as +[[Manually deploying a new EAR file to an ECE instance]] except that you +must add =-type search= to the =ece= command: + +#+BEGIN_SRC sh +$ ece -i search1 \ + -t search \ + --uri http://<%= trail_builder_host %>/<%= trail_builder_user %>/releases/<%= trail_builder_user %>-trunk-rev4121-<%= trail_today_date %>_1524.ear \ + deploy \ + restart +#+END_SRC + +** Instance deployment log +Each of the ECE, EAE and search instances also have their own +deployment log where the EAR used whenever running +=ece -i <%= trail_presentation_host %> deploy= along with its MD5 sum and the date +of deployment is available: +#+BEGIN_SRC sh +$ ece -i engine1 list-deployments +[ece#engine-engine1] These are all the deployments made on engine1: +Wed Dec 12 19:11:39 IST 2012 <%= trail_customer_acronym %>-trunk-rev6259-2012-10-12_1322.ear c6c7643234asdfasdfdf7f7f0001612e +Wed Dec 12 19:31:35 IST 2012 <%= trail_customer_acronym %>-trunk-rev6260-2012-12-12_1401.ear c6c762523db66ae21cdf7f7f00016f7f +#+END_SRC +This log file is automatically updated when you use the =ece-deploy= command. + +** Updating Server Configuration +*** Make changes to the =server-admin= tree +In the <%= trail_builder_user %> source tree, there is a directory +called =server-admin=. This contains all the files that are hand +crafted because the file values cannot be generated by simply running +=ece-install= with the correct parameters. + +The structure is as follows: =server-admin/<common|<machine>>/<full +file path>=. Below are some examples to help illustrate how to use +this file tree: + +#+BEGIN_SRC text +(1) server-admin/common +(2) server-admin/common/etc/hosts.d +(3) server-admin/<%= trail_presentation_host %>/etc/escenic/ece-engine1.conf +(4) server-admin/<%= trail_db_master_host %>/etc/mysql/my.cnf +#+END_SRC +|------+-----------------------------------------------------------------------------------------| +| Path | Description | +|------+-----------------------------------------------------------------------------------------| +| (1) | Common files for all machines. | +| (2) | Files that together generate the =/etc/hosts= when you [[Building new configuration packages]] | +| (3) | The =/etc/escenic/ece-engine1.conf= specific for <%= trail_presentation_host %> | +| (4) | The =/etc/mysql/my.cnf= specificf for the <%= trail_db_master_host %> machine. | +|------+-----------------------------------------------------------------------------------------| + +There will always be _some_ files in your =server-admin= tree, but as +a rule of thumb, try to keep this to a minimum. + +=ece-install= (and the OS package of course) should provide sensible +defaults for most components given that you pass it the appropriate +settings in the machine's =ece-install.conf=, so ultimately, you'd +only have to check in the =ece-install.conf= for the +<%= trail_control_host %> machine so that it's able to install the +other machines, plus the appropriate file(s) in +=server-admin/common/etc/hosts.d=. + +Let's say we want to change the memory setting in =ece-engine1.conf= +for the =<%= trail_presentation_host %>= machine only. Go to your +checked out <%= trail_builder_user %> source code and edit the file +(or indeed add it if it's not already there, in which case would mean +that you're running with the defaults set up by =ece-install=): + +#+BEGIN_SRC text +$ vi ~/src/<%= trail_builder_user %>/server-admin/<%= trail_presentation_host %>/etc/escenic/ece-engine1.conf +#+END_SRC + +Make your changes and then commit them using an appropriate ticked ID +in the log message, e.g.: +#+BEGIN_SRC sh +$ svn ci ~/src/<%= trail_builder_user %>/server-admin/<%= trail_presentation_host %>/etc/escenic/ece-engine1.conf \ + -m "<%= trail_builder_user %>-344: increased the max and min heap sizes to 4GB because we've got so many objects" +#+END_SRC + +That's it, your changes will be included in all the relevant +configuration packages when you [[Building a New EAR File and Configuration Packages][issue a new build]]. + +*** Deploying a Configuration Package +Log on to the different hosts and call =ece-deploy= with the =--conf= +parameter to install the package (you normally do this together with +the EAR file, but for the sake of the example, you /can/ just deploy +the conf package): Here, we use <%= trail_presentation_host %> as an +example: + +#+BEGIN_SRC text +$ ssh <%= trail_presentation_host %> +$ sudo ece-deploy \ + --conf http://<%= trail_builder_host %>/<%= trail_builder_user %>/releases/vosa-conf-<%= trail_presentation_host %>-1-<%= trail_builder_user %>-trunk-r4899.deb +#+END_SRC + +Because of the mighty =dpkg= and the =DEB= package format, you'll get +prompted for any abnormalities, like if someone has changed any of the +conf package files locally since you last updated the package, if +you've got other, conflicting configuration packages installed on so +on. + +Now, you have full control over your configuration being in sync with your EAR deployment. You can easily confirm this, see +[[Get the current version of the EAR and configuration package]] + +** Get the current version of the EAR and configuration package +There are two ways to see which EAR file is currently deployed, the easiest is: +#+BEGIN_SRC text +$ ece -i engine1 list-deployments | tail -1 +Thu Dec 13 15:05:41 IST 2012 <%= trail_customer_acronym %>-trunk-rev6260-2012-12-12_1401.ear c6c762523db66ae21cdf7f7f00016f7f +#+END_SRC +The second way, is to search in the output from the =ece info= command: +#+BEGIN_SRC text +$ ece -i engine1 info | grep -A 1 EAR +[ece#engine-engine1] |-> EAR used: +[ece#engine-engine1] http://<%= trail_builder_host %>/<%= trail_builder_user %>/releases/<%= trail_builder_user %>-trunk-rev4899-<%= trail_today_date %>_1524.ear +#+END_SRC + + +To see which version of the system configuration is deployed on the machine, do: +#+BEGIN_SRC text +$ dpkg -l vosa-conf-<%= trail_presentation_host %> | grep ^ii +ii vosa-conf-<%= trail_presentation_host %> 1-<%=trail_customer_acronym %>-trunk-r6260 Server configuration for <%= trail_presentation_host %> +#+END_SRC +This version should correspond to the EAR version. If not, you should +ask around to the other operators to find out why these +differ. Normally, these two should always be in sync. +** Schedule downtime +Whenever you're going to make changes that you know or fear will +disrupt services so that you'll activate the monitoring system's +checks, you should schedule the downtime so that the monitoring server +is on your team and doesn't [[http://en.wikipedia.org/wiki/Cry_Wolf][cry wolf]]. + +You can either schedule down time of a particular machine by using the +web interface at http://<%= trail_monitoring_host %>.<%= trail_network_name %>/icinga or by +logging on to <%= trail_control_host %> and use the command +=downtime=: + +#+BEGIN_SRC sh +$ ssh <%= trail_control_host %> +$ echo "Upgrading <%= trail_presentation_host %> to fix caching problem" | \ + downtime -i <%= trail_presentation_host %> 1 hours +#+END_SRC diff --git a/usr/share/doc/vizrt/vosa-handbook/development-process.org b/usr/share/doc/vizrt/vosa-handbook/development-process.org new file mode 100644 index 00000000..d6187d53 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/development-process.org @@ -0,0 +1,400 @@ +* Development Process + +This chapter tells you how to install and set up your development +environment, and then describes how you can use the +tools in the environment to carry out everyday development tasks. + +The main components of the VOSA development environment are: + + - Apache Subversion :: All your project code is stored here, under strict + version control. + - Apache Maven :: Apache Maven provides dependency management and + controls the SW build process. + - IntelliJ IDEA Ultimate :: Your editor and coding environment. + - VirtualBox :: Lets you run a complete + copy of your production site in a virtual + machine on your PC, making it easy to test your code changes + as you make them. + - HipChat :: A chat room service that you can use to communicate with + other members of your project, stakeholders in the + project (customers, users and so on) and Vizrt support + staff. The chat room for your project also has a special + member called Guru. Guru is a robot who will do various + jobs for you such as building and deploying new versions + of the site. + - JIRA :: A bug tracking service that is used to track all work done + in the project. All changes to be made are registered as + change requests in JIRA, and Subversion will only allow you + to check in changes that have a valid JIRA ticket number. + +** Installation + +You need to install the following items on your PC or Mac: + + - The Oracle Java Development Kit, version 6 or 7. You can get this + from [[http://www.oracle.com/technetwork/java/javase/downloads/jdk6u37-downloads-1859587.html]] + and [[http://www.oracle.com/technetwork/java/javase/downloads/java-se-jdk-7-download-432154.html]] + - Apache Subversion client software. You can get this from + [[http://subversion.apache.org/packages.html]] (or on Linux you can + most likely install it using your package manager). + - Apache Maven version 3. You can get this from + [[http://maven.apache.org/download.html]] (or on Linux you can + most likely install it using your package manager). + - IntelliJ IDEA Ultimate, version 11.x. You can download this from + [[http://www.jetbrains.com/idea/download/]]. Be sure to download the + Ultimate version, which is licensed software. Depending on the + licensing arrangement at your site you may need to enter a license + number after installation - talk to your project manager for more + information. Note that version 11.x is required: earlier versions of IntelliJ IDEA do not + contain all the functions required by the VOSA development process + and will not work properly. + - Oracle VirtualBox. You can get this from [[https://www.virtualbox.org/wiki/Downloads]] + (or on Linux you can most likely install it using your package manager). You + can use another virtual machine hypervisor such as VMWare if you prefer, but this + documentation assumes you are using VirtualBox. + +During the IntelliJ IDEA installation process you are allowed to +choose which plug-ins to activate. You should activate at least the +following plug-ins (although you may find you have use for others): + + - Maven integration plug-in + - Subversion integration plug-in + +All of the above products have their own installation +procedures. Follow the relevant procedures for your operating system, +and make sure they are correctly installed before continuing with the +set-up of your development environment. + +** Set-up + +Setting up your development environment involves: + + - Configuring the SW you have installed + - Obtaining copies of the code and systems you are going to work on + - Getting access to the project infrastructure (HipChat, Jira and so on) + +*** Set up your local test system + +You will have your own test copy of the site you are working on. It is a +complete working copy of the production server and runs in a virtual machine +on your PC. It is supplied as an Ubuntu Enterprise Cloud Image, modified to boot +under almost any hypervisor (VirtualBox, for example). + +The supplied image consists of an Ubuntu server OS, with the following SW installed +and preconfigured: + +- Java +- Escenic Content Engine +- Varnish cache (like the production server) +- Memcached (like the production server) +- Solr running in a separate Java VM (like the production server) +- A publication and .ear file from your project + +The system is configured with two virtual network cards (one for outbound NAT traffic +and the other for inbound traffic), and has remote debugging enabled on port 5005. + +Click on this link to download the virtual machine image: + +[[<%= trail_dev_image_uri %>]] + +When the file has finished downloading, you can open it with your chosen +hypervisor. If you are using VirtualBox then the procedure for +installing the image is as follows: + +1. Start VirtualBox and open the downloaded <%= trail_dev_image_name %> file. + +2. VirtualBox recognizes the format and displays an import dialog. Click *Import*. + +3. A license agreement dialog is displayed. Click *Agree* to continue. + +4. When the image file has been imported, the VirtualBox Manager window is displayed, + and the imported image is displayed in a list. Select the image and click on + *Settings*. + +5. The settings page displays various parameters that you can use to configure the + virtual machine. Unless you know that you have specific changes that you want to make regarding + memory allocation, number of CPUs etc., the default settings should be good enough + so you can just click *OK* to accept them. + +6. Make sure your image is still selected and click *Start* to start the virtual machine. + +A terminal window is displayed showing the virtual machine boot-up messages. When the boot sequence +has finished, the terminal window will contain an information page starting with the +following welcome message: + +#+BEGIN_SRC default +Welcome to the <%= trail_dev_host_name %> development environment +#+END_SRC + +Below this welcome messages are the IP addresses used by the virtual +machine and the user name/password combinations you can use to log in +to the system. + +In order to be able to use the *<%= trail_dev_host_name %>* +URLs, you need to add the host name *<%= trail_dev_host_name %>* to +your computer's =hosts= file. To do this: + +1. Open your hosts file in a text editor. On a Windows PC, you will find the + hosts file at =C:\Windows\System32\drivers\etc\hosts=. On a Mac or Linux + machine you will find it at =/etc/hosts=. +2. Add the following line to the file: + #+BEGIN_SRC default + <ip-address> <%= trail_dev_host_name %> + #+END_SRC + where =<ip-address>= is the =eth1= IP address listed on the virtual machine's welcome page. +3. Save your changes. + +Your test system is now up and running. You should be able to verify your system by opening a browser +and navigating to [[http://<%= trail_dev_host_name %>:8080/escenic-admin/status.jsp?tests=all]]. + +*** Verify available services and publications + +- With the image runing you should now be able to access the following services: +|---------------------+---------------------------------------------------------------| +| Service | URL | +|---------------------+---------------------------------------------------------------| +| Escenic Admin | http://<%= trail_customer_acronym %>-dev:8080/escenic-admin | +|---------------------+---------------------------------------------------------------| +| Web Studio | http://<%= trail_customer_acronym %>-dev:8080/escenic | +|---------------------+---------------------------------------------------------------| +| Content Studio | http://<%= trail_customer_acronym %>-dev:8080/studio | +|---------------------+---------------------------------------------------------------| +| Solr | http://<%= trail_customer_acronym %>-dev:8180/solr | +|---------------------+---------------------------------------------------------------| + +You should also be able to access the following publications: + +|--------------------+------------------------------------+-----------------------------------------| +| Publication | username / password | URL | +|--------------------+------------------------------------+-----------------------------------------| +| <publication-name> | <publication-name>\_admin / admin | [[http://local.<publication-name>.com]] | +|--------------------+------------------------------------+-----------------------------------------| + + +*** Configure Maven + +You need to add a reference to the Vizrt software repository to your +Maven settings files. To do this: + +1. Open the Maven settings file for editing. On Windows you will find + it at =??=. On Mac and Linux you will find it at + =/home/<your-user>/.m2/settings.xml=. +2. Add a repository definition like the one shown below to the file: + #+BEGIN_SRC xml + <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 + http://maven.apache.org/xsd/settings-1.0.0.xsd"> + <profiles> + <profile> + <id>default</id> + <activation> + <activeByDefault>true</activeByDefault> + </activation> + <repositories> + <repository> + <id>escenic-repo</id> + <name>Oslo Releases</name> + <url>http://repo.dev.escenic.com/content/groups/trunk</url> + <layout>default</layout> + </repository> + </repositories> + </profile> + </profiles> + </settings> + #+END_SRC + (The above example shows the minimum content required for <%= trail_dev_project_name %> to + work. Your =settings.xml= may of course contain other settings. For a complete description of + how to use =settings.xml=, see http://maven.apache.org/settings.html.) + TODO Is the above example correct? +3. Save your changes. + +*** Check out your project + +To check out your project: + +1. Start Intellij. +2. Click on the *Check out from Version Control* link. +3. If a list of version control systems is displayed, select Subversion. +4. Click on the + icon to add a new repository. +5. Paste this: + #+BEGIN_SRC default + https://vizrtcustomers.jira.com/svn/<%= trail_dev_project_name %> + #+END_SRC + into the displayed dialog, and click on *OK*. +6. Select the repository you have added to the list and select *Checkout*. +7. Choose/create a destination folder for the project and click *OK* three times. +8. When asked if you want to create an IntelliJ project for the source files, select + *No*. + +*** Set up your project + +To set up your project: + +1. Click on the *Open project* link. +2. In the *Open Project* dialog, open the project's =trunk= folder (or one of its =branches/n.n= + folders if that is where you are going to work). +3. Select the =pom.xml= file in the folder and click *OK*. +4. Select *File* > *Settings...*. +5. In the displayed *Settings* dialog, select *Maven* > *Importing*. +6. Make sure that the *Import Maven projects automatically* option is checked and click *OK*. +7. Select *Tools* > *Deployment* > *Configuration*. +8. In the displayed *Deployment* dialog, click on the + icon. +9. Enter a name for your deployment set-up in the *Name* field, *SFTP* in the *Type* field and + click on *OK*. +10. On the *Connections* tab of the displayed form, enter *<%= trail_dev_host_name %>* in + the *SFTP Host* field, *escenic* in the *User name* field and the corresponding password + in the *Password* field. +11. Click on *Test SFTP Connection...* to check you have entered the correct login credentials. +12. On the form's *Mappings* tab, enter the following two mappings: + + | Local path | Deployment path | + |-----------------------------------------------------------------------------+-----------------------------------------------------------------------------| + | /<path-from-root>/publications/shared-war/src/main/webapp/template/ | /opt/tomcat-engine1/webapps-<publication-name>/<publication-name>/template/ | + | /<path-from-root>/publications/<publication-name>/src/main/webapp/template/ | /opt/tomcat-engine1/webapps-<publication-name>/<publication-name>/template/ | + +13. Click *OK*. +14. Select *Tools* > *Deployment* > *Automatic upload*. + +*** Test automatic deployment + +Any changes you make to your project should now be automatically deployed to the correct location +on your test server. To test that this is actually the case: + +1. Display your test server's file system by selecting *Tools* > *Deployment* > *Browse Remote Host*. +2. Navigate down the displayed tree to show the contents of the =/opt/tomcat-engine1/webapps-<publication-name>/<publication-name>/template/widgets= folder. + (=/opt/tomcat-engine1/webapps-<publication-name>/<publication-name>/template/= and all its children should be highlighted in green. If this is not the case, + go back and and check that you entered the deployment path correctly in the *Deployment* dialog.) +3. In your local code tree, navigate to publications/<publication-name>/src/main/webapp/template/widgets/dummy. +4. Create a file (any name) in this folder. The dummy folder and the file you have created should immediately be duplicated in the deployment tree. If this is not the case, + go back and and check that you entered the correct local paths in the *Deployment* dialog. +5. Delete the test file you created. It should also disappear from the deployment tree. + + +** Procedures + +You have now installed and configured all the local components of your development +environment. However, your development environment consists of many other systems +in your local network and on the Internet, and also includes the people you will be working +with. This section provides: + + - An overview of how all these systems and people fit together + - Some suggested procedures for making the best use of your environment + +*** Change management + +VOSA aims to provide a development environment that is organized but flexible +and above all *transparent*. It should be easy for any interested party to find +out what is going on at any time: + + - What tasks have been carried out, what tasks are in progress, what + tasks are planned. + - Who is involved in carrying out the tasks. + - Why the tasks are being carried out (what requirements they are intended to satisfy). + - Where the task is in the development cycle. + +The environment is designed to support agile development methodologies such as +[[http://en.wikipedia.org/wiki/Scrum_(development)][Scrum]] and is centered around a web-based bug-tracking and change +management system called *Jira*. Every VOSA project +has a corresponding Jira project at [[https://vizrtcustomers.jira.com]]. All the work +to be carried out in the VOSA project must be registered as *issues* in this Jira project. + +Jira supports a hierarchy of issues so that high level requirements (called *epics* in +Scrum terminology) can be broken down into smaller requirements (*stories*) and the individual +development tasks that you will actually work with. Jira tracks the progress of these +tasks (plus bug correction tasks, which are also registered as issues) through a lifecycle +of development work, testing, error correction and release. It keeps track of who is working +on the tasks and mantains a log of all code changes associated with them, plus comments, notes +and discussions. + +Jira is a large and very powerful change management system that can be used in many different +ways. You will need to find your way around at least some parts of it, and you are recommended +to make use of its help facilities and documentation to do so. As a VOSA developer, +however, the most important points you need to understand are: + + - Jira is where your work comes from :: If you haven't got the number of a Jira issue describing + the work you are doing, then you should not be writing or modifying any code. + - Jira needs to know how you're getting on :: When you are satisfied enough with a change to + check it in to Subversion, then you must include the issue number in your check-in comment. + The check-in is then recorded in the issue. Add notes and comments to the issue as you go + along so it contains a complete record of decisisons and changes made along the way. When + you are finished with the task, record the fact in Jira by *resolving*?? the issue. + +You can access your Jira project(s) from: + +[[http://start.vizrtsaas.com/<%= trail_dev_project_name %>]] + +*** Version control + +All code (JSP, HTML, XML, Java, Javascript, CSS etc.), documentation and other resources (images, videos, +sound, configuration files) in a VOSA project is stored in a Subversion repository. The code +you work with on your machine is only a copy of the code stored in Subversion, and you cannot +build or deploy your changes for others to see until you have checked them back in to Subversion. + +Subversion is a very widely-used version control system and there are plenty of resources +available on the Internet for learning how to use it. The Subversion client you have +installed only has a command line user interface. If you are not comfortable with using +that, there are [[http://en.wikipedia.org/wiki/Comparison_of_Subversion_clients][many]] graphical front ends available for it. In +addition, most of the simple everyday operations can be carried out + +using IntelliJ's built-in Subversion support. + +Note that VOSA's Subversion server is set up to prevent you checking in any changes +without a Jira issue number in your check-in comment. + + +*** TODO Communicating with your colleagues + +(about using HipChat) + + +*** TODO Development + +The <%= trail_dev_project_name %> publication(s) is/are made using the *Escenic Widget Framework*. This means that +all publication layout and functionality is assembled from *widgets*. A widget is a package of JSP, CSS and graphics files +that together provide a web site component. A component may be primarily graphical (such as the =storyContent= widget that +governs the layout of a story in a publication), primarily functional (such as the =webAnalytics= widget) or a combination +of the two (such as the =navigation= widget). + +Escenic Widget Framework is supplied with a comprehensive set of ready-made widgets from which web site designers can +construct web sites using a point and click interface in *Content Studio*, Vizrt's web site editor. The widgets can also +be customized using this interface, so a wide range of different web sites can be constructed without ever needing to write +any JSP, HTML or CSS code. + +If the standard widgets do not provide all the functionality you need you can extend the Widget Framework +in two ways: + + - By adding widgets of your own + - By customizing existing widgets + +This manual does not cover the details of Widget Framework development, since this is covered elsewhere +([[http://documentation.vizrt.com/widget-framework-2.0.html]]). + +The development environment you have set up is designed to support and simplify the process of extending +the standard widgets supplied with the Widget Framework. The =publications/<publication-name>/src/main/webapp/template/= +tree is a *customization layer* that you can use to store any modifications you want to make to the standard widgets +in the =publications/shared-war/src/main/webapp/template/= tree. During the application build process, +the files in your customization layer are merged with the standard widgets to create a customized widget +set for deployment. + +The deployment mappings you have set up mimic this process: any changes you make in +=publications/<publication-name>/src/main/webapp/template/= are instantly copied to your development image. T +his means you can instantly test all changes you've made by using the browser to request the appropriate page from the +publication on your development image. + +**** TODO Customizing a widget + +**** TODO Creating a new widget + +*** TODO Building and deployment + +(about using guru to do it for you) + +** TODO Jira + +** TODO Source code repository + +** TODO The builder + + + diff --git a/usr/share/doc/vizrt/vosa-handbook/glossary.org b/usr/share/doc/vizrt/vosa-handbook/glossary.org new file mode 100644 index 00000000..a3d61cd6 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/glossary.org @@ -0,0 +1,15 @@ +* Glossary + +# Tip 1: if you use Emacs to edit this file, you can hit C-c C-c to +# re-format/structure the table. +# +# Tip 2: if you use Emacs to edit this file, you can get alphabetical +# sorting of the entries in this table if you mark the entries and +# then do: M-x sort-lines + +|----------+----------------------------------------------------------------------------------------------| +| Term | Explanation | +|----------+----------------------------------------------------------------------------------------------| +| Habitat | Collection of machines which makes up an environment, such as testing, staging & production. | +| Trending | Seeing the behavioural statistics of a server over longer amount of time | +|----------+----------------------------------------------------------------------------------------------| diff --git a/usr/share/doc/vizrt/vosa-handbook/graphics/cygwin-openssh.png b/usr/share/doc/vizrt/vosa-handbook/graphics/cygwin-openssh.png new file mode 100644 index 00000000..0ef96a0c Binary files /dev/null and b/usr/share/doc/vizrt/vosa-handbook/graphics/cygwin-openssh.png differ diff --git a/usr/share/doc/vizrt/vosa-handbook/graphics/pagent.png b/usr/share/doc/vizrt/vosa-handbook/graphics/pagent.png new file mode 100644 index 00000000..b5b53ab0 Binary files /dev/null and b/usr/share/doc/vizrt/vosa-handbook/graphics/pagent.png differ diff --git a/usr/share/doc/vizrt/vosa-handbook/graphics/puttygen1.png b/usr/share/doc/vizrt/vosa-handbook/graphics/puttygen1.png new file mode 100644 index 00000000..0359c4c1 Binary files /dev/null and b/usr/share/doc/vizrt/vosa-handbook/graphics/puttygen1.png differ diff --git a/usr/share/doc/vizrt/vosa-handbook/graphics/puttygen2.png b/usr/share/doc/vizrt/vosa-handbook/graphics/puttygen2.png new file mode 100644 index 00000000..b9b5814f Binary files /dev/null and b/usr/share/doc/vizrt/vosa-handbook/graphics/puttygen2.png differ diff --git a/usr/share/doc/vizrt/vosa-handbook/graphics/puttygen3.png b/usr/share/doc/vizrt/vosa-handbook/graphics/puttygen3.png new file mode 100644 index 00000000..cf5f19cd Binary files /dev/null and b/usr/share/doc/vizrt/vosa-handbook/graphics/puttygen3.png differ diff --git a/usr/share/doc/vizrt/vosa-handbook/graphics/puttygen4.png b/usr/share/doc/vizrt/vosa-handbook/graphics/puttygen4.png new file mode 100644 index 00000000..8df7282e Binary files /dev/null and b/usr/share/doc/vizrt/vosa-handbook/graphics/puttygen4.png differ diff --git a/usr/share/doc/vizrt/vosa-handbook/graphics/vmeo-browser-playing-video.blockdiag b/usr/share/doc/vizrt/vosa-handbook/graphics/vmeo-browser-playing-video.blockdiag new file mode 100644 index 00000000..8670c1be --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/graphics/vmeo-browser-playing-video.blockdiag @@ -0,0 +1,15 @@ +blockdiag { + orientation = "portrait"; + + browser [shape = "actor"]; + browser -> video-widget [label="gets"]; + content-engine [label="content-engine", color="orange"]; + adactus-control [label="vmeo-controller", color="green"]; + adactus-streaming [label="vmeo-streaming", color="cyan"]; + + video-widget [label="video-widget", color="brown"]; + video-widget -> cache-server -> content-engine -> adactus-control [label="gets XML"]; + browser -> adactus-streaming [label="streams"]; + + +} diff --git a/usr/share/doc/vizrt/vosa-handbook/graphics/vmeo-browser-playing-video.svg b/usr/share/doc/vizrt/vosa-handbook/graphics/vmeo-browser-playing-video.svg new file mode 100644 index 00000000..4242a0a3 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/graphics/vmeo-browser-playing-video.svg @@ -0,0 +1,67 @@ +<?xml version='1.0' encoding='UTF-8'?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.0//EN" "http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd"> +<svg viewBox="0 0 448 440" xmlns="http://www.w3.org/2000/svg" xmlns:inkspace="http://www.inkscape.org/namespaces/inkscape" xmlns:xlink="http://www.w3.org/1999/xlink"> + <defs id="defs_block"> + <filter height="1.504" id="filter_blur" inkspace:collect="always" width="1.1575" x="-0.07875" y="-0.252"> + <feGaussianBlur id="feGaussianBlur3780" inkspace:collect="always" stdDeviation="4.2" /> + </filter> + </defs> + <title>blockdiag + blockdiag { + orientation = "portrait"; + + browser [shape = "actor"]; + browser -> video-widget [label="gets"]; + content-engine [label="content-engine", color="orange"]; + adactus-control [label="vmeo-controller", color="green"]; + adactus-streaming [label="vmeo-streaming", color="cyan"]; + + video-widget [label="video-widget", color="brown"]; + video-widget -> cache-server -> content-engine -> adactus-control [label="gets XML"]; + browser -> adactus-streaming [label="streams"]; + + +} + + + + + + + + + + + + video-widget + + vmeo-streaming + + content-engine + + vmeo-controller + + cache-server + + + + gets + + + + + + streams + + + + gets XML + + + + gets XML + + + + gets XML + diff --git a/usr/share/doc/vizrt/vosa-handbook/graphics/vmeo-import-to-ece.blockdiag b/usr/share/doc/vizrt/vosa-handbook/graphics/vmeo-import-to-ece.blockdiag new file mode 100644 index 00000000..7a567446 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/graphics/vmeo-import-to-ece.blockdiag @@ -0,0 +1,17 @@ +blockdiag { + orientation = "landscape"; + + content-engine [label="content-engine", color="orange"]; + video-widget [label="video-widget", color="brown"]; + + video.xml [label="video.xml"]; + key-frames [label="key-frames"]; + adactus-control [label="vmeo-controller", color="green"]; + + content-engine -> video.xml [label="imports"]; + video.xml -> adactus-control; + + content-engine -> key-frames [label="imports"]; + key-frames -> adactus-control [label="gets"]; + content-engine -> video-widget [label="renders"]; +} diff --git a/usr/share/doc/vizrt/vosa-handbook/graphics/vmeo-import-to-ece.svg b/usr/share/doc/vizrt/vosa-handbook/graphics/vmeo-import-to-ece.svg new file mode 100644 index 00000000..5b4fa4ab --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/graphics/vmeo-import-to-ece.svg @@ -0,0 +1,67 @@ + + + + + + + + + blockdiag + blockdiag { + orientation = "landscape"; + + content-engine [label="content-engine", color="orange"]; + video-widget [label="video-widget", color="brown"]; + + video.xml [label="video.xml"]; + key-frames [label="key-frames"]; + adactus-control [label="vmeo-controller", color="green"]; + + content-engine -> video.xml [label="imports"]; + video.xml -> adactus-control; + + content-engine -> key-frames [label="imports"]; + key-frames -> adactus-control [label="gets"]; + content-engine -> video-widget [label="renders"]; +} + + + + + + + + content-engine + + video-widget + + video.xml + + key-frames + + vmeo-controller + + + + renders + + + + + + imports + + + + + + imports + + + + + + + + gets + diff --git a/usr/share/doc/vizrt/vosa-handbook/graphics/vmeo-mam-to-vmeo.blockdiag b/usr/share/doc/vizrt/vosa-handbook/graphics/vmeo-mam-to-vmeo.blockdiag new file mode 100644 index 00000000..58090669 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/graphics/vmeo-mam-to-vmeo.blockdiag @@ -0,0 +1,14 @@ +blockdiag { + orientation = "landscape"; + + media-asset-management [label="MAM" color="yellow"]; + adactus-control [label="vmeo-controller", color="green"]; + adactus-streaming [label="vmeo-streaming", color="cyan"]; + + video.raw -> media-asset-management [label="copied"]; + media-asset-management -> adactus-control [label="triggers"]; + adactus-control -> video.mp4, video.ogg [label="encodes"]; + video.mp4, video.ogg -> adactus-streaming [label="uploads"]; + adactus-control -> video.xml [label="exports"]; + adactus-control -> key-frames [label="creates"]; +} diff --git a/usr/share/doc/vizrt/vosa-handbook/graphics/vmeo-mam-to-vmeo.svg b/usr/share/doc/vizrt/vosa-handbook/graphics/vmeo-mam-to-vmeo.svg new file mode 100644 index 00000000..1d2c57e5 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/graphics/vmeo-mam-to-vmeo.svg @@ -0,0 +1,89 @@ + + + + + + + + + blockdiag + blockdiag { + orientation = "landscape"; + + media-asset-management [label="MAM" color="yellow"]; + adactus-control [label="vmeo-controller", color="green"]; + adactus-streaming [label="vmeo-streaming", color="cyan"]; + + video.raw -> media-asset-management [label="copied"]; + media-asset-management -> adactus-control [label="triggers"]; + adactus-control -> video.mp4, video.ogg [label="encodes"]; + video.mp4, video.ogg -> adactus-streaming [label="uploads"]; + adactus-control -> video.xml [label="exports"]; + adactus-control -> key-frames [label="creates"]; +} + + + + + + + + + + + MAM + + vmeo-controller + + vmeo-streaming + + video.raw + + video.mp4 + + video.ogg + + video.xml + + key-frames + + + + triggers + + + + encodes + + + + + + creates + + + + + + exports + + + + + + encodes + + + + copied + + + + uploads + + + + + + uploads + diff --git a/usr/share/doc/vizrt/vosa-handbook/graphics/webpagetest.png b/usr/share/doc/vizrt/vosa-handbook/graphics/webpagetest.png new file mode 100644 index 00000000..a601a153 Binary files /dev/null and b/usr/share/doc/vizrt/vosa-handbook/graphics/webpagetest.png differ diff --git a/usr/share/doc/vizrt/vosa-handbook/host-naming.org b/usr/share/doc/vizrt/vosa-handbook/host-naming.org new file mode 100644 index 00000000..266c1005 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/host-naming.org @@ -0,0 +1,83 @@ +* VOSA DNS naming conventions +This document assumes knowledge of the [[http://en.wikipedia.org/wiki/Domain_Name_System][global DNS system]]. + +Using hostnames to reach a service is preferred over using IP addresses directly because it gives us more routing flexibility. We can for instance change the IP address of a service without changing the client configuration. + +We have a standard for naming policy for machines and their roles. The following paragraphs describes it. + +** VOSA standard network naming +We use a standard naming scheme for all machines and virtual roles that we control. These names are built up out of the following name components: + +- :: Often named after a location or a datacenter provider. +- :: A name that reflects the capabilities of the machine. It might not perform all of those capabilities at all times. +- :: A name that reflects a specific role that is performed. One machine can have more than one virtual role. +- :: The lowercase name that represents an organizational unit often referred to as a customer or company. + +Applying the above to make full names: + +- Machine roles :: ...cust.vizrtsaas.com +- Global virtual roles :: ..cust.vizrtsaas.com +- Local virtual roles :: ...cust.vizrtsaas.com + +** <%= trail_customer_shortname %> Network naming + +| <%= trail_customer_shortname %> customer domain name | <%= trail_customer_acronym %>.cust.vizrtsaas.com | +| <%= trail_customer_shortname %> virtual role names | .<%= trail_customer_acronym %>.cust.vizrtsaas.com | +| <%= trail_customer_shortname %> remote datacentre assets | .<%= trail_customer_remote_datacentre %>.<%= trail_customer_acronym %>.cust.vizrtsaas.com | +| <%= trail_customer_shortname %> office network assets | .<%= trail_customer_office_network %>.<%= trail_customer_acronym %>.cust.vizrtsaas.com | + + +** Machine names + +The names of machine should reflect their capabilities to perform specific virtual roles. It is our goal to install machines in ways that allow to perform as many virtual roles as possible. However we are not quite there yet so many standard machine names resemble virtual role names. + +The follwing machine names are common in VOSA clusters: + +*** General +- Control :: (control) Dedicated machine to perform as an SSH entrypoint for operators. +- Monitoring :: (mon) Dedicated machine that harvest samples of performance metrics from all the machines and sending out events when performance or capacity limits are exceded. + +*** VME Online +All VME Online machines can be made redundant if there is a stable network connection between them. If a role is only represented once it does not get a number. if there are more, a number is added to the name. +/Controller:/ +(vmeo-controller*) All vmeo clusters need a VME controller. It might do everything or delegate some work to the other vmeo machines. + +The following machine roles are controlled by the controller: +- Transcoder :: (vmeo-transcoder*) transcodes any ingested video to any variant +- Encoder :: (vmeo-encoder*) can encode an incoming stream or SDI cable to an ipstream that is used for streaming. +- Delivery :: (vmeo-delivery*) Delivers video files over http for video on demand(VOD). It can also take streams from reporter or encoder and deliveres [[http://en.wikipedia.org/wiki/Flash_Video][Flash]] and [[http://en.wikipedia.org/wiki/Real_Time_Streaming_Protocol][RTSP]] +- Segmenter :: (vmeo-segmenter*) Takes streams from reporter of encoder and delviers [[http://en.wikipedia.org/wiki/HTTP_Live_Streaming][HLS]] +- Reporter :: (vmeo-reporter*) receives live streams form the viz reporter app on mobile devices. + +*** Escenic +- Escenic Presentation :: (pres*) Dedicated machine for generating webpages by merging content from a database with html from JSP files. +- Escenic Editorial :: (edit*) Dedicated machine for Content Studio webservice, Imports +- Escenic Analysis :: (analysis) Dedicated machine for counting pageimpressions for use by the webpages to present lists that depend on performance of specific content items. + +** Virtual roles and IP addresses +Virtual roles are often a single point of failure so we must distribute the capability to perform any virual role over multiple machines or privide very easy restore from a backup. Many machines should have the ability to perform a virtual role but at least one machine should actually play the virtual role at any time. If a virtual role is clustered, the virtual role hostname might resolve to several ip addresses or the virtual role hostnames might be numbered. + +When a machine assumes a virtual role it often claims the corresponding [[http://en.wikipedia.org/wiki/Virtual_IP_address][virtual IP address]] (VIP). A VIP can sometimes be claimed automatically if a [[http://en.wikipedia.org/wiki/Heartbeat_(program)][Linux-HA setup]] is installed. + +*** Virtual role names + +- Monitoring :: (mon..cust.vizrtsaas.com) The monitoring role +- Control :: (control..cust.vizrtsaas.com) The control role +- Proxy Gateway :: (proxy-..vizrtsaas.com) + +*** VME-online + +- Controller :: (vmeoc..cust.vizrtsaas.com) The virtual role of runing the database and the JBOSS server containing the Diactus Web Application. VMEO machines are all installed with all VMEO capabilities (Escenic migrating to the same policy but is not quite there yet) +- Transcoder :: (vmeot*..cust.vizrtsaas.com) Providing transcoding capacity to the controller. +- Delivery :: (vmeod*..cust.vizrtsaas.com) Endpoint for publishing video on demand and live video streaming. + +*** Escenic + +- Analysis :: (analysis..cust.vizrtsaas.com) the analysis webservices +- Editorial :: (edit..cust.vizrtsaas.com) Content Studio webservice and Indexer Webservice. Currently ther can only be one. +- Import jobs :: (import..cust.vizrtsaas.com) All periodic downloads of external content currently going on. +- NFS master :: (nfs..cust.vizrtsaas.com) The nfs server +- Presentation :: (pres*..cust.vizrtsaas.com) The presentation hosts currently configured on the loadbalancer. + + + diff --git a/usr/share/doc/vizrt/vosa-handbook/import-jobs.org b/usr/share/doc/vizrt/vosa-handbook/import-jobs.org new file mode 100644 index 00000000..1c4c0714 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/import-jobs.org @@ -0,0 +1,170 @@ +* Escenic Import Configuration + +An Escenic import configuration is a collection of +files packaged in an archive that can import data into the Escenic +Content Engine. It can be deployed to any machine that has Escenic +Content Engine running on it that was installed by =ece-install=. + +** Import configurations have the following features +- An execution plan :: You can include cron configuration to specify when and how often the import runs. +- A transformation plan :: You can specify the transformations that + will be applied to the incoming data before it is imported to the + Escenic Content Engine. +- Standard locations :: A standard set of file locations to read from and write to. +- Standard monitoring :: Standard monitoring is available to ensure that performance can be analysed and operators can be notified about failures. +An Import Configuration Archive can be deployed to an Escenic Content Engine machine by adhering to the standards discribed in the rest of this document. + +** Conventions +All file and directory names are to be lower cased with hyphens +between sub words, e.g. =my-import-configuration= and +=01-convert-from-atom-to-escenic-xml.xsl=. + +All import configurations delivered to the VOSA operators are =zip= +archives with a strict structure, as defined in [[Specifying an import configuration]] + +** Specifying an import configuration +When creating a new import configuration for your project, the following directory +structure is required: + +#+BEGIN_SRC text +/ +//transformers/-.xsl +//cron.hourly/ +//cron.every..minutes/ +#+END_SRC + +- publication name :: the name of the publication for which the import + job(s) are defined. You can have more than one publication in + each =zip= archive. +- import job name :: lowercase with hyphens between words (if more + than one) +- transformers :: directory with files prefixed with =-=, indicating + the order of transformation to apply to your import job. If + this is a =xsl= file, the escenic importer will run + =xsltproc= on the file, whereas =.sh= files will be + run in a =bash= wrapper. + + Each of the transformers will be called with one + argument, namely the input XML data. Each + transformer is responsible to write changes back to + the file. +- cron.hourly :: scripts to be run every our. These will be put in + =/etc/cron.hourly= on the import server. Be sure to + set the execute bit on the file and note that as with + all cron jobs, the file cannot have a file suffix. +- cron.every..minutes :: scripts to run every == minutes. + +We're calling the import configuration =moo= since we're +setting up an import feed from our imaginary content provider, "Moo +Cool Videos" and our publication is the ubiquitous =mypub=. + +#+BEGIN_SRC text +$ unzip -t my-great-import-archive.zip.zip +mypub/moo/transformers/01-fix-encoding.sh +mypub/moo/transformers/02-convert-all-cows-to-ducks.xsl +mypub/moo/transformers/02-convert-duck-to-escenic-xml.xsl +mypub/moo/cron.hourly/get-files-from-moo-ftp +mypub/moo/cron.every.10.minutes/ask-for-public-ip +otherpub/foo/transformers/01-convert-from-foo-to-escenic-xml.xsl +#+END_SRC + +As you can guess from the file names, the +=02-convert-all-cows-to-ducks.xsl= stylesheet will be first applied to +the incoming data (normally XML) and the +=02-convert-duck-to-escenic-xml.xsl= will be applied next before the +resulting Escenic XML will be imported into the Escenic Content +Engine. + +*** Pulling content from an FTP server +We have ready made BASH libraries to do this. You only need to put a +file in =mybub/moo/cron.hourly/get-files-from-moo-ftp= like: + +#+BEGIN_SRC text +#! /usr/bin/env bash +source /usr/share/escenic/engine/import/common-import-functions.sh + +ftp_user="user@server.com" +ftp_password="foobar" +ftp_url=ftp://ftp.server.com/myfeed/ +download_dir=/var/spool/escenic/import/mypub/moo/new +log=/var/log/escenic/cron.$(basename $0 .sh).log +ftp_download_history=/var/lib/escenic/ftp-history-cron.$(basename $0 .sh) +lock_file=/var/lock/$(basename $0 .sh).lock + +now=$(date +%s) +max_file_age_in_hours=2000 + +echo $0 "called @ $(date)" >> $log +download_latest_ftp_files +fix_ownership_of_download_files +echo $0 "finished @ $(date)" >> $log +#+END_SRC + +The only values you need to touch is are: +- =ftp_user= +- =ftp_password= +- =ftp_url= :: full URL to the directory on the FTP server. +- =download_dir= :: it's really just the =/mypub/moo= part of the + above example you'd have to change. + +The rest of the options should do just fine. If you really want to +tweak, you can of course decide how far back you want each run of the +import to look for files (it will only download each file once) by +setting =max_file_age_in_hours=. + +This will give you many features including: +- lock file support :: only one instance of your cron FTP script will + run at any given point in time. +- state :: only files that previously haven't been downloaded will be + downloaded with a new run of the cron job. +- log files :: logging of your cron scripts dedicated files + +** Import configuration deployed by VOSA + +*** Setting up a new import configuration +When an import configuration is received from a developer, the VOSA operator +will create the import configuration harness, infrastructure as with the +=ece-import= command as follows: +#+BEGIN_SRC text +$ ece-import \ + --publication-name mypub \ + --name video \ + --import-archive /tmp/my-great-import-archive.zip \ + create +#+END_SRC + +The =ece-import= script will take care of putting transformers in the +right place, create spool directories and create the Nursery +configuration needed for the import job. + +The =ece-import= will with the above command create an import job +with sensible defaults, the operator may override a few import +settings with these parameters: +- =--import-user= :: (the ECE user which will be the author of the + imported contents). Default is ==_admin +- =--import-section-name= :: the default, fallback section of imported + content. Default is =ece_incoming= + +*** Directories and Files +When an import job has been deployed by VOSA, it will use the following locations to read from and write to. + +|--------------------------------------------------------------------------------------------+-------------------------------------------| +| Path | Description | +|--------------------------------------------------------------------------------------------+-------------------------------------------| +| =/var/spool/escenic/import///new= | Input folder for the 3rd party (XML) data | +| =/usr/share/escenic/engine/import///transformers= | The transformers, such as XSLs | +| =/usr/share/escenic/engine/import/mypub/moo/transformers/02-convert-all-cows-to-ducks.xsl= | | +| =/var/spool/escenic/import/mypub/moo/new= | | +| =/var/spool/escenic/import///error= | Failed 3rd party XML files | +| =/var/spool/escenic/import/mypub/moo/error= | | +| =/var/spool/escenic/import///archive= | Successful imports of 3rd party files | +| =/var/spool/escenic/import/mypub/moo/archive= | | +| =/etc/cron.hourly/= | Cron script running every hour | +| =/etc/cron.hourly/get-files-from-moo-ftp= | | +| =/var/log/escenic/cron..log= | The log for your cron script | +| =/var/log/escenic/cron.get-files-from-moo-ftp.log= | | +|--------------------------------------------------------------------------------------------+-------------------------------------------| + + + + diff --git a/usr/share/doc/vizrt/vosa-handbook/jarfiles.org b/usr/share/doc/vizrt/vosa-handbook/jarfiles.org new file mode 100644 index 00000000..e9771c71 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/jarfiles.org @@ -0,0 +1,17 @@ +* How to add jar files to your project + +First of all there is a pre-requisite for all types of jar files you want to add to a publication .war file or to the escenic/lib/ folder which requires the jar file to be part of the overall project build. For jar built within your project this means they need to have a module under i.e. libraries, which is the setup used most frequently, and for third party jars they have to be listed as a dependency. + +** Case I - how to add a jar file to a publication .war + +- if you add a jar dependency, with scope compile or no scope at all, to the pom.xml of the publication x module the jar file will go into x.war + +- if you add a jar dependency, with scope compile or no scope at all, to the pom.xml of the shared-war module the jar file will go into all .war files depending on shared-war + +** Case II - how to add a jar built in the project to escenic/lib/ + +- open project-assembly/src/main/assembly/assembly.xml and locate the include statements for jar files, i.e *:some-module-name:jar. If you add another line with the name of your jar, i.e. *:my-new-plugin:jar, in both places, then the next build on the SaaS builder will include your jar in the .ear file and it will be deployed to escenic/lib when you ece-deploy the .ear file. + +** Case III - how to add third-party jars to escenic/lib/ + +- make sure the third-party jar you want to add is present in your project as a dependency with scope provided!!! Open project-assembly/src/main/assembly/assembly.xml and locate the include statements for jar files, i.e *:some-module-name:jar. If you add another line with the name of your third-party jar, i.e. *:xmlrpc-server:jar, in both places, then the next build on the SaaS builder will include your jar in the .ear file and it will be deployed to escenic/lib when you ece-deploy the .ear file. diff --git a/usr/share/doc/vizrt/vosa-handbook/monitoring.org b/usr/share/doc/vizrt/vosa-handbook/monitoring.org new file mode 100644 index 00000000..9929bdfa --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/monitoring.org @@ -0,0 +1,66 @@ +* Monitoring + +On a VOSA system you have several tools to aid you in getting an +overview of how your system is doing, how it is configured and how it +has been behaving from its inception till the present. + +** system-info + +*** On the Command Line +On any VOSA host, you can execute the command system-info. If you +execute this command as root, you better pass a parameter to tell it +the user under which ECE, EAE and the Search instance are running as +(typically "escenic"): + +#+BEGIN_SRC text +# system-info -u escenic +#+END_SRC + +*** In the Web Browser +All VOSA hosts also has a (maximum) one minute old HTML report from +system-info running on port ~5678~. + +** Hugin +Hugin is an automatic, frequent aggregation of the system-info data of +all the hosts in your habitat. It does this by asking the hossts to +produce a system-info report in YAML and check the files into a +version repository. + +When browsing the Hugin web interface at +http://<%= trail_monitoring_host %>.<%= trail_network_name %>:5679/ you can quickly +see all changes on the various servers. A change could be a new +service which is all of a sudden listening on a new port or that the +version of a installed package has changed. Nothing escapes Hugin's +watchful eyes! + +** Munin +Each VOSA habitat has its own Munin installation where you can see the +trending of your site. I.e. how it has been behaving the last days, +weeks and months. + +The address to your Munin instance is is: + http://<%= trail_monitoring_host %>.<%= trail_network_name %>/munin/ + +You can add/edit server nodes to monitor here in the monitoring server - +/etc/munin/munin-conf.d/escenic.conf + +** Icinga (enhanced Nagios) +Icinga gives a good overview of the current health of the +habitat. Head over to + http://<%= trail_monitoring_host %>.<%= trail_network_name %>/icinga/ +to get updates on host & service uptimes and disk fill rates. This is +also the place were you'll schedule any maintenance work you do to +prevent unnecessary alerts filling up your mobile phone and email +inboxes. + +check\_mk plugin for nagios does the automated monitoring configuration, which saves a lot of manual configuration effort. If you need to add/edit new nodes to monitor then- +- Add/edit node entry in /etc/check\_mk/main.mk file +- Create Inventory of the new checks - +#+BEGIN_SRC text +# check_mk -I +#+END_SRC +- Update the checks for check\_mk - +#+BEGIN_SRC text +# check_mk -O +#+END_SRC + diff --git a/usr/share/doc/vizrt/vosa-handbook/network-file-system-sync.org b/usr/share/doc/vizrt/vosa-handbook/network-file-system-sync.org new file mode 100644 index 00000000..3186fa1b --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/network-file-system-sync.org @@ -0,0 +1,9 @@ +** Syncing the Master & Slave NFS + +The NFS slave on <%= trail_nfs_slave_host %> replicates the master NFS +on <%= trail_nfs_master_host %> using an hourly cron job. + +This is initiated by a file on <%= trail_nfs_slave_host %>'s +=/etc/cron.hourly= directory which calls the +=/usr/bin/sync-network-drive= with the appropriate parameters to sync +the <%= trail_nfs_export_list %>. diff --git a/usr/share/doc/vizrt/vosa-handbook/network-file-system.org b/usr/share/doc/vizrt/vosa-handbook/network-file-system.org new file mode 100644 index 00000000..c781981a --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/network-file-system.org @@ -0,0 +1,15 @@ +* Network File System + +** NFS server +<%= trail_nfs_master_host %> has these network file shares: +~<%= trail_nfs_export_list %>~ These file share(s) are made available using +NFS. + +** NFS clients +All of the hosts running an presentation or publication server will +have these shares mounted. The most important one is the one holding +the multimedia archive of Escenic Content Engine. Typically, this +share is called ~multimedia~. + +In your habitat, all the shares from ~<%= trail_nfs_export_list %>~ are +mounted on ~<%= trail_nfs_client_mount_point_parent %>~. diff --git a/usr/share/doc/vizrt/vosa-handbook/operating-system-maintenance.org b/usr/share/doc/vizrt/vosa-handbook/operating-system-maintenance.org new file mode 100644 index 00000000..267c0107 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/operating-system-maintenance.org @@ -0,0 +1,56 @@ +* Operating System Maintenance + +Before proceeding, please note that this is serious stuff. To update +the OS packages, you should feel comfortable resolving typical RPM or +DEB conflicts and set aside enough time to deal with any package +breakage which might occur. + +All servers, both the Ubuntu and RedHat ones, are running stable +branches of their OS, so normally, there will be no breakage, and +upgrades should be safe to be carried out on a regular (bi-weekly for +instance) basis. However, you should always be fully alert when making +these updates and be sure to have enough time on your hands to resolve +any issues before going ahead with the upgrades below. + +** Debian based operating systems +This applies to OSs like Debian, Ubuntu and Mint. + +In the example below, we update the Ubuntu servers called +<%= trail_presentation_host%> <%= trail_editor_host %> <%= trail_import_host %> +<%= trail_monitoring_host %> <%= trail_analysis_host %> from the comfort of +<%= trail_control_host %>. + +#+BEGIN_SRC sh +$ for el in \ + <%= trail_presentation_host %> \ + <%= trail_editor_host %> \ + <%= trail_import_host %> \ + <%= trail_monitoring_host %> \ + <%= trail_analysis_host %> \ + ; do \ + echo "Now upgrading $el"; \ + sudo ssh root@$el 'apt-get -qq update && apt-get -q upgrade' ; \ + done +#+END_SRC + + +** RedHat based operating systems +This applies to OSs like CentOS, RedHat and Fedora: + +In the example below, we update the RedHat servers called +<%= trail_presentation_host%> <%= trail_editor_host %> <%= trail_import_host %> +<%= trail_monitoring_host %> <%= trail_analysis_host %> from the comfort of +<%= trail_control_host %>: + +#+BEGIN_SRC sh +$ for el in \ + <%= trail_presentation_host %> \ + <%= trail_editor_host %> \ + <%= trail_import_host %> \ + <%= trail_monitoring_host %> \ + <%= trail_analysis_host %> \ + ; do \ + echo "Now upgrading $el"; + ssh $el yum update; + done +#+END_SRC diff --git a/usr/share/doc/vizrt/vosa-handbook/overview.org b/usr/share/doc/vizrt/vosa-handbook/overview.org new file mode 100644 index 00000000..b8a56b9b --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/overview.org @@ -0,0 +1,6 @@ +* Getting the Overview + +#+ATTR_HTML: alt="ECE architecture" +[[file:./graphics/architecture.svg][./graphics/architecture.svg]] + +#+INCLUDE: "overview-generated.org" diff --git a/usr/share/doc/vizrt/vosa-handbook/privatekeys.org b/usr/share/doc/vizrt/vosa-handbook/privatekeys.org new file mode 100644 index 00000000..5ef84130 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/privatekeys.org @@ -0,0 +1,27 @@ +* How to manage your private keys +Vizrt needs to make it very easy to allow an operator to make changes to the configuration of a system that we maintain for our customers. At the same time we must protect systems against unauthorized access. +We have chosen to use SSH keys to solve this problem. This means that ssh keys are at the center of our security policy. This section explains how we need people to work with them. + +Read the following rules as if it is about your personal credit card: +- Never duplicate a private key. Just make a new one if you loose it. +- Always put a pass phrase on a private key. +- A pass phrase contains at least 3 words. The longer the better. +- Keys are personal. Do not share keys amongst people. +- Use agent forwarding when ssh hopping. (ssh -A thv@somehost.com) +- Notify the SaaS team if you need to block your key immediately. + +** Why is it so important? +Each private key file represents a security threat to our customer or Vizrt. Imagine that we would somehow use physical keys. How secure is a building that can be opened by a key for which we have many different copies? +It is as simple as that. We have many doors. When we enable your key, we adjust all those doors to open with your key. If an unauthorized person can use your key, our customers and Vizrt are in big, big trouble. + +** Personal responsibility +Vizrt and customers are ensured for security incidents. This insurance does not protect us at all if: +1. The harm is done to Vizrt by a Vizrt employee or by the customer to the customer. +2. The harm is caused by a Security Breach that could have been avoided by the design of the software. +In other words: +#+BEGIN_QUOTE +Keep your keys safe and we will be okay even if we make bad mistakes. +#+END_QUOTE + +** Help others be safe and be safe yourself. +We use SSH keys to protect our dearest secrets. You are the last line of defense. This is why we have to behave in a way that resembles your behavior around physical safety. What do you do when you boss is about to fall of a ledge? You grab him and drag him to safety. Your boss will be startled at first and thank you afterward. In fact he might accuse you of neglecting to tell you of the danger if you didn't act. That is how you should act when you see others act UN responsible with keys. Even your boss. diff --git a/usr/share/doc/vizrt/vosa-handbook/search-server.org b/usr/share/doc/vizrt/vosa-handbook/search-server.org new file mode 100644 index 00000000..058d83ff --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/search-server.org @@ -0,0 +1,43 @@ +* Search Server +A search server is an application server only running the ~solr~ and +~indexer-webapp~ web applications. It's highly recommendable to run +these web applications on a dedicated JVM as their behaviour with +regards to object creation, connections and garbage collection is +highly different from that of a publication or presentation server. + +** Changing the search server used by Content Studio. + +When updating the search server, you might want to reconfigure the +ECE instance(s) that use the search server you're updating to allow Content Studio instances to search. To do +this, you can use the Component Browser in the escenic-admin +interface. + +For instance, let's say you're upgrading the Content Studio search server for +<%= trail_editor_host %> and want it to temporarily use +<%= trail_import_host %>'s search instance. + +First, go to: +http://<%= trail_editor_host %>.<%= trail_network_name %>:<%= trail_editor_port %>/escenic-admin/browser/Global/com/escenic/webservice/search/DelegatingSearchEngine?property=solrURI + +Then, change the value from: +#+BEGIN_SRC conf +http://localhost:8081/solr/select +#+END_SRC + +to + +#+BEGIN_SRC conf +http://<%= trail_import_host %>:8081/solr/select +#+END_SRC + +** Changing the search server used by the website. +The following configuration points influence the search behavior in the website: +- Site Search (Widget Framework): com.escenic.framework.search.solr.SolrHttpClient +- Site Search (Widget Framework): com.escenic.framework.search.solr.SolrSearchEngine +- Site Search (DEPRECATED): com.escenic.lucy.LucySearchEngine + +** Changing the search server used for web studio person search +For some strange reason the web studio person search has a unique configuration for the search server: +- escenic/solr-base-uri environment variable in context.xml (also used by indexer webapp) + + diff --git a/usr/share/doc/vizrt/vosa-handbook/ssh-access.org b/usr/share/doc/vizrt/vosa-handbook/ssh-access.org new file mode 100644 index 00000000..23a3769a --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/ssh-access.org @@ -0,0 +1,127 @@ +* SSH Access + +This chapter describes how you can access the <%= trail_network_name %> +network using SSH and assumes that you have your [[http://en.wikipedia.org/wiki/Secure_Shell#Key_management][SSH key]] on the +<%= trail_gk_host %> and all over the <%= trail_network_name %>. + +It also assumes that the gateway IP of your office/network has been +added to the access control list of hosts/networks which are allowed +to access the gatekeeper. + +** Generate an SSH key on Linux or Mac OS X +*** Check to see if you already have a key you can use +You only need to do this if you don't already have an SSH key. The +keys are normally stored in ~$HOME/.ssh~ so have a look here to see if +you have any: +#+BEGIN_SRC text +$ ls ~/.ssh +authorized_keys id_rsa id_rsa.pub known_hosts +#+END_SRC +If you see the ~id_rsa~ and ~id_dsa.pub~, you already have one. They +can also be called something with ~dsa~ or indeed anything (the +~id_rsa[.pub]~ are just the default). + +*** Generate a new SSH key +To generate a new key, you can do: +#+BEGIN_SRC text +$ ssh-keygen -t rsa +#+END_SRC + +Just hit enter when you're prompted for the name of the file. When +asked for a key password, be sure to set one. This is important, all +your SSH keys should have a password set. If not, anyone can copy your +keys and log on to all the servers where you've previously been +granted access. + +If you wish to set the password on an old key, or set a new password, +you can do this by +#+BEGIN_SRC text +$ ssh-keygen -f id_rsa -p +#+END_SRC + +The ~id_rsa~ above corresponds to the ~$HOME/.ssh/id_rsa~ file. + +*** Add your SSH key to your key agent +On your local machine, load your SSH key into your key agent. On Linux +and Mac OS X, this should only be a matter of: +#+BEGIN_SRC text +$ ssh-add +#+END_SRC +Asking the SSH key agent for all keys should now list your key. Note +that if you have multiple keys, or have called your one key something +different that what's default, you must pass the key name to the +~ssh-add~ command above. + +Once you're done adding your key(s), confirm their presence in the key +ring with: +#+BEGIN_SRC text +$ ssh-add -l +2048 33:49:02:c5:30:8d:2e:9g:28:82:3a:89:5c:b3:5d:a3 /home/myuser/.ssh/id_rsa (RSA) +#+END_SRC + + +** Generate an SSH key on Windows + +*** Using Putty + +/Generating your key:/ +- Install the full Putty package including puttygen by using the Windows Installaller that is downloadable [[http://putty][here]] +- Run puttygen +- Generate your personal key (It wants you to move your mouse around a bit) +[[./graphics/puttygen1.png]] +- Give your key a name and enter a passphrase. We use keys for better security. Choosing a weak password will not help. It is called a [[http://en.wikipedia.org/wiki/Passphrase][passphrase]] because your are expected to enter a sentence and not just a word. +[[./graphics/puttygen2.png]] + +/Storing your key:/ + +Make a directory like C:\Users\thv.VIZRTINT\.ssh. Do not spread your key around further. If you loose your key because your laptop crashes, you can allways create a new one. Save your key as .pkk and as .pem +- Save your key in and Privacy-enhanced Electronic Mail format(.pem) +[[./graphics/puttygen3.png]] + +- Save your key in a Putty Private Key File(.pkk) +[[./graphics/puttygen4.png]] + +**** Using Pagent +Pagent is Putties [[http://en.wikipedia.org/wiki/Ssh-agent][SSH agent]]. Using am SSH agent makes the use of keys a lot easier. An SSH agent runs in the background to supply a key to any putty-ssh login you might perform during the day without retyping your passphrase. + +Pagent lets you load keys interactively or on the command line. +[[./graphics/pagent.png]] + +*** Using Cygwin + +Install [[http://cygwin.com][Cygwin]] and be sure to check for the =openssh-client= package +when running =setup.exe=: + +[[./graphics/cygwin-openssh.png]] + +Once the installation is done, you should have a =cygwin= icon to +click on: +[[http://x.cygwin.com/cygwin-icon.gif]] This will open a Unix like shell +on your Windows computer, in which you can use many of the standard +Unix commands, such as =ls=, =cd=, =cp=, =rsync= & =grep=. + +In the Cygwin shell, you now run the same =ssh= command that you'd +have on Linux. This command is what you can see being used and +you can use the instructions above in the [[Generate an SSH key on Linux +or Mac OS X]] section to generate and use your SSH key. + + +** Log on to the gatekeeper +On your local machine, do: +#+BEGIN_SRC text +$ ssh +#+END_SRC + +** Log on to the control server +Once you're on <%= trail_gk_host %> should be able to log on to the +control server: +#+BEGIN_SRC text +$ ssh <%= trail_control_host %> +#+END_SRC + +From the control server, you can access any of the machines in the +<%= trail_network_name %>, e.g.: +#+BEGIN_SRC text +$ ssh <%= trail_presentation_host %> +#+END_SRC + diff --git a/usr/share/doc/vizrt/vosa-handbook/upgrade-minor.org b/usr/share/doc/vizrt/vosa-handbook/upgrade-minor.org new file mode 100644 index 00000000..4744c26e --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/upgrade-minor.org @@ -0,0 +1,20 @@ +* Template plan for upgrading from Escenic *.x.* to *.y.* (minor version upgrade) +When an Escenic cluster needs to be upgraded we perform the following steps. +1. TODO (operator) Make a branch in subversion of the escenic customer project. +2. TODO (operator) Write a script to Upgrade the dev image db and mm (make available as deb package). +3. TODO (operator) Change the root pom in the customer project to reflect the target version. +4. TODO (operator) Make an ear and test it in the dev image. +5. TODO (operator) Provide the tested dev image to the customer and point out the standard template upgrade issues. +6. TODO (template dev) Wait for template development to upgrade the templates. +7. TODO (operator) Create a test server so the new stuff can be tried by editorial when the templates are ready. +8. TODO (operator) Create an automatic daily copied and upgraded new db and mm. +9. TODO (operator) Create a production platform with full redundancy but small scale that points to it. +10. TODO (users) Do user acceptence testing on the new production platform. +11. TODO (operator, users) Agree with customers when to switch to the new system. +12. TODO (operator) stop the automatic daily copy and upgrade script. +13. TODO (operator) make the old system unavailable +14. TODO (users) Start using the new system +15. TODO (users) Report problems as they arise +16. TODO (operator) rollback: make the new system unavailable and the old system available. +17. TODO (operator) commit: destroy the old system + diff --git a/usr/share/doc/vizrt/vosa-handbook/vizrt-branding-org-mode.el b/usr/share/doc/vizrt/vosa-handbook/vizrt-branding-org-mode.el new file mode 100644 index 00000000..a8b5b669 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/vizrt-branding-org-mode.el @@ -0,0 +1,257 @@ +;; load all locally installed packages +(let ((default-directory "/usr/local/src/emacs")) + (normal-top-level-add-subdirs-to-load-path)) + +(require 'org-export) + +;; don't create a footer +(setq org-export-html-postamble nil) + +;; These two settings make exported HTML look like our Vizrt branded +; release notes. +(setq org-export-html-preamble " + + + + + + + + + + + + +
    +" + + org-export-html-postamble t + org-export-html-postamble-format (quote (("en" "
    +

    + Author: %a (%e) + Date: %d +

    +
    + +"))) + org-export-html-style " +" + ) diff --git a/usr/share/doc/vizrt/vosa-handbook/vizrt-services.org b/usr/share/doc/vizrt/vosa-handbook/vizrt-services.org new file mode 100644 index 00000000..d00ca307 --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/vizrt-services.org @@ -0,0 +1,21 @@ +* Vizrt SaaS Services + +|----------------+---------------------------------+-----------------------------+--------------------------| +| Service | Production | Staging | Test | +|----------------+---------------------------------+-----------------------------+--------------------------| +| Website | <%=trail_production_websites %> | <%=trail_staging_websites%> | <%=trail_test_websites%> | +| Content Studio | <%=trail_production_studio%> | <%=trail_staging_studio%> | <%=trail_test_studio%> | + +| Jira | https://vizrtcustomers.jira.com/browse/<%=trail_vizrtcustomer_jira_name%> | +| SVN | https://vizrtcustomers.jira.com/svn/<%=trail_vizrtcustomer_jira_name%> | +| Hipchat | https://vizrtcustomers.hipchat.com/chat | +| Dev Image | <%=trail_dev_image_location%> | +|----------------+------------------------------------------------------------------------------------------| + +Content Studio service is special because - +1. It usually requires to run on port 80 or 443 +2. It usually requires to have a nice hostname aliasing +3. It does not run over a proxy server + +We strongly recommend to setup SSL with authorized certificate for this service. + diff --git a/usr/share/doc/vizrt/vosa-handbook/vmeo.org b/usr/share/doc/vizrt/vosa-handbook/vmeo.org new file mode 100644 index 00000000..b1e0c45b --- /dev/null +++ b/usr/share/doc/vizrt/vosa-handbook/vmeo.org @@ -0,0 +1,132 @@ +* Viz Media Engine - Online (VMEO) +VME-Online is a content mangement system and online publishing system for Video. + +** VMEO & ECE Integration + +*** Getting video into VMEO +[[./graphics/vmeo-mam-to-vmeo.svg]] + +Video is normally stored on a MAM, a huge file server onto which the +users copy their videos. + +VMEO controller watches a folder on the MAM for updates and imports +new raw video files as soon as they occur. The new video file is then +transcoded into a number of pre-set formats. Once all the video +formats and variants are created, the files are uploaded to the +=vmeo-streamer=. + +The VMEO controller also generates a number of pictures of the video +file at several places in the clip. These snapshots are called key +frames. + +The last thing VMEO does, is to export an XML file describing the new +video file available in the VMEO controller. + +*** Getting the video (meta data) into ECE +[[./graphics/vmeo-import-to-ece.svg]] + +The Escenic Content Engine downloads the exported video XML, applies a +number of transformers on it (these can be XSL, BASH, Python or Perl +files) into ECE Syndication XML and downloads the key frames from the +VMEO controller. + +The ECE syndication XML files is then imported into ECE and the users +can see the video content item in the Escenic Content Studio with the +key frame pictures available as related content items on the video +article. + +*** Displaying video in the web browser +[[./graphics/vmeo-browser-playing-video.svg]] + +When one of these video content items is desked on the website, ECE +will render the video widget. In addition to the markup, the video +widget will render some JavaScript which will ask ECE to proxy a +request to =vmeo-controller= for the URL the video stream or file. The +URL the browser calls via JavaScript looks something like this: +=/escenic-adactus-proxy/videoId=123= + +The response from the =vmeo-controller= is cached in the caching +server and returned to the JavaScript running in the browser which +will inject the URL in a player. Normally this is a Flash player like +Flowplayer or an HTML =