Skip to content

ckanhosted.com

Jump to bottom Edit New page
johnglover edited this page Dec 10, 2012 · 6 revisions

Also see staging.ckanhosted.com

  • Demo sites are installed on s082.okserver.org and s083.okserver.org

  • Each server has multiple sites running off one CKAN installation and one set of CKAN extensions

  • CKAN and dependencies are all installed in a Python virtualenv at /usr/lib/ckan

  • This means that all sites on a given server must be upgraded at the same time (see How To Upgrade CKAN below)

  • Demo sites' CKAN configuration files are in /etc/ckan/{SITE}/production.ini,
    and their Apache config files are in the usual place: /etc/apache2/sites-available/{SITE}

TODO: Should CKAN and Apache config files be kept in private github repos that are checked out on the server?

How to Setup the Frontend and Database Servers

There is some initial setup of the website/frontend server and the solr and database server that only needs to be done once (doesn't need to be repeated each time you boot a new site on the server). I'm not entirely clear on the details of this so maybe someone else can fill this section out a little better. We used ckanbuild to do the initial setup. I think the main steps are:

  • Run ckanbuild's dbserver.sh script on the database server to setup postgres, solr, a firewall, etc.
  • Run ckanbuild's webserver.sh script on the frontend server to setup apache, nginx, a firewall, etc.
  • Use ckanbuild to build a Debian package for a CKAN website.
  • Install the Debian package on the frontend server. Note that we don't actually want to use the CKAN site contained in this package, we'll follow a different process (below) to setup each new site on the server, but we're installing the package because it does some more webserver setup stuff that we need.
  • Use ckanbuild's dbserver.sh addweb command on the database server to add the internal IP address of the frontend server to the firewall, so that the frontend server can access the database server over their internal network.

The SOLR and PostgreSQL Server

s085.okserver.org is currently the SOLR and PostgreSQL server for all CKAN sites on s082 and s083 (but if in doubt check the solr and postgres settings in a site's production.ini file).

Each server has two active network devices: one with an internal IP address, and one with an external IP address. Communication between the servers is always done over the internal network.

SOLR

Each instance has its own solr index, named after the instance name.

PostgreSQL

  • There are two databases for each site, one for the site's CKAN metadata with the same name as the site e.g. 'ghana', and one for the site's datastore called e.g. 'ghana-datastore'.
  • Each site has its own database user with the same name as the site e.g. ghana etc. Each site's database user has readwrite access to the site's database and the site's datastore database.
  • There's a single database user called 'datastore' that has readonly access to all of the sites' datastore databases

Celery and ckanext-datastorer

TODO: Currently datastorer is not working on s082.

ckanext-datastorer is installed in CKAN's virtualenv.
TODO: Currently it's the new_datastore branch of ckanext-datastorer, this should be changed to master since it has been merged in https://github.com/okfn/ckanext-datastorer/commit/c385dd0a2c4f6f3e32cff8979328c48b5d3e8563.

Each site should have datastorer in the plugins list in its production.ini file.

Celery daemon runs, one process per site. The paster celeryd command can be used to controll the celery daemon. TODO: How do we run celeryd? e.g. with supervisord?

ckanext-hosted

Some of the sites use a ckanext-hosted extension from a private git repo: https://github.com/okfn/ckanext-hosted. Look in /etc/ckan/{SITE}/production.ini to see whether a site is using this extension.

Other CKAN Extensions

Some of the sites have their own extensions, e.g. ckanext-colombia, ckanext-ghana, etc. This extensions are kept in private repos in the okfn organization on github. Look in /etc/ckan/{SITE}/production.ini to see whether a site is using an extension.

How to Run Paster Commands

There's a ckan command at /usr/bin/ckan (which is on $PATH) that lets you run paster commands without having to give all the options manually, e.g. you can just do ckan -i ghana db upgrade.
seanh: But this command seems to run off a copy of ckanbuild at /home/okfn/build/ckanbuild that uses its own copy of the ckan source code that's different from the code the sites are running off, so it might be best to avoid using it?

To run a paster command on a site manually you have to:

  1. Use the paster executable from the ckan virtualenv
  2. Pass the --plugin=ckan option to paster
  3. Pass the -c /path/to/ini_file option to the paster subcommand

For example:

/usr/lib/ckan/bin/paster --plugin=ckan sysadmin -c /etc/ckan/ghana/production.ini add seanh

Log Files

When something goes wrong (e.g. a site is giving 500 Server Errors) the log files can be found at:

/var/log/apache2/{SITE}.error.log
/var/log/apache2/{SITE}.custom.log

How to Set Up a New Site

Since all demo sites are pretty much the same, it's best to use an existing demo site as a template for the new one. FIXME: This is no good as an existing site might have customisations e.g. logo whereas a new site might want the CKAN default. Need a template dir or git repo to start from?

Create Config Files

Create a new CKAN config directory:

cd /etc/ckan
cp -r ./greebo ./${demo-name}

Edit /etc/ckan/{demo-name}/production.ini:

  • Remove references to the old site
  • Edit the site description

Copy the Apache configuration:

cd /etc/apache2/sites-available
cp ./aerc ./${demo-name}

and edit it appropriately.

Create Database

Create the new site's CKAN metadata database and database user. Login to the database server (see the new site's production.ini file to see what database server it's using):

cd ~/ckanbuild/bin
./dbserver createdb ${demo-name}

This will generate a password for you, which you'll need to copy into the new production.ini file on the frontend server.

Create Datastore Database

Create the new site's datastore database and set the permissions for the database users. There's a script on s085.okserver.org for doing this, at /home/okfn/scratch/create-datastore-database/create-datastore-database.sh.
seanh: There is a paster command for this, but it isn't working yet so we're using this script.

Initialise Databases

Back on the frontend server:

/usr/lib/ckan/bin/paster --plugin=ckan db -c /etc/ckan/{SITE}/production.ini init

Create FileStore Directory

Create the directory where CKAN's filestore will store uploaded files, and give www-data rwx permissions:

sudo mkdir /var/lib/ckan/SITE
sudo chown www-data /var/lib/ckan/SITE

(The foolproof way to see where the filestore directory should be created is to look at the ofs.storage_dir setting in the site's production.ini file, but assuming you started from the template production.ini file it should be /var/lib/ckan/SITE.)

Enable Site

Enable the new site. Back on the front-end server:

sudo a2ensite ${demo-name}
sudo service apache2 reload

Create Admin User

Create a CKAN sysadmin user (called admin) and password for the new site:

/usr/lib/ckan/bin/paster --plugin=ckan sysadmin -c /etc/ckan/{SITE}/production.ini add admin

Email the username and password to the client, and ask them to login and change it.

How to Set a Custom Site Logo

There are various ways you could change the site logo in CKAN, by overriding the template file using an extension or extra_template_paths, overriding the logo image file using extra_public_paths, using the /ckan-admin/config web page.

It's best if we always do this the same way for ckanhosted sites, so I suggest:

  1. Make an extra_public_path and upload the logo file to it, e.g.:

     /etc/ckan/SITE/extra_public_path/SITE_logo.png

  2. Set the unix permissions for the new dir and file, extra_public_path should be rwxr-xr-x and SITE_logo.png should be rw-r--r--.

  3. Find the ckan.site_logo and extra_public_paths in production.ini and change them:

     ckan.site_logo = /SITE_logo.png
 extra_public_paths = %(here)s/extra_public_path

  4. Restart Apache:

     sudo service apache2 restart

  5. You'll often have to tweak the CSS to get the new logo to look right, e.g. changing the height or background colour of the masthead, see the next section.

How to Set Custom CSS

There are various ways you can change the CSS in CKAN: by overriding the template files using an extension or the extra_template_paths ini file setting, by overriding CSS files using extra_public_paths, by providing additional CSS files using template_head_end ini file setting, or by using the /ckan-admin/config web page.

It's best if we always do this the same way for ckanhosted sites, so I suggest:

  1. Login to the site with an admin account.

  2. To go /ckan-admin/config, and enter custom CSS into the text box.

    A common example is to change the height of the masthead to be at least as high as the site's custom logo:

    .masthead {
  min-height: 120px;
}

This way means that designers or clients can tweak the CSS themselves without needing ssh access to the server or having to restart Apache.

But note that some sites may have custom CSS in files in their extra_public_paths dirs.

How to Upgrade CKAN

Since all the sites on a given server run off a shared copy of the CKAN source code, upgrading CKAN on the server means upgrading all of the sites at once.

Backup the Site's Databases

TODO: Backup the datastore db for each site as well. How do we upgrade the datastore DB's if the schema has changed? How do we restore a site's datastore db from backup? (pg_dump, pg_load, but can we make paster commands for this?)

Backup each site's database before upgrading. There is a script that does this:

/home/okfn/backup_ckan_databases.py

It looks in /etc/ckan for any top-level subdirectories containing a production.ini file. Taking the directory name to be the site name, it runs this command for each site:

/usr/lib/ckan/bin/paster --plugin=ckan db dump --config=/etc/ckan/{SITE}/production.ini /etc/ckan/pg_dumps/{SITE}_{DATESTR}.pg_dump

Upgrade the CKAN Source Code

Now upgrade the CKAN source code:

cd /usr/lib/ckan/src/ckan
git pull

If the git pull command fails because there are untracked changes to *.min.js and/or *.min.css files in the working tree (created by the minify command below) then delete these changes from the working copy and run git pull again. Don't commit the min files to git!

TODO: In the future we'll be checking out a release branch instead of just pulling the latest commit from master.

Run python setup.py develop to register any new paster command etc. that have been added by the new upgrade. Be sure to run the copy of python in the virtualenv so it runs with the virtualenv active:

/usr/lib/ckan/bin/python setup.py develop

Updated the Minified Files

Since we are running off CKAN's master branch, you need to run paster minify to update the minified javascript and CSS files to the latest development versions. Otherwise the site will run with the latest Python code and templates but older versions of the javascript and CSS, and things will look and function wrong. You have to pass a production.ini file as an argument to the minify command or it won't run, because it tries to rebuild the SOLR index. But I don't think it matters which site's ini file you pass as I don't think the SOLR index actually needs to be rebuilt at this point:

/usr/lib/ckan/bin/paster minify -c /etc/ckan/ausgoal/production.ini /usr/lib/ckan/src/ckan/ckan/public/base

This means that we're actually modifying the contents of the CKAN git repo on the server which is really bad. If you cd to /usr/lib/ckan/src/ckan and run git status you'll see many *.min.js and *.min.css files have been modified or newly created. Do not commit these changes to the git repo. At some point the git pull command above will fail because of these changes and we'll have to clean out these changes from the working tree in order to run git pull.

Upgrade any CKAN Extensions

TODO

Upgrade each Site's Database

Now we have to upgrade each site's database to the new database schema (if the CKAN upgrade we just did has changed the schema). For each site on the server run this command:

/usr/lib/ckan/bin/paster --plugin=ckan db upgrade --config=/etc/ckan/{SITE}/production.ini

replacing {SITE} with the name of the site.

Restart Apache

Finally, restart Apache:

sudo service apache2 restart

Now load each site in your browser and check that they're all working.

How to Disable a Site

sudo a2dissite {SITE}
sudo service apache2 restart

TODO: Should we do anything more here to disable a site?

How to Add Demo Data to a Site

We have a script for loading demo data (gold prices, Malawi aid projects, etc.). This script is checked out at /home/okfn/load_demo_data/src/load_demo_data. See the scripts README for usage.

Add a custom footer
Add a custom sidebar
Clone this wiki locally