Skip to content

Setup guide

Jump to bottom
Zmago Devetak edited this page Jan 24, 2018 · 84 revisions

Build & run

sciNote is developed in Ruby on Rails. It also makes use of Docker technology, so the easiest way to run it is inside Docker containers.

Prerequisites

Quick start

The following are minimal steps needed to start sciNote in development environment:

  1. Clone this Git repository onto your development machine.
  2. Create a file config/application.yml. Populate it with mandatory environmental variables (see environmental variables).
  3. In sciNote folder, run the following command: make docker. This can take a while, since Docker must first pull an image from the Internet, and then also install all neccesary Gems required by sciNote.
  4. Once the Docker image is created, run make cli command. Once inside the running Docker container, run the following commands: rake db:create - rake db:migrate - rake db:seed. This should initialize the database and fill it with (very minimal) seed data.
  5. Exit the Docker container by typing exit.
  6. To start the server, run command make run. Wait until the server starts listening on port 3000.
  7. Inside another CLI window (also from sciNote folder), run command make worker. This command should start the background worker process which is used by sciNote for a lot of demanding tasks.
  8. Open your favourite browser and navigate to http://localhost:3000. Use the seeded administrator account from seeds.rb to login, or sign up for a new account.

If you need to run server in production please check this section

OS-specific Install Instructions

Debian

  1. Install Docker and add user 1000 to the docker group as described here.
  2. Install Docker Compose as described here.
  3. Follow Quick Start Guide above as user 1000.

Mac OS X

  1. Install command line developer tools (there are many resources online, like this).
  2. Install Docker Toolbox as described here.
  3. Inside CLI, run git clone https://github.com/biosistemika/scinote-web.git.
  4. Run Docker Quickstart Terminal (also described here).
  5. Inside this terminal, navigate to cloned Git folder.
  6. Follow the Quick Start Guide above.
  7. When opening sciNote in browser, instead of navigating to localhost:3000, navigate to <docker-machine-ip>:3000 (you can get the docker machine IP by running command docker-machine ip default).

Windows XP, Vista, 7, 8

There is currently no good way to deploy sciNote onto older Windows operating systems. You can run sciNote inside Docker Machine - the trouble is that Docker Machine is basically a lightweight Linux VirtualBox VM inside which Docker runs, and is very slow (due to nested virtualization), as well as having problems with file system mounting.

Windows 10

On Windows 10, you can use Docker for Windows - this will install Docker inside a Hyper-V container, which is much faster than Docker Machine. We still suggest to ramp-up the VM's memory to at least 4GB though (right click Docker icon in Windows taskbar -> Settings... -> Advanced).

  1. Install latest version of Docker for Windows.
  2. Check version of docker-compose by running docker-compose -v from command prompt. It needs to be at least version 1.9.0 for sciNote docker-compose scripts to work. If you have an older version of docker-compose, download newer version for Windows from releases repository, and overwrite the .exe file of current docker-compose (it should be located in <docker-install-dir>/Docker/resources/bin directory).
  3. Follow the Quick Start Guide above.

Important! As of right now, sciNote does not provide any Windows make script (Linux make command will not work on Windows), therefore you need to call docker-compose commands directly. See Makefile to inspect which docker-compose calls make commands execute.

Docker structure

For version 1.10.0 and later When starting sciNote application in development mode it runs in a Docker container called scinote_web_development, which uses volume scinote_development_files for storing uploaded files and scinote_development_bundler for storing program dependencies. The database runs in a separate container, called scinote_db_development. This database container makes use of a persistent volume called scinote_development_postgres for storing database files.

For versions prior to 1.10.0 The main sciNote application runs in a Docker container called web. The database runs in a separate container, called db. This database container makes use of a special, persistent container called dbdata.

For running application in production please check this section

Commands

Call make commands to build Docker images and build Rails environment, including database.

Following commands are available:

Command Description
make docker Downloads the Docker image and build Gems. This should be called whenever Gemfile is changed.
make db-cli Runs a /bin/bash inside the db container.
make run Runs the db container & starts the Rails server in web container.
make start Runs the db container & starts the Rails server in web container in background.
make stop Stops the db & web containers.
make worker Runs the rake jobs:work (worker process) in web container.
make cli Runs a /bin/bash inside the web container.
make tests Execute all Rails tests.
make console Enters the Rails console in web container.
make export Zips the head of this Git repository into a .tar.gz file.

Environmental variables

sciNote reads configuration parameters from system environment parameters. On production servers, this can be simply be system environmental variables, while for development, a file config/application.yml can be created to specify those variables.

Below is an example application.yml file, which should get the server running; for full reference, check the below table of all available variables.

# Secrets
SECRET_KEY_BASE: "<<secret_key_base>>"
PAPERCLIP_HASH_SECRET: "<<paperclip_hash_secret>>"

# Mailer settings
MAIL_FROM: "Example sciNote Assistant <[email protected]>"
MAIL_REPLYTO: "Example sciNote Assistant <[email protected]>"
SMTP_ADDRESS: "your.yoursmtp.com"
SMTP_PORT: "25"
SMTP_DOMAIN: "yoursmtp.com"
SMTP_USERNAME: "[email protected]"
SMTP_PASSWORD: "<<smtp_password>>"
MAIL_SERVER_URL: "scinote.yourdomain.com"

# File storage

# For local file storage
PAPERCLIP_STORAGE: "filesystem"

# For S3 file storage
#PAPERCLIP_STORAGE: "s3"
#AWS_ACCESS_KEY_ID: ""
#AWS_REGION: ""
#AWS_SECRET_ACCESS_KEY: ""
#S3_BUCKET: ""
#S3_HOST_NAME: ""

# This is needed for 'better errors' gem in development mode
TRUSTED_IP: "172.17.0.1"

# Various settings
ENABLE_TUTORIAL: "true"

NEW_TEAM_ON_SIGNUP: "true"

ENABLE_EMAIL_CONFIRMATIONS: "false"

ENABLE_RECAPTCHA: "true"
ENABLE_USER_REGISTRATION: "true"
RECAPTCHA_SITE_KEY: "<<recaptcha_site_key>>"
RECAPTCHA_SECRET_KEY: "<<recaptcha_secret_key>>"

NAVBAR_SHOW_VERSION: "false"

DELAYED_WORKER_DESTROY_FAILED_JOBS: "false"
DELAYED_WORKER_SLEEP_DELAY: "60"
DELAYED_WORKER_MAX_ATTEMPTS: "6"
DELAYED_WORKER_MAX_RUN_TIME: "30"
DELAYED_WORKER_READ_AHEAD: "10"
DELAYED_WORKER_DEFAULT_QUEUE_NAME: "default"

The following table describes all available environmental variables for sciNote server.

Variable Mandatory Description
General
SECRET_KEY_BASE Yes Random hash for Rails encryption. Can be generated by running rake secret.
PAPERCLIP_HASH_SECRET Yes Random key for generating Paperclip hash key for URLs. Can be generated via following Ruby function (from e.g. irb): SecureRandom.base64(128).
ENABLE_TUTORIAL Yes Whether to display tutorial (and auto-generate demo project) to first-time users. Defaults to false on development, and to true on production.
NAVBAR_SHOW_VERSION No Whether to show application version in the navbar. Defaults to true on development and test environments, and to false on production environment.
File storage
PAPERCLIP_STORAGE Yes Set to 's3' to store files on Amazon S3, or 'filesystem' to store files on local server. If storing on S3, additional parameters need to be specified.
AWS_SECRET_ACCESS_KEY No* If storing files on Amazon S3, this must contain access key for accessing AWS S3 API.
AWS_ACCESS_KEY_ID No* If storing files on Amazon S3, this must contain access key ID for AWS S3.
S3_BUCKET No* If storing files on Amazon S3, this must contain S3 bucket on which files are stored.
S3_HOST_NAME No* If storing files on Amazon S3, this must contain the S3 service host name.
AWS_REGION No* If storing files on Amazon S3, this must contain the AWS region.
Mailer
MAIL_FROM Yes The from address for emails sent from sciNote.
MAIL_REPLYTO Yes The reply to address for emails sent from sciNote.
SMTP_ADDRESS Yes The server address of the SMTP mailer used for delivering emails generated in sciNote.
SMTP_PORT Yes The port of the SMTP server. Defaults to 587.
SMTP_DOMAIN Yes The server domain of the SMTP mailer used for delivering emails generated in sciNote.
SMTP_USERNAME Yes The username for SMTP mailer used for delivering emails generated in sciNote.
SMTP_PASSWORD Yes The password for SMTP mailer used for delivering emails generated in sciNote.
MAIL_SERVER_URL Yes The root URL address of the actual sciNote server. This is used in sent emails to redirect user to the correct sciNote server URL. Defaults to localhost.
Sign-up process
NEW_TEAM_ON_SIGNUP No Enables the creation of new team on sign up. If set false no team will be created on sign up. Defaults to true.
ENABLE_EMAIL_CONFIRMATIONS No Whether to require users to confirm the following actions: signing-up, signing-up after being invited, and changing their e-mail. Defaults to true on development and production environments, and to false on test environment.
ENABLE_RECAPTCHA No Whether to enable Google reCaptcha on sign-up and confirm invitation pages. Defaults to false.
ENABLE_USER_REGISTRATION No Whether to publicly enable users to register/sign-up. Defaults to true.
RECAPTCHA_SITE_KEY No* If using reCaptcha, this is the site key (see reCaptcha docs).
RECAPTCHA_SECRET_KEY No* If using reCaptcha, this is the secret key (see reCaptcha docs).
Office Online integration
WOPI_ENABLED No Whether to enable Office Online integration. Defaults to false.
WOPI_TEST_ENABLED No* Whether to enable testing of WOPI integration by opening .wopitest files. Defaults to false.
WOPI_DISCOVERY_URL No* If using WOPI, this is the discovery URL provided by Microsoft.
WOPI_ENDPOINT_URL No* If using WOPI, this is the endpoint URL on which the WOPI API will be available.
WOPI_BREADCRUMBS_HOST No* If using WOPI, this is the URL of sciNote server to be reachable from Microsoft Office Online.
USER_SUBDOMAIN No* If using WOPI, the only subdomain that will be allowed to access all non-WOPI routes.
WOPI_SUBDOMAIN No* If using WOPI, the WOPI subdomain.
Delayed worker
DELAYED_WORKER_DESTROY_FAILED_JOBS No If you want to keep failed jobs, set DELAYED_WORKER_DESTROY_FAILED_JOBS to false. The failed jobs will be marked with non-null failed_at. Default to false
DELAYED_WORKER_SLEEP_DELAY No If no jobs are found, the worker sleeps for the amount of time specified by the sleep delay option. Default to 60 seconds.
DELAYED_WORKER_MAX_ATTEMPTS No The default is 6 attempts. After this, the job either deleted or left in the database with "failed_at" set depends on the DESTROY_FAILED_JOBS value
DELAYED_WORKER_MAX_RUN_TIME No The default DELAYED_WORKER_MAX_RUN_TIME is 30.minutes. If your job takes longer than that, another computer could pick it up. It's up to you to make sure your job doesn't exceed this time. You should set this for the longest time you think the job could take.
DELAYED_WORKER_READ_AHEAD No The defaults behavior is to read 10 jobs from the queue when finding an available job. You can configure this by setting.
DELAYED_WORKER_DEFAULT_QUEUE_NAME No Set the default queue name
Misc
NEW_RELIC_LICENSE_KEY No New Relic code metrics license key, in case you want to log code metrics with New Relic service.

Production mode (in Docker containers)

Running server in production mode

New functionality for running in production mode was introduced with sciNote version 1.10.0. In order to run sciNote application in production inside Docker you'll need the following:

  • docker engine (version 1.12.0+),
  • docker-compose (version 1.6.0+).

You'll also need to set environment variables in production.env file; variables are the same as described in Environmental variables section, the only differences are:

  • format of records is changed - use KEY=VALUE record format;
  • you should set DATABASE_URL=postgresql://postgres@db/scinote_production without quotation marks.

The mentioned variable is used to configure database connection and by default it contains credentials to connect to the PostgreSQL database inside db container; you will need to change this variable if you decide to use another DB or set password for default db user.

You can generate default (minimal) production.env file with the following command: make config-production. This command will also generate SECRET_KEY_BASE and PAPERCLIP_HASH_SECRET variables automatically.

After setting all the required variables you need to build the Docker image with make docker-production; this command will install all needed Gems and precompile assets.

When the above build is finished, database needs to be initialized (if you're performing a new install). This can be done by running make database-production. This will initialize the database and start a running process; the database will be hosted in the scinote_db_production container. This container is configured to keep all database files in the persistent Docker volume named scinote_production_postgres, which is (by default) placed in /var/lib/docker/volumes of the host system.

There is also another persistent Docker volume called scinote_production_files; this container is used for storing generated and uploaded files if you configure local file storage.

All these volumes will be kept safe in case you delete your Docker containers. Running docker rm with -v flag, however, will also delete these volumes (same thing happens if you run docker-compose down with -v flag).

Background jobs worker will be running in another separate container called scinote_jobs_production.

Now (finally), you can start sciNote with the following command: docker-compose -f ./docker-compose.production.yml up. Or you can use docker-compose -f ./docker-compose.production.yml up -d to start the server in background mode. Please consult docker-compose documentation for the rest of supported commands.

In short, the sequence of commands to setup sciNote (for fresh install) would be:

make config-production
make docker-production
make database-production

docker-compose -f ./docker-compose.production.yml up

Migrating to new production setup

If you are using some old/custom setup, and want to switch to production mode which was introduced in version 1.10.0 (see above), you will need to perform the following steps before updating the application files.

1. Migrate database data

You will need to migrate database data from old database to the new one running in scinote_db_production container. The easiest way to do this is to dump data from the old container and restore it into the new one. You can do this by using your preferred database client and login credentials from your config file. For example, you can do it inside the web container after entering shell with make cli and pg_dump or with any GUI tool (like pgAdmin). You can find the IP address of the local database container by running docker inspect and inspecting the output (in most cases, name of the database container will be similar to e.g. web_db_1).

Important! You should extract the old database dump before updating the source code tree of the application.

2. Migrate files

If you are using local file storage, you will also need to move user's files from ./public/system directory in application directory to the Docker volume scinote_production_files.

3. Migrate ENV variables

Don't forget to migrate ENV variables from config/application.yml into production.env. See previous section on rules for production.env.

4. Update source files

After all of the above steps are done, you can finally update application files and run DB migrations as described here.

Upgrading sciNote installation running in production mode

After downloading the new version of sciNote application files, the following needs to be done to upgrade a running sciNote instance running in production Docker mode.

1. Rebuild Docker images

First of all, you will need to rebuild Docker images using the new version of sciNote. That means that you need to stop the current instance (if it is running), and then run the command make docker-production.

2. Migrate database data

Open command prompt inside the web container with make cli-production and, once inside it, run migrations with the following command: rake db:migrate. On rare occasions, you will also need to run additional migration scripts, please check Release Notes for that.

3. Restart

After migrations are finished, you can start the application normally with the command docker-compose -f ./docker-compose.production.yml up.

Rake tasks

Delayed jobs

sciNote uses delayed jobs library to do background processing, mostly for the following tasks:

  • Sending emails,
  • Extracting text from uploaded files (full-text search),
  • Generating introduction tutorials,
  • Copying/cloning large database objects,
  • etc.

Best option to run delayed jobs is inside a worker process. To start a background worker process that will execute delayed jobs, run the following command:

rake jobs:work

To clear all currently queued jobs, you can use the following command:

rake jobs:clear

Warning! This is not advised to do on production environments.

Adding users

To simplify adding of new users to the system, couple of special rake tasks have been created.

The first, rake db:add_user simply queries all the information for a specific user via STDIN, and then proceeds to create the user.

The second task, rake db:load_users[file_path,create_orgs] takes 2 parameters as an input:

  • Path to .yml file containing list of users & teams to be added. The YAML file needs to be structured properly - field names must match those in the database, users need to have a name user_<id>, and teams name team_<id>. For an example load users file, see db/load_users_template.yml file.
  • A boolean ('true' or 'false') whether to create individual teams for each user or not.

Web statistics

To check current login statistics of registered users, use rake web_stats:login task. To check the last login of any user to the system, use rake web_stats:last_login task.

Clearing data

Execute rake data:clean_temp_files to remove all temporary files. Temporary files are used when importing samples. Execute rake data:clean_unconfirmed_users to remove all users that registered, but never confirmed their email. Calling rake data:clean will execute both above tasks.

Notifications

To generate a new system notification that will be sent to all sciNote users, execute rake notifications:new_system[title, message]. Title & message can contain HTML tags.

Mailer

sciNote needs a configured SMTP mail server to work properly. See environmental variables for configuration of the mailer.

Office Online integration

sciNote can be integrated with an Office Online Server to enable viewing and edition of documents directly in the browser. To enable this integration using the Microsoft Office Online servers, you will need to join the Office Cloud Storage Partner Program and set up the WOPI_* environment variables.

Deploy onto Heroku

Before deploying to Heroku, install heroku client as describe on offical website. To use existing heroku application, add new git remote repository.

git remote add heroku [email protected]:my-random-app-name.git

Or create new heroku application by executing following command.

heroku create

Add additional heroku buildpacks in the same order as specified in .buildpacks:

heroku buildpacks:add --index <i> <buildpack>

e.g. for adding graphviz write:

heroku buildpacks:add --index 2 https://github.com/weibeld/heroku-buildpack-graphviz.git

Before pushing to heroku master branch, some environmental variables should be set.

Heroku environmental variables

For deployment of sciNote onto Heroku, additional environmental variables need to be specified.

Variable Mandatory Description
LANG Yes The default localization language (e.g. en_US.UTF-8).
RAILS_ENV Yes Rails environment: production, test or development.
RACK_ENV Yes Rack environment: production, test or development.
RAILS_SERVE_STATIC_FILES Yes Whether to serve static files. Must be set to enabled.
WEB_CONCURRENCY Yes The concurrency of the server. See Heroku specifications for details.
MAX_THREADS Yes The max. number of threads. See Heroku specifications for details.
PORT Yes The port on which the application should run. See Heroku specifications for details.
RAILS_FORCE_SSL Yes If set to 1, enforce SSL communication on all levels of application.
DATABASE_URL Yes Full URL for connecting to PostgreSQL database.

Testing

In current version, only model tests are implemented for sciNote. To execute them, call rake test:models.

Home

Clone this wiki locally