How to install Digdag Server on Windows

This guide work through the steps when you want to setup a digdag server on windows. I'm also installing embulk since it's much more beneficial when they are used together.

Components

Name	Type	Description
Chocolatey	Optional	The package manager for Windows https://chocolatey.org
JRE	Mandatory	Install JRE8 Embulk v0.9 series runs on Java 8. Java 9 is not supported in any version for the time being.
Postgresql	Mandatory	Digdag repository Digdag stores all data in a database (PostgreSQL or H2 database). https://www.postgresql.org/
Digdag	Mandatory	SIMPLE, OPEN SOURCE, MULTI-CLOUD WORKFLOW ENGINE https://www.digdag.io/
Embulk	Mandatory	Embulk is a open-source bulk data loader that helps data transfer between various databases, storages, file formats, and cloud services. http://www.embulk.org/docs/
JDBC driver for SQL Server	Mandatory	Both embulk and digdag are developed using Jruby and need JDBC driver to query SQL server. https://docs.microsoft.com/en-us/sql/connect/jdbc/download-microsoft-jdbc-driver-for-sql-server?view=sql-server-2017
SQL Server Native Client	Mandatory	Need this to use "native" insert mode of Embulk https://github.com/embulk/embulk-output-jdbc/tree/master/embulk-output-sqlserver "native" means bulk insert using native client. It is faster than "normal". It requires both SQL Server JDBC driver and SQL Server Native Client (or newer version of Microsoft ODBC Driver). https://docs.microsoft.com/en-us/sql/relational-databases/native-client/sql-server-native-client?view=sql-server-2017 https://docs.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-2017
Python	Optional	For python operator https://www.python.org/downloads
Pip	Optional	pip is a package management system used to install and manage software packages written in Python.
Pyodbc	Optional	DB access https://github.com/mkleehammer/pyodbc

Install components

Chocolatey

Follow the guide from https://chocolatey.org/install Run the following code when using cmd.exe

@"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"

JRE

Both Digag and Embulk require JRE. Embulk v0.9 series runs on Java 8

C:\WINDOWS\system32>choco install jre8

Postgresql

You will be asked to provide password for postgres user.

C:\WINDOWS\system32>choco install postgresql

If you encounter "Failed to load SQL Modules into database Cluster" error.

Uninstall postgresql by running "C:\Program Files\PostgreSQL\10\uninstall-postgresql.exe"
Download installer manually from https://www.postgresql.org/download/windows/
Run cmd as adminstrator

C:\WINDOWS\system32>net user postgres /add
C:\WINDOWS\system32>net user postgres postgresql@1
C:\WINDOWS\system32>net localgroup administrators postgres /add
C:\WINDOWS\system32>net localgroup "Power Users" postgres /add
C:\WINDOWS\system32>runas /user:postgres cmd.exe

Run this in the new cmd window

C:\WINDOWS\system32>C:\Download\postgresql-9.4.14-1-windows-x64.exe

Finish installation

Delete postgres os user

C:\WINDOWS\system32>net user postgres /delete

Digdag

http://docs.digdag.io/getting_started.html

PowerShell -Command "& {[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::TLS12; mkdir -Force $env:ALLUSERSPROFILE\digdag\bin; Invoke-WebRequest http://dl.digdag.io/digdag-latest.jar -OutFile $env:ALLUSERSPROFILE\digdag\bin\digdag.bat}"

Above command downloads a file named digdag.bat to a folder c:\ProgramData\digdag\bin.
Add c:\ProgramData\digdag\bin to system path
After reopen your command window, run digdag --help to check if digdag was installed successfully.

C:\WINDOWS\system32>digdag --help
2018-11-04 10:03:26 -0800: Digdag v0.9.31
Usage: digdag <command> [options...]
  Local-mode commands:
    init <dir>                         create a new workflow project
    r[un] <workflow.dig>               run a workflow
    c[heck]                            show workflow definitions
    sched[uler]                        run a scheduler server
    selfupdate                         update cli to the latest version
  Server-mode commands:
    server                             start server

  Client-mode commands:
    push <project-name>                create and upload a new revision
    download <project-name>            pull an uploaded revision
    start <project-name> <name>        start a new session attempt of a workflow
    retry <attempt-id>                 retry a session
    kill <attempt-id>                  kill a running session attempt
    backfill <project-name> <name>     start sessions of a schedule for past times
    reschedule                         skip sessions of a schedule to a future time
    log <attempt-id>                   show logs of a session attempt
    workflows [project-name] [name]    show registered workflow definitions
    schedules                          show registered schedules
    disable <schedule-id>              disable a workflow schedule
    disable <project-name>             disable all workflow schedules in a project
    disable <project-name> <name>      disable a workflow schedule
    enable <schedule-id>               enable a workflow schedule
    enable <project-name>              enable all workflow schedules in a project
    enable <project-name> <name>       enable a workflow schedule
    sessions                           show sessions for all workflows
    sessions <project-name>            show sessions for all workflows in a project
    sessions <project-name> <name>     show sessions for a workflow
    session  <session-id>              show a single session
    attempts                           show attempts for all sessions
    attempts <session-id>              show attempts for a session
    attempt  <attempt-id>              show a single attempt
    tasks <attempt-id>                 show tasks of a session attempt
    delete <project-name>              delete a project
    secrets --project <project-name>   manage secrets
    version                            show client and server version

  Options:
    -L, --log PATH                   output log messages to a file (default: -)
    -l, --log-level LEVEL            log level (error, warn, info, debug or trace)
    -X KEY=VALUE                     add a performance system config
    -c, --config PATH.properties     Configuration file (default: C:\Users\XXXX\.config\digdag\config)

Use `<command> --help` to see detailed usage of a command.

Embulk

https://github.com/embulk/embulk#windows

PowerShell -Command "& {[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::TLS12; mkdir -Force $env:ALLUSERSPROFILE\digdag\bin; Invoke-WebRequest http://dl.embulk.org/embulk-latest.jar -OutFile $env:ALLUSERSPROFILE\digdag\bin\embulk.bat}"

Above command downloads a file named embulk.bat to a folder c:\ProgramData\digdag\bin.
Run embulk --help to check if embulk was installed successfully.

JDBC driver for SQL Server

https://docs.microsoft.com/en-us/sql/connect/jdbc/download-microsoft-jdbc-driver-for-sql-server?view=sql-server-2017
Download, unzip and copy below two files to a folder of your choice(i.e. c:\utils\drivers).

mssql-jdbc-7.0.0.jre8.jar
sqljdbc_auth.dll - choose the right version based on your os

You also need to copy sqljdbc_auth.dll to c:\windows\system32 to use windows authentication with mssql operator.

c:\Util\drivers>dir
 Volume in drive C is Windows
 Volume Serial Number is 5005-FE25
 
 Directory of c:\Util\drivers
 
10/17/2018  07:43 AM    <DIR>          .
10/17/2018  07:43 AM    <DIR>          ..
07/31/2018  10:23 AM         1,178,666 mssql-jdbc-7.0.0.jre10.jar
07/31/2018  10:23 AM         1,166,240 mssql-jdbc-7.0.0.jre8.jar
07/31/2018  10:23 AM           301,648 sqljdbc_auth.dll
               3 File(s)      2,646,554 bytes
               2 Dir(s)  36,017,549,312 bytes free
 
c:\Util\drivers>

SQL Server Native Client

Download the latest version of odbc and install it
https://docs.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-2017
Make sure that you select ODBC Driver for SQL Server SDK to be installed.

Python

Run cmd as admin mode and install python using choco.

C:\WINDOWS\system32>choco list python3
Chocolatey v0.10.11
python3 3.7.1 [Approved]
python3-x86_32 3.5.2.20170425 [Approved] - Possibly broken
miniconda3 4.5.11 [Approved]
pypy3 6.0.0 [Approved] Downloads cached for licensed users
4 packages found.
 
C:\WINDOWS\system32>choco install python3

pip

C:\WINDOWS\system32>choco install pip

pyodbc

C:\WINDOWS\system32>pip install pyodbc

Running Digdag in Server-mode

Configure Repository

Make sure that you change the db user password of your own.

C:\WINDOWS\system32>psql -U postgres
Password for user postgres:
psql (10.4)
WARNING: Console code page (437) differs from Windows code page (1252)
         8-bit characters might not work correctly. See psql reference
         page "Notes for Windows users" for details.
Type "help" for help.
 
postgres=# CREATE USER digdag PASSWORD 'digdag123';
CREATE USER
postgres=# CREATE DATABASE digdag WITH OWNER digdag;
CREATE DATABASE
postgres=# \c digdag
You are now connected to database "digdag" as user "postgres".
digdag=# GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO digdag;
GRANT
digdag=# CREATE EXTENSION "uuid-ossp";
CREATE EXTENSION
digdag=# ALTER USER digdag WITH SUPERUSER;
ALTER ROLE
digdag=# \q

Create Digdag Server configuration file

Make the following and place it on c:\ProgramData\digdag.

c:\ProgramData\digdag>notepad server.properties

Add below to the notepad and save
  
database.type = postgresql
database.user = digdag
database.password = digdag123
database.host = 127.0.0.1
database.port = 5432
database.database = digdag
 
c:\ProgramData\digdag>

Create log folders

mkdir logs
cd logs
mkdir tasklogs
mkdir accesslogs

Start Digdag in Server-mode

c:\ProgramData\digdag>digdag server --bind 0.0.0.0 --config server.properties -O logs/tasklogs -A logs/accesslogs -L logs/server.log

Validate if you can access UI(http://localhost:65432/) without any problem

Register Digdag as a windows service

Instructions from http://yajsw.sourceforge.net/

Download yajsw
Download: https://sourceforge.net/projects/yajsw
Copy downloaded file and Unzip to C:\ProgramData\digdag\ServiceWrapper folder

c:\ProgramData\digdag\ServiceWrapper>dir
 Volume in drive C is Windows
 Volume Serial Number is 5005-FE25
 
 Directory of c:\ProgramData\digdag\ServiceWrapper
 
10/30/2018  10:48 AM    <DIR>          .
10/30/2018  10:48 AM    <DIR>          ..
10/30/2018  10:48 AM    <DIR>          bat
10/30/2018  10:48 AM    <DIR>          bin
10/30/2018  10:48 AM    <DIR>          build
10/30/2018  10:48 AM    <DIR>          conf
10/30/2018  10:48 AM    <DIR>          doc
10/30/2018  10:48 AM    <DIR>          lib
08/12/2018  11:24 AM            11,358 LICENSE.txt
10/30/2018  10:48 AM    <DIR>          log
08/12/2018  11:24 AM               207 readme.txt
10/30/2018  10:48 AM    <DIR>          scripts
10/30/2018  10:48 AM    <DIR>          src
10/30/2018  10:48 AM    <DIR>          templates
10/30/2018  10:48 AM    <DIR>          tmp
10/30/2018  10:48 AM    <DIR>          webapps
08/31/2018  11:44 PM           789,584 wrapper.jar
08/31/2018  11:44 PM            34,084 wrapperApp.jar
08/12/2018  11:24 AM                91 yajsw.policy.txt

Start digdag in server-mode
If you didn't stop the previous exectuion, you can use that.
Check PID using Task Manager
Locate java.exe from Task Manager. If you find two the one with bigger size is the right. Smaller one is a java update process.

c:\ProgramData\digdag\ServiceWrapper>tasklist /v | findstr /i "java"
java.exe   50560 Services    0     38,184 K Unknown      N/A    0:27:27 N/A
java.exe   51312 Console     0    105,816 K Unknown      N/A    0:09:15 N/A

Execute genConfig.bat
51312 is the PID from previous Step.

c:\ProgramData\digdag\ServiceWrapper>cd bat
c:\ProgramData\digdag\ServiceWrapper\bat>genConfig.bat 51312

Stop digdag
Kill process from task manager or ctrl+c from the cmd where you start digdag in server-mode
Edit config file - wrapper.conf
C:\ProgramData\digdag\ServiceWrapper\conf\wrapper.conf
If you want to access DB using windows authentication, you need to use a windows account with DB access privilege.
- wrapper.ntservice.account=DigdagServerAccount
- wrapper.ntservice.password=MYPWD

#********************************************************************
# working directory
#********************************************************************
wrapper.working.dir=c:\\\\ProgramData\\\\digdag\\\\
 
#********************************************************************
# Application Console Properties
#********************************************************************
# Title to use when running as a console
wrapper.console.title=digdag
 
#********************************************************************
# Wrapper Windows Service and Posix Daemon Properties
#********************************************************************
# Name of the service
wrapper.ntservice.name=digdag
 
# Display name of the service
wrapper.ntservice.displayname=digdag
 
# Description of the service
wrapper.ntservice.description=digdag
 
# Service dependencies.  Add dependencies as needed starting from 1
# wrapper.ntservice.dependency.1=
 
# Mode in which the service is installed.  AUTO_START or DEMAND_START
wrapper.ntservice.starttype=AUTO_START
 
# Allow the service to interact with the desktop.
# wrapper.ntservice.interactive=false
 
# wrapper service user. only for windows. on *nix system this is root.
# see also wrapper.app.account
wrapper.ntservice.account=DigdagServerAccount
wrapper.ntservice.password=MYPWD
 
#********************************************************************
# genConfig: further Properties generated by genConfig
#********************************************************************
placeHolderSoGenPropsComeHere=
wrapper.java.app.jar = c:\\\\ProgramData\\\\digdag\\\\digdag.bat
wrapper.java.command = java
wrapper.app.parameter.1 = server
wrapper.app.parameter.2 = --bind
wrapper.app.parameter.3 = 0.0.0.0
wrapper.app.parameter.4 = --config
wrapper.app.parameter.5 = server.properties
wrapper.app.parameter.6 = -O
wrapper.app.parameter.7 = logs/tasklogs
wrapper.app.parameter.8 = -A
wrapper.app.parameter.9 = logs/accesslogs
wrapper.app.parameter.10 = -L
wrapper.app.parameter.11 = logs/server.log
wrapper.java.additional.1 = -Dio.digdag.cli.launcher=selfrun
wrapper.java.additional.2 = -XX:+AggressiveOpts
wrapper.java.additional.3 = -XX:TieredStopAtLevel=1
wrapper.java.additional.4 = -Xverify:none

Test setting
- Run CMD as administrator privilege
- Goto C:\ProgramData\digdag\ServiceWrapper\bat
- Execute your wrapped application as console application by calling runConsole.bat
- Check you can access UI(http://localhost:65432/)
- Closing the CMD window will stop rundeck Install the application as service call Run installService.bat
- Start the service by executing startService.bat Verify
- Check you can access UI(http://localhost:65432/)
Verify from windows service manager

C:\WINDOWS\system32>sc query digdag

SERVICE_NAME: digdag
        TYPE               : 10  WIN32_OWN_PROCESS
        STATE              : 4  RUNNING
                                (STOPPABLE, NOT_PAUSABLE, ACCEPTS_SHUTDOWN)
        WIN32_EXIT_CODE    : 0  (0x0)
        SERVICE_EXIT_CODE  : 0  (0x0)
        CHECKPOINT         : 0x0
        WAIT_HINT          : 0x0

C:\WINDOWS\system32>sc qc digdag
[SC] QueryServiceConfig SUCCESS

SERVICE_NAME: digdag
        TYPE               : 10  WIN32_OWN_PROCESS
        START_TYPE         : 2   AUTO_START
        ERROR_CONTROL      : 1   NORMAL
        BINARY_PATH_NAME   : java.exe -classpath C:\ProgramData\digdag\ServiceWrapper\wrapper.jar -Xrs -Dwrapper.service=true -Dwrapper.working.dir=C:\ProgramData\digdag -Djna_tmpdir=C:\ProgramData\digdag\ServiceWrapper\bat\/../tmp -Dwrapper_home=C:\ProgramData\digdag\ServiceWrapper\bat\/.. -Dwrapper.config=C:\ProgramData\digdag\ServiceWrapper\conf\wrapper.conf -Dwrapper.additional.1x=-Xrs -Djna_tmpdir=C:\ProgramData\digdag\ServiceWrapper\bat\..\tmp org.rzo.yajsw.boot.WrapperServiceBooter
        LOAD_ORDER_GROUP   :
        TAG                : 0
        DISPLAY_NAME       : digdag
        DEPENDENCIES       :
        SERVICE_START_NAME : DigdagServerAccount

Restart desktop and check if digdag service is running

Provide feedback

Saved searches

Use saved searches to filter your results more quickly