Skip to content

Building rawproc

Jump to bottom
Glenn Butcher edited this page Apr 20, 2020 · 23 revisions

Building rawproc is a bit challenging. It uses a set of libraries that can be hard to de-conflict in your operating system. I've recently simplified things somewhat by consolidating the gimage library into the rawproc source tree, but that created some new challenges. One thing is probable; after successfully compiling rawproc, you'll probably have learned something new about how your operating system organizes libraries.

I'll describe the Linux and Windows situations separately. As of 0.6.2, the autoconf configure script is capable enough to handle both environments.

Prerequisites

rawproc requires the following libraries:

  • JPEG: libjpeg, 6b or better. TurboJPEG will also work.
  • TIFF: libtiff, 4.0.6 or better.
  • PNG: libpng, right now 1.6.34 is tested, needs to be this or better to handle the new PNG EXIF specification.
  • wxWidgets: 3.1.1 is required, rawproc depends on some of its file handling routines.
  • LittleCMS2: liblcms2, 2.8 works,but 2.9 is now available
  • LibRaw: libraw, most rawproc development was done with 0.17, but 0.18.6 is now available. If you want to keep up with new cameras, you'll need to set up to compile this one regularly. Of note is that the current release, 0.19, no longer includes the GPL2/3 demosaic packs, so I've turned off the raw parameters exposed by those routines. You can re-insert them for compiling by including -DOLD_LIBRAW in your CXXFLAGS variable on the ./configure command line, e.g., CXXFLAGS=-DOLD_LIBRAW

Linux

First, its significant to note that most Linux flavors have some version of libjpeg, libtiff, and libpng that will probably work. Further, wxWidgets also depends on these three libraries, so it's probably easiest to just use the OS libraries for both wxwidgets and rawproc. You'll probably need to install the -dev packages to make the header files available. The problem child will likely be libpng, as its API has changed with almost every major release, and some OSs don't use the latest version. As of 11/3/2018, Ubuntu 18.04 has current and rawproc/compatible libraries for everything but wxWidgets and Libraw, yea.

Second, it's probably most straightforward to compile wxWidgets. You can attempt to install it with your OS's packages, but it's not likely to be version 3.1. wxWidgets has a configure script, wx-config, that rawproc recognizes, and you don't even have to make install wxWidgets to use it. I previously noted problems using GTK3, but it works fine now and is the preferred version for a nice-looking GUI.

Here's the steps I used to make rawproc on Ubuntu 18.04:

  1. sudo apt install libjpeg-turbo8-dev libpng-dev libtiff5-dev liblcms2-dev liblensfun-dev

  2. Compile wxWidgets:

  • Get the source from http://wxwidgets.org/downloads/, unpack in a directory you can refer to later in the rawproc configure.
  • Make a build-linux directory under the top level wxWidgets directory, cd into it
  • ../configure with appropriate switches. Here's my ../configure:

../configure --disable-shared --enable-unicode

  • Inspect the configure output, insure the JPEG, TIFF, and PNG libraries are system-provided, not built-in. Also make sure GTK2 was found.
  • Run make. The result will be a bunch of .a static library files in lib/.
  • In your build-linux directory, note the presence of the wx-config script. You'll need that when you compile rawproc.
  1. Compile LibRaw:
  • Get the sources from https://www.libraw.org/download.
  • Unpack Libraw in a directory you can refer to later in the rawproc configure.
  • Make a build-linux directory under the top level Libraw directory, cd into it.
  • ../configure with appropriate switches. Here's my ../configure:

../configure --prefix=/usr/local --disable-shared --enable-openmp --enable-jpeg --enable-lcms --disable-examples

  • run make, then sudo make install. The headers and .a files will be installed in /usr/local/include and /usr/local/lib, respectively.
  1. And now, rawproc:
  • Get the sources from https://github.com/butcherg/rawproc, either a .zip or .tar.gz to unpack, or clone the git repository.
  • Make a build-linux directory under the top level rawproc directory, cd into it.
  • ../configure with appropriate switches. Here's my ../configure:

../configure --enable-lensfun --enable-demosaic --with-wx-config=/home/glenn/ImageStuff/wxWidgets-3.1.0/build-linux/wx-config

A few of the switches beg explanation:

  • --with-wx-config= provides the path to the wx-config script for the wxWidgets you just built. It'll be used to generate LDFLAGS and CPPFLAGS parameters to use the wxWidgets libraries.

  • Run make, then make makedoc. This will provide rawproc and img executables, as well as the rawprocdoc.zip help file.

  • Copy the rawproc.conf file from the source code directory to either the directory with the rawproc executable, or to ~/.rawproc/. You can customize the configuration parameters in rawproc at the Edit->Properties menu selection.

Windows

Note: Scroll down to the MSYS2 section. A lot easier to do than this, but I'm leaving it in for reference for the time being...

In some ways, compiling for Windows is more controllable than in Linux, as you have to assemble all the libraries yourself, and don't have to worry the operating system's assemblage already installed. But, that's a two-edged sword; You Have To Assemble All The Libraries Yourself, and getting the order and dependencies straight can be a pain. I'm going to address two situations here: 1) MinGW/MSYS on native Windows, and 2) mingw-w64 cross-compiling on Linux. I actually found #2 to be easier, but that's probably due to the doggy Surface 3 I used to work out #1. Pretty much the same general approach, but I'll illustrate the differences in the ./configure commands I used to to do each.

Native MinGW/MSYS

Firstoff, install MinGW and MSYS on your computer. If you already have a Visual Studio environment, ignore it, sorry, took long enough to work out MinGW/MSYS. For this missive, I used version MinGW-5.3. MSYS will install all the POSIX-based tools you'll need to build and install all the libraries and executables; wxWidgets docs will encourage you to use their Makefile.gcc with the CMD.exe shell, but I'll describe what/how for the POSIX build, which is much more straightforward to integrate with rawproc. You'll need to do all the following commands in a MSYS shell; what I did was to make a shortcut on my desktop to C:\MinGW\msys\1.0\msys.bat, which will open a POSIX shell window when you double-click it.

Secondoff, the order you build the libraries is important, due to the dependencies some of them have on the others. Gotta make sure libzip is there before libpng, for instance. Generally, the configure scripts will tell you if anything is missing, but in one case, Libraw's dependency on LittleCMS is checked with pkgconfig, which I haven't yet gotten to work in the MinGW/MSYS environment, so you won't get Libraw to work with it here. I hate dangling things like this, so I'll eventually figure it out. Okay, here we go...

Thirdoff, you need to consider a build organization approach. For building, I'm organizing by rawproc version, so I do all my building under a corresponding directory, for this example /d/Documents/rawproc7. Note the MSYS path conventions, e.g., /d/ for the Windows D:\ drive. In that directory, I unpack or git clone all the libraries as well as rawproc. One of the advantages of this is the ability to use relative paths to things in a way that'll let you drag things around the directory structure if you feel the need. However, I'm going to use absolute paths below for clarity.

Fourthoff, for MinGW on Windows you need to decide on where to put libraries so your subsequent builds can find them. You could just install them in the root MinGW lib and include directories along with everything else gcc needs, but I don't recommend it as that approach locks you into particular library versions for any other things you might want to compile. What I do here is to create a local/ directory under my build directory, and --prefix= to it in the configures. So, you'll see a --prefix=/d/Documents/rawproc7/local in almost all the ../configure invocations below. Really, doing it this way makes the build a lot cleaner. Note: MSYS2 has a proper /usr directory, so anything you configure/compile that installs to /usr/local will install there just fine, and be available for -Including and -linking. Okay, here we go...

  1. libz: Needed by quite a few of the subsequent packages, so it gets installed first. Actually, the MinGW/MSYS environment will let you install it to the MinGW root, and that's the easiest way to do this. MinGW comes with a GUI package manager, which you can run from the MSYS shell:

$ mingw-get

Under "MinGW Standard Libraries", make sure mingw-libz dev is installed.

  1. libjpeg: Also needed by a few of the subsequent packages. You can use either the JPEG6b implementation released by the IJG or TurboJPEG; TurboJPEG has an advantage in it's use of a more recent autoconf for its configure script. Once you have the source package, from the top build directory:
$ tar -xzvf libjpeg-turbo-1.5.1.tar.gz
$ cd libjpeg-turbo-1.5.1
$ mkdir build
$ cd build
$ ../configure --disable-shared --prefix=/d/Documents/rawproc7/local
$ make
$ make install

This'll put the headers and libraries in the /d/Documents/rawproc7/local/include and /d/Documents/rawproc7/local/lib directories, respectively.

  1. libtiff: Needs to see libz and libjpeg, so to its configure we're going to add CPPFLAGS= and LDFLAGS= to point to the local/ directory where we're installing stuff. Get the tiff source package, and do this:
$ tar -xzvf tiff-4.0.9.tar.gz
$ cd tiff-4.0.9
$ mkdir win    #tiff already has a build directory...
$ cd win
$ ../configure --disable-shared --prefix=/d/Documents/rawproc7/local CPPFLAGS=-I/d/Documents/rawproc7/local/include LDFLAGS=-L/d/Documents/rawproc7/local/lib
$ make
$ make install

Same result as for libjpeg, but note as ../configure runs, it finds libz and libjpeg using the CPPFLAGS and LDFLAGS -I and -L paths. You can even copy/paste this --prefix/CPPFLAGS/LDFLAGS part of the ../configure line, because we'll use it a lot in the subsequent builds.

  1. libpng: Needs libz.
$ tar -xzvf libpng-1.6.34.tar.gz
$ cd libpng-1.6.34
$ mkdir build
$ cd build
$ ../configure --disable-shared --enable-hardware-optimizations --prefix=/d/Documents/rawproc7/local CPPFLAGS=-I/d/Documents/rawproc7/local/include LDFLAGS=-L/d/Documents/rawproc7/local/lib
$ make
$ make install

TIFF loading and saving can take a while, so I included --enable-hardware-optimizations. I haven't tested it, however. Note the rest of the ../configure, starting to see the pattern?

  1. LittleCMS2: Needs libjpeg.
$ tar -xzvf lcms2-2.9.tar.gz
$ cd lcms2-2.9
$ mkdir build
$ cd build
$ ../configure --disable-shared --prefix=/d/Documents/rawproc7/local CPPFLAGS=-I/d/Documents/rawproc7/local/include LDFLAGS=-L/d/Documents/rawproc7/local/lib
$ make
$ make install

I did have a bit of trouble, lcms configure couldn't see libjpeg, so I used their --with-jpeg= switch to point to the /d/Documents/rawproc7/local/lib library. Try it without first, it may be something I was doing.

  1. Libraw: Needs libjpeg, LittleCMS. However, if it doesn't find LittleCMS don't worry it much, unless you want to use the Libraw/dcraw color profile capabilities other than the dcraw internal ones. I don't use them. Now, Libraw has two other packages containing additional demosaic routines; download them as well as the Libraw source, and unpack them as follows:
$ tar -xzvf LibRaw-0.18.6.tar.gz
$ cd LibRaw-0.18.6
$ tar -xzvf ../Libraw-demosaic-pack-GPL2-0.18.6.tar.gz
$ tar -xzvf ../Libraw-demosaic-pack-GPL3-0.18.6.tar.gz
$ #See below before building...
$ mkdir build
$ cd build
$ ../configure --disable-shared --enable-lcms --enable-openmp --disable-examples --prefix=/d/Documents/rawproc7/local CPPFLAGS="-I/d/Documents/rawproc7/local/include -DLIBRAW_NODLL" LDFLAGS=-L/d/Documents/rawproc7/local/lib
$ make
$ make install

Ha, hope you were reading the above before blindly copy/pasting it. Once you have the source tree unpacked, if you want to get OpenMP to work, you'll need to edit libraw/libraw_types.h and change the "#undef LIBRAW_USE_OPENMP" to "#define LIBRAW_USE_OPENMP" at about line 69 (Libraw 0.18.6). The developers don't allow OpenMP for any compiler/os they haven't tested, and the #undef turns it off for MinGW. I use it, haven't properly tested it, so use at your own risk.

  1. wxWidgets: This one proved challenging, but I finally got it to work relatively simply. wxWidgets also depends on libjpeg, libtiff, and libpng, and if it can't see them in the search paths, will compile its own copies in-tree with wx-prepended file names. All good until you go to link, then you can find a plethora of duplicate symbols between wxWidgets and your carefully built ones. If you follow the wxWidgets recommendations for building on Windows, you'll use the Makefile they supply for use in a CMD.exe shell, which will build the wx libraries. Also, nominal wxWidgets makes a script, wx-config, that computes all the compile and link flags needed for that particular build; the CMD.exe process doesn't build that script. There is a way with a separate Windows program, wx-config.exe, but getting that to work with the wxWidgets autoconf tools is a bit daunting.

So, my approach gets wxWidgets to compile under MinGW/MSYS, just like all the other stuff above. But even then there are a couple of glitches to work around. First, I tried with the wxWidgets-3.1.0.tar.gz available from their download link, but there are some MinGW-oriented bugs there that have been subsequently fixed. How I got things to work was to get the wxWidgets github master branch, git-cloning it into /d/Documents/rawproc7. You have to have Git for Windows installed to do that; if you don't want to install that, you can get a .zip file with the tree from the wxWidgets github page and extract wxWidgets-master to /d/Documents/rawproc7. Once you've done either, there are some special instructions for compiling straight from the git source tree you need to do, particularly copying the setup0.h to setup.h. AND, as of this writing (1/21/2018), you'll need to edit src/expat/expat/lib/xmlparse.c, comment out line 171 as follows:

//#error memmove does not exist on this platform, nor is a suitable substitute available

Geesh.

After that, wxWidgets compiles just fine like this:

$ cd wxWidgets
$ mkdir win      #yep, build/ is already taken...
$ cd win
$ ../configure --build=x86-winnt-mingw32 --disable-shared --disable-debug CPPFLAGS=-I/d/Documents/rawproc7/local/include LDFLAGS=-L/d/Documents/rawproc7/local/lib
$ make

Ta-da, wxWidgets should compile cleanly, did for me overnight. Note no 'make install', you're going to use it in-place with wx-config.

My goal is to eventually use a release source tree; when that works successfully, I'll modify these instructions accordingly.

  1. And finally, rawproc: You can git-clone the github repository to your build directory, or extract from a downloaded zip, just like wxWidgets. Once you have that:
$ cd rawproc
$ mkdir build
$ cd build
$ ../configure --enable-openmp --enable-optimization --with-pixeltype=half --with-wx-config=/d/Documents/rawproc7/wxWidgets/win/wx-config CPPFLAGS=-I/d/Documents/rawproc7/local/include LDFLAGS=-L/d/Documents/rawproc7/local/lib
$ make
$ make makedoc
$ cp ../rawproc.conf .

This will build img.exe and rawproc.exe in the build directory. You can move them wherever you want, or make desktop shortcuts to them in-place. The cp of rawproc.conf doesn't have to be done for every build; indeed, you're probably going to make changes to properties while using rawproc, and you don't want to write over them. Also, you don't have to 'make makedoc' every time, unless one of the files in ../doc changes.

The Makefile also has 'make zip' and 'make wininstaller' targets. The first one works in MSYS, the second one is currently configured for Wine under Linux, and also requires the Inno Installer program. If you have Inno, you can do it by hand with the rawproc.iss file made by ../configure.

MSYS2

This is the environment I use on my personal Windows machines. To start you need a MSYS2 environment that has the GCC C++ compiler, I'll leave that to you to figure out at https://msys2.org.

  1. Install required build support:
$ pacman -S msys/make mingw-w64-x86_64-make mingw-w64-x86_64-cmake autoconf automake-wrapper

I had to install these at various points, but some were due to alternative paths I later abandoned. YMMV...

  • bzip2 1.0.8-2
  • git 2.26.1-1
  • glib2-devel 2.54.3-3
  • gzip 1.10-1
  • msys2-runtime-devel 3.0.7-6
  • zip 3.0-3
  1. Install rawproc prerequisites:
$ pacman -S mingw-w64-x86_64-lcms2  mingw-w64-x86_64-libjpeg-turbo mingw-w64-x86_64-libpng mingw-w64-x86_64-libraw mingw-w64-x86_64-libtiff
  1. Build and install lensfun, librtprocess, and wxWidgets. The lensfun package in MSYS2 is version 0.3.95, which is not stable, so these instructions will get you 0.3.2. librtprocess isn't packaged in any distro yet. wxWidgets in MSYS2 is 3.0.2, rawproc uses functions in 3.1.3. I recommend building all three statically.

Lensfun:

$ git clone https://github.com/lensfun/lensfun.git
$ mkdir build && cd build
$ cmake -G "MSYS Makefiles" -DCMAKE_INSTALL_PREFIX="$MSYSTEM_PREFIX" -DBUILD_STATIC=ON ..
$ make
$ make install

librtprocess:

$ git clone https://github.com/CarVac/librtprocess.git
$ mkdir build && cd build
$ cmake -G "MSYS Makefiles" -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX="$MSYSTEM_PREFIX" -DCMAKE_BUILD_TYPE="Release" ..
$ make
$ make install

wxWidgets:

$ wget https://github.com/wxWidgets/wxWidgets/releases/download/v3.1.3/wxWidgets-3.1.3.tar.bz2
$ tar -xjvf wxWidgets-3.1.3.tar.bz2
$ cd wxWidgets-3.1.3 && mkdir build-win && cd build-win
$ ..configure --enable-unicode --disable-shared --disable-debug
$ make
(no make install, wxWidgets lets you link in-place...)
  1. Build rawproc:
$ git clone https://github.com/butcherg/rawproc.git
$ cd rawproc
$ aclocal
$ autoconf
$ automake --add-missing
$ mkdir build && cd build
$  ../configure --enable-librtprocess --enable-lensfun --enable-lensfunupdate --enable-openmp --with-wx-config=/home/<you>/wxWidgets-3.1.3/build-win/wx-config CXXFlags=-O3 LIBS="-llzma -lzstd -ljasper -lglib-2.0 -lregex"
$ make`

From here, you can make install; I just make desktop shortcuts to src/rawproc.exe. You'll need to put c:\msys2\mingw64\bin and c:\msys2\mingw64\lib in your PATH environment variable so the rawproc executable can find a few libraries.

mingw-w64 Cross-Compiling On Linux using mxe

This is the build system I use to compile the Windows architecture binaries releases. The website is at mxe.cc. The only drawback is the age of the compiler, 5.1 at this writing. Follow their instructions to make gcc, jpeg, png, tiff, lcms2, and lensfun statically, and use their instructions to compile wxWidgets and LibRaw with the --host parameter to configure. Works like a charm.

Clone this wiki locally