Skip to content
This repository has been archived by the owner on Feb 13, 2020. It is now read-only.

Release/calendar server 9.1 docker #489

Open
wants to merge 2 commits into
base: release/CalendarServer-9.1
Choose a base branch
from

Conversation

Pluies
Copy link

@Pluies Pluies commented Jun 20, 2017

Hello!

This PR is not meant to be merged straight in, but to server as a discussion base :)

I've set up a Dockerfile for the calendarserver, is that of interest to you, dear maintainers?

If so...

  • Could you look through this Dockerfile and tell me whether the way it's built suits you, or whether there's things that should be fixed?
  • I had to disable OpenSSL in the script -- we should probably make that configurable, because this possibly breaks the base script
  • Should we put this Dockerfile the base folder and make it a bona fide supported way of installing the calendarserver?
  • Should we integrate with Docker Hub to provide official images?

Thanks!

  • Florent

@dreness
Copy link
Member

dreness commented Jun 22, 2017

Hi @Pluies! Your proposal is of interest, and although I don't have answers to all your questions right now, I will review the Dockerfile and we can go from there. Also I can say that we generally prefer to not install OpenSSL ourselves, and prefer to rely on a system-wide installation.

@dreness
Copy link
Member

dreness commented Feb 16, 2018

I played around with this a bit tonight, and in general I think it looks good, although I wasn't able to test it as written because in Docker for Mac, "Extended attributes are not yet supported."

Ignoring that problem for a moment, the general flow goes something like this (note: I'm kind of a docker noob):

  • install and set up Docker
mkdir ~/ccs-docker
cd ~/ccs-docker
git clone https://github.com/Pluies/ccs-calendarserver -b release/CalendarServer-9.1-docker
cd ccs-calendarserver
docker build -t ${USER}/jessie-ccs .

The end of a successful build looks something like:

...
Step 9/9 : CMD ["./bin/run", "-n"]
 ---> Running in 80f4bdcdebfd
Removing intermediate container 80f4bdcdebfd
 ---> 502792819e32
Successfully built 502792819e32
Successfully tagged dre/jessie-ccs:latest

Note that an attempt was made to start the container ("run -n"), but if you run docker ps, you don't see it. The reason will be clear soon. Let's manually start a container that maps the 'data' and 'conf' directories from the host to the container:

docker run --rm --name ccs \
  --volume /path/to/ccs-docker/ccs-calendarserver/data:/opt/ccs/data \
  --volume /path/to/ccs-docker/ccs-calendarserver/conf:/opt/ccs/conf \
  -p 8800:8800 -p 8443:8443 dre/jessie-ccs

... and it says:

Missing config file: /opt/ccs/conf/caldavd-dev.plist
You might want to start by copying the test configuration:

  cp conf/caldavd-test.plist conf/caldavd-dev.plist

Would you like to copy the test configuration now? [y/n]

Since the 'conf' directory being used is the one on the host, not in the container, run this (on the host):

cp conf/caldavd-test.plist conf/caldavd-dev.plist

re-run the 'docker run ...' command. This time, the service should start. To check the plumbing, open a web browser and hit https://localhost:8443/ - maybe accepting any self-signed SSL cert warnings. If you see a collection listing, hooray! If you're on a Mac, it doesn't work due to the extended attribute problem mentioned earlier, and you see the following in the container:

IOError: [Errno 95] Operation not supported: '/opt/ccs/data/Data/Documents/.calendarserver_version'

This could be avoided by not using the 'osxfs' docker filesystem to mount the 'data' directory from the host. Don't store 'data' inside the container unless you're very confident that you know how to backup / restore / preserve that data directory (which contains every bit of the data created by users of this service). Of particular note: the 'rm' option to 'docker run' deletes the container after it exits, which is probably not healthy unless all the persistent data lives (or is reliably synced NOT USING FILE UTILS because that's a great way to break a running database) outside the container. It's probably possible to use nfs instead of osxfs for the 'data' directory by nfs exporting it from the host and mounting it in the container at /opt/ccs/data/.

@Pluies
Copy link
Author

Pluies commented Feb 18, 2018

Ah, I hadn't realised Docker for Mac mounted the folder differently and CCS then relied on extended attributes - I run this particular container inside a Debian box and everything is hunky-dory. After nearly a year of using this setup, everything is still rock solid and the only issues I've ran into were due to my SSL cert needing renewal :)

Regarding volumes, indeed, NFS could work, or any other non-OSX volume mounting system - I'd expect this to work nicely in e.g. Kubernetes' PersistentVolumes.

@stefanoco
Copy link

Trying to use this container for test purposes, and it seems to me that mailgateway operations (invites managed by email with IMAP) are missing, no activity and no errors. I configured a working IMAP gateway but no sign of activity...

@dreness
Copy link
Member

dreness commented Mar 30, 2018

@stefanoco Hi - did you perhaps find our FAQ item about this? https://www.calendarserver.org/FAQ.html. The mail gateway stuff requires no extra code-level dependencies, so should work as long as the IMAP and SMTP credentials are valid on a real mail server (local or external). It will poll the mailbox, so you don't need any inbound connectivity (other than whatever is needed if you're hosting a mail service also).

@stefanoco
Copy link

Yes @dreness sorry for not mentioning, I followed the faq and other examples and configured into conf/caldavd-dev.plist credentials for accessing dedicated email account in a remote SMTP and IMAP service.

@borgoat
Copy link

borgoat commented Mar 31, 2018

Hi all!

Was working on the same point as @Pluies as I wanted to run CCS on OpenShift.
I'm trying to achieve a "productive" image, ie not using bin/develop (as according to the docs bin/package should be used for this). I saw @dreness reply on #488 , if more documentation or at least some hints were available I'd be happy to implement the Dockerfile.
The main reasons why I don't like either bin/develop nor bin/package is that they install Postgres and memcached, however, I'd rather run those as separate containers.

Thanks!
Giorgio

@borgoat
Copy link

borgoat commented Apr 1, 2018

Hi

Just a quick update as I made some progress with this.
Here you can find the Dockerfile.

I simply used pip install -r requirements-default.txt and everything was installed under /usr/local.
Postgres and Memcached are running as separate containers, what I have to do now though is the setting of passwords and hosts, as right now this is hardcoded in etc/caldavd/caldavd.plist.

I'll do this then also run it on OpenShift to test a cluster deployment (despite the database lots of folders are created, some are for attachments and have to be shared between all nodes, don't know about the others...)
Once I've done all this I'll be happy to contribute to the documentation.

If anyone else is interested in this please let me know!

Cheers
Giorgio

@dreness
Copy link
Member

dreness commented Apr 12, 2018

Looks cool, @giorgioazzinnaro - I'm interested to hear your experiences with this.

@borgoat
Copy link

borgoat commented Apr 14, 2018

Hey @dreness , glad to know that there's some interest for this.
I actually moved the work here master...giorgioazzinnaro:master
So that if you like it can eventually be merged.

As mentioned earlier, I'm now focusing on an OpenShift deployment (because I have a production use case), however, different images could be created for plain Docker, K8S, or OpenShift. The issue I'm having right now is because OpenShift runs with a random UID, but calendarserver seems to expect the UID to exist in /etc/passwd, I'm still trying to understand better.

If you like, I could recreate the deployment into OpenShift Online so that I can share the progress

@dreness
Copy link
Member

dreness commented Apr 16, 2018

@giorgioazzinnaro I'll just keep an eye on what you're doing for now. I think you're the first person to want to run CCS in an OpenShift context (that I've heard about, anyway), but I still wouldn't mind shipping extra docker files under contrib or something. Feel free to open a new PR when you're ready.

Looks like you figured out the stuff with the random UID - you're basically the same thing that the OpenShift docs suggest. I guess they do the random UID thing to help encourage stateless behavior, heh.

Also, you might find some useful info tidbits in the MultiserverDeployment doc. Even if you're only hosting with a single instance of CCS, it sounds like you're using a configuration similar to what could be used in a cluster. In particular, I don't see any shared storage in your config, which would be needed to store calendar event attachments.

@borgoat
Copy link

borgoat commented Apr 18, 2018

When you mention the shared disk between nodes, isn't it correct that I should share /var/db/caldav (or ServerRoot)? Right now I configured that as a Docker volume.
So it will be persisted across restarts, and shared in different pods in Kubernetes.

@dreness
Copy link
Member

dreness commented Apr 18, 2018

@giorgioazzinnaro The only hard requirement for shared filesystem space (in a multi-server operation) is to support file attachments. Other stuff can be shared as well, with a few important exceptions:

  • RunRoot
  • LogRoot
  • SocketRoot

Those directories should be kept separate (unless you configure unique paths for all items contained by each). The default relative locations of the various configurable paths are defined in the RELATIVE_PATH array in twistedcaldav/stdconfig.py. Note also that the config parser supports an 'include' directive which is useful to define server-specific configuration.

@mschoengens
Copy link

@giorgioazzinnaro Thanks for providing a Dockerfile for CCS. I was actually thinking that it might be interesting to have also a docker-compose file for smaller deployments (with separate Postgres and Memcached containers). Have you had any thoughts about this? Is there a more general interest?

@borgoat
Copy link

borgoat commented May 1, 2018

Hey @mschoengens , yes setting up a Docker Compose or Stack is also an option. Right now I focusing on openshift and plain k8s compatibility, with that in place compose should be a piece of cake.

I can tell you that right now I'm focusing on an openldap image, once I do that I'll write both the k8s/oc template and the docker-compose.yml

I'll keep pushing updates, let me know if you want to collaborate! Cheers!

@mschoengens
Copy link

Hi @giorgioazzinnaro, we could definitely collaborate on this. I actually created a Docker Compose file using a modified version of your Dockerfile. I can create a pull request to your repository, so you can have a look at it.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants