Using lxc or lib-container instead of Docker

[stuartpb]

It feels like we're fighting against Docker's features as frequently as we're using them. What we use Docker for:

• Giving each app its own network stack
• Isolating apps from the host system
• buildstep
• logging (badly)
• Having a root-equivalent user to run Nginx with

Basically it feels like the bulk of what Docker is built to offer is largely a hypervisor for the meta-structure we're already tracking in $PLUSHU_ROOT. I feel like we could make a better system by rolling something better fitting out of some combination of lxc and/or libcontainer, systemd, and iptables directly.

On the other hand, this works right now, having prebuilt Docker images is kind of nice for addons/services, and it's something that has a lot of external appeal. I don't know that the advantages in reducing overhead by removing Docker would be so great.

[stuartpb]

So, there are definitely at least two things here where Docker is a good solution, for Nginx and addons.

However, Docker's image/link model isn't so great for building/running apps. What I'd really like is a system that lets me do copy-on-write layers and cgroup functionality, but without having to reify a ton of images.

This might also help with seamless failover?