Switching Docker AUFS to Docker ZFS

Below I will show you, how to switch from the default Docker AUFS storage driver to the ZFS storage driver on Ubuntu 17.04.
First lets make sure that ZFS support is load in the kernel.

lsmod | grep zfs
zfs                  2711552  5
zunicode              331776  1 zfs
zavl                   16384  1 zfs
zcommon                49152  1 zfs
znvpair                77824  2 zcommon,zfs
spl                    98304  3 znvpair,zcommon,zfs

Next, lets install the ZFS utilities, this is needed to be able to initialize a ZFS Zpool.

apt-get install zfsutils-linux

Preparing a ZFS file system for Docker use

Since I don’t have any spare drives to play with, I will be using files to create the ZFS Zpool, feel free to replace that with /dev/sd* or whatever you like to use. I will be using the /zfsdevs to store the Zpool files.

mkdir /zfsdevs

Next, I am creating the files used to store/construct the Zpool file system.

# About 60Gb total
for i in {1..4}; do fallocate -l 15G /zfsdevs/file$i; done

Finally, created the Zpool, I named my pool zdocker.

zpool create -f zdocker /zfsdevs/file1 /zfsdevs/file2 /zfsdevs/file3 /zfsdevs/file4

Now lets change Docker to use the ZFS driver

First, lets stop the Docker daemon

service docker stop

Modify the DOCKER_OPTS in /etc/default/docker with the below ZFS options.

DOCKER_OPTS="--storage-driver=zfs zfs.fsname=zdocker/docker"

Move the old images aside, by coping the directory.

mv /var/lib/docker /var/lib/docker-org

Create the new zfs file system, this will be used by Docker.

zfs create -o mountpoint=/var/lib/docker zdocker/docker

Instruct docker to use the ZFS driver

touch /etc/docker/daemon.json
{
  "storage-driver": "zfs"
}

Finally, start the docker daemon(now with ZFS support).

service docker start

To make sure Docker is using the ZFS driver, run the below

docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 17.03.1-ce
Storage Driver: zfs
 Zpool: zdocker
 Zpool Health: ONLINE
 Parent Dataset: zdocker/docker
...[snip]

To make the ZFS mounts permanent, make sure to check the /etc/default/zfs options

grep MOUNT /etc/default/zfs
ZFS_MOUNT='yes'
ZFS_UNMOUNT='yes'

Docker using ZFS in action

First lets pull some images, like so.

docker pull ubuntu

You can check ZFS in action, by doing the below, about six snap’s(layers) are created for this image.

zfs list
NAME                                                                              USED  AVAIL  REFER  MOUNTPOINT
zdocker                                                                           131M  57.5G    19K  /zdocker
zdocker/docker                                                                    131M  57.5G   716K  /var/lib/docker
zdocker/docker/45858d92d11ea34c91599ac77af24640f85175b1eb332789ca5676dd9dabb6b8  96.5K  57.5G   126M  legacy
zdocker/docker/6c4044d3c2712e81387640004943dc81d8acfb4a8d4157ed9e28d405cd0224c2  37.5K  57.5G   106M  legacy
zdocker/docker/855bd21fe12ca3adc054eb371032f239bb5099fc1f4b7f80a9fc4dde6cdcbe5e   126M  57.5G   126M  legacy
zdocker/docker/cdbe0bde79cef5d336407aa9a1071860f536922354e5793d97f91abcef5fe3a1  4.29M  57.5G  4.29M  legacy
zdocker/docker/cf7ba8a9d54dea19f2ede6670f91cb2483f29b41cfd92f73f47b9b85c69b3975    54K  57.5G   106M  legacy
zdocker/docker/f7c84c515e88ba9de22578f233c80d92956822826adc2a7db3e11cb5e141b132    43K  57.5G   106M  legacy

Next, lets start some ubuntu containers, to see the full ZFS COW in action. This below will start three Ubuntu containers.

for i in {1..3} ;do docker run -it -d --name=ubuntu${i} ubuntu;done
63eff1d4d84bae1ab9553912a1a4cf621be04a4dcd5fe126701b305a12fcb042
b3efa40255331977f963768b239c0c4b2e38797bba70d4747b985895694e8158
f8d662bbd570fb3719d3fadc4dd1ce19a114a59a73d0693f0ea3d13846b3e052

Finally, You can see the ZFS snapshots below from the three new images created, once removed by docker rm they will go away.

zfs list        
NAME                                                                                   USED  AVAIL  REFER  MOUNTPOINT
zdocker                                                                                132M  57.5G    19K  /zdocker
zdocker/docker                                                                         131M  57.5G   780K  /var/lib/docker
zdocker/docker/45858d92d11ea34c91599ac77af24640f85175b1eb332789ca5676dd9dabb6b8       96.5K  57.5G   126M  legacy
zdocker/docker/5c476d97993605c1bc1e779b6a8db091e8bd255f94259bdad50e474a063bca01        114K  57.5G   106M  legacy
zdocker/docker/5c476d97993605c1bc1e779b6a8db091e8bd255f94259bdad50e474a063bca01-init    52K  57.5G   106M  legacy
zdocker/docker/6c4044d3c2712e81387640004943dc81d8acfb4a8d4157ed9e28d405cd0224c2       37.5K  57.5G   106M  legacy
zdocker/docker/855bd21fe12ca3adc054eb371032f239bb5099fc1f4b7f80a9fc4dde6cdcbe5e        126M  57.5G   126M  legacy
zdocker/docker/95ce14c78d5bcdb3bcfa62fca8d350d752a3fe07ee33147173a5648f884739b0        114K  57.5G   106M  legacy
zdocker/docker/95ce14c78d5bcdb3bcfa62fca8d350d752a3fe07ee33147173a5648f884739b0-init    52K  57.5G   106M  legacy
zdocker/docker/a128a60bf9967c1aa1a6e7cb2a4242b498aae0cce5692f1d62da84cf726e207c        114K  57.5G   106M  legacy
zdocker/docker/a128a60bf9967c1aa1a6e7cb2a4242b498aae0cce5692f1d62da84cf726e207c-init    52K  57.5G   106M  legacy
zdocker/docker/cdbe0bde79cef5d336407aa9a1071860f536922354e5793d97f91abcef5fe3a1       4.29M  57.5G  4.29M  legacy
zdocker/docker/cf7ba8a9d54dea19f2ede6670f91cb2483f29b41cfd92f73f47b9b85c69b3975         54K  57.5G   106M  legacy
zdocker/docker/f7c84c515e88ba9de22578f233c80d92956822826adc2a7db3e11cb5e141b132         43K  57.5G   106M  legacy

To clean this containers, you can just run the below, then verify the ZFS snap’s

# Stop the containers
docker ps -a|grep ubuntu |awk '{print $1}'|xargs docker stop 
f8d662bbd570
b3efa4025533
63eff1d4d84b

# Remove the containers
docker ps -a|grep ubuntu |awk '{print $1}'|xargs docker rm 
f8d662bbd570
b3efa4025533
63eff1d4d84b

# Finally, ZFS snaps back to normal
zfs list
NAME                                                                              USED  AVAIL  REFER  MOUNTPOINT
zdocker                                                                           132M  57.5G    19K  /zdocker
zdocker/docker                                                                    131M  57.5G   721K  /var/lib/docker
zdocker/docker/45858d92d11ea34c91599ac77af24640f85175b1eb332789ca5676dd9dabb6b8  96.5K  57.5G   126M  legacy
zdocker/docker/6c4044d3c2712e81387640004943dc81d8acfb4a8d4157ed9e28d405cd0224c2  37.5K  57.5G   106M  legacy
zdocker/docker/855bd21fe12ca3adc054eb371032f239bb5099fc1f4b7f80a9fc4dde6cdcbe5e   126M  57.5G   126M  legacy
zdocker/docker/cdbe0bde79cef5d336407aa9a1071860f536922354e5793d97f91abcef5fe3a1  4.29M  57.5G  4.29M  legacy
zdocker/docker/cf7ba8a9d54dea19f2ede6670f91cb2483f29b41cfd92f73f47b9b85c69b3975    54K  57.5G   106M  legacy
zdocker/docker/f7c84c515e88ba9de22578f233c80d92956822826adc2a7db3e11cb5e141b132    43K  57.5G   106M  legacy

Whats your experience using Docker and alternative file systems, please let me know in the comments below?
You might also like, Tips creating private docker registry with self signed certificate