Linux 4.0 ate my docker images

I have previously written about the gitlab CI runners that use docker.  Yesterday I made some changes to procps and pushed them to gitlab which would then start the CI.  This morning I checked and it said build failed – ok, so that’s not terribly unusual. The output from the runner was:

gitlab-ci-multi-runner 0.3.3 (dbaf96f)
Using Docker executor with image csmall/testdebian ...
Pulling docker image csmall/testdebian ...
Build failed with Error: image csmall/testdebian: not found

Hmm, I know I have that image, it just must be the runner so, let’s see what images I have:

$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE

Now, I know I have images, I had about 10 or so of them, where did they go? I even looked in the /var/lib/docker directories and can see the json configs, what have you done with my images docker?

Storage Drivers

The first hint I got from stackexchange where someone lost their AUFS images and needed to load the aufs kernel module. Now I know there are two places or methods where docker stores its images. They are called aufs and devicemapper. There is some debate around which one is better and to be honest with what I do I don’t much care, I just want it to work.

The version of kernel is significant. It seems the default storage container was AUFS and this requires the aufs.ko kernel module.  Linux 4.0 (the version shipped with Debian) does NOT have that module, or at least I couldn’t find it.

For new images, this isn’t a problem. Docker will just create the new images using devicemapper and everyone is happy. The problem is where you have old aufs images, like me. I want those images.

Rescue the Images

I’m not sure if this is the best or most correct way of getting your images, but for me it worked. I got the idea basically from someone who wanted to switch from aufs to devicemapper images for other reasons.

You first need to reboot and select at the grub prompt a 3.x kernel that has aufs support. Then when the system comes up, you should see all your images, like this:

$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
csmall/testdebian latest 6979033105a4 5 weeks ago 369.4 MB
gcc 5.1 b063030b23b8 5 weeks ago 1.225 GB
gcc 5.1.0 b063030b23b8 5 weeks ago 1.225 GB
gcc latest b063030b23b8 5 weeks ago 1.225 GB
ruby 2.1 236bf35223e7 6 weeks ago 779.8 MB
ruby 2.1.6 236bf35223e7 6 weeks ago 779.8 MB
debian jessie 41b730702607 6 weeks ago 125.1 MB
debian latest 41b730702607 6 weeks ago 125.1 MB
debian 8 41b730702607 6 weeks ago 125.1 MB
debian 8.0 41b730702607 6 weeks ago 125.1 MB
busybox buildroot-2014.02 8c2e06607696 8 weeks ago 2.433 MB
busybox latest 8c2e06607696 8 weeks ago 2.433 MB

What a relief to see this! Work out what images you need to transfer over. In my case it was just the csmall/testdebian one. You need to save it to a tar file.

$ docker save csmall/testdebian > csmall-testdebian.tar.gz

Once you have all your images you want, reboot back to your 4.x kernel. You then need to load each image back into docker.

$ docker load csmall-testdebian.tar.gz

and then test to see its there

$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
csmall/testdebian latest 6979033105a4 5 weeks ago 368.3 MB

The size of my image was slightly different. I’m not too sure why this is the case but assume it is to do with the storage types.  My CI builds now run, they’re failing because my test program is trying to link with the library (it shouldn’t be) but at least its not a docker problem.