I have previously written about the gitlab CI runners that use docker. Yesterday I made some changes to procps and pushed them to gitlab which would then start the CI. This morning I checked and it said build failed – ok, so that’s not terribly unusual. The output from the runner was:
gitlab-ci-multi-runner 0.3.3 (dbaf96f) Using Docker executor with image csmall/testdebian ... Pulling docker image csmall/testdebian ... Build failed with Error: image csmall/testdebian: not found
Hmm, I know I have that image, it just must be the runner so, let’s see what images I have:
$ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
Now, I know I have images, I had about 10 or so of them, where did they go? I even looked in the /var/lib/docker directories and can see the json configs, what have you done with my images docker?
The first hint I got from stackexchange where someone lost their AUFS images and needed to load the aufs kernel module. Now I know there are two places or methods where docker stores its images. They are called aufs and devicemapper. There is some debate around which one is better and to be honest with what I do I don’t much care, I just want it to work.
The version of kernel is significant. It seems the default storage container was AUFS and this requires the aufs.ko kernel module. Linux 4.0 (the version shipped with Debian) does NOT have that module, or at least I couldn’t find it.
For new images, this isn’t a problem. Docker will just create the new images using devicemapper and everyone is happy. The problem is where you have old aufs images, like me. I want those images.
Rescue the Images
I’m not sure if this is the best or most correct way of getting your images, but for me it worked. I got the idea basically from someone who wanted to switch from aufs to devicemapper images for other reasons.
You first need to reboot and select at the grub prompt a 3.x kernel that has aufs support. Then when the system comes up, you should see all your images, like this:
$ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE csmall/testdebian latest 6979033105a4 5 weeks ago 369.4 MB gcc 5.1 b063030b23b8 5 weeks ago 1.225 GB gcc 5.1.0 b063030b23b8 5 weeks ago 1.225 GB gcc latest b063030b23b8 5 weeks ago 1.225 GB ruby 2.1 236bf35223e7 6 weeks ago 779.8 MB ruby 2.1.6 236bf35223e7 6 weeks ago 779.8 MB debian jessie 41b730702607 6 weeks ago 125.1 MB debian latest 41b730702607 6 weeks ago 125.1 MB debian 8 41b730702607 6 weeks ago 125.1 MB debian 8.0 41b730702607 6 weeks ago 125.1 MB busybox buildroot-2014.02 8c2e06607696 8 weeks ago 2.433 MB busybox latest 8c2e06607696 8 weeks ago 2.433 MB
What a relief to see this! Work out what images you need to transfer over. In my case it was just the csmall/testdebian one. You need to save it to a tar file.
$ docker save csmall/testdebian > csmall-testdebian.tar.gz
Once you have all your images you want, reboot back to your 4.x kernel. You then need to load each image back into docker.
$ docker load csmall-testdebian.tar.gz
and then test to see its there
$ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE csmall/testdebian latest 6979033105a4 5 weeks ago 368.3 MB
The size of my image was slightly different. I’m not too sure why this is the case but assume it is to do with the storage types. My CI builds now run, they’re failing because my test program is trying to link with the library (it shouldn’t be) but at least its not a docker problem.
Well that sucks. I have been considering bringing back aufs in the short term as overlayfs doesn’t play nice with LSMs and Tails needs both a union filesystem and AppArmor. This would be another reason to bring it back. Though I still hope we can drop it for stretch.
Could you open a release-notes bug for stretch to cover this?
I’ll open a bug tonight. It would be nice to have a warning. Not that I read them very often but someone will.
You might want to consider using the overlay storage driver that uses overlayfs (which is available in the 4.0 kernel). Just add –storage-driver=overlay to the daemon startup script. I have found it to be considerably faster than devicemapper.
Thanks for the tip. I basically was trying to get the images going and I’m not terribly worried about performance. My docker use-case is a CI runner which only runs for about a minute then sits around not doing much else. It seems that overlayfs has some issues for some people, see Ben’s comment below (I think it is below).