Category Archives: DevOps

Alpine Linux sucks for hosting Docker Containers

After setting up Alpine Linux to run docker containers and fixing a whole bunch of issues I encountered, I now came to the point where I tried to run a node.js app inside a Docker Container that runs on an Alpine Linux Host. But node immediately exited with a Segmentation Fault. After some experimenting, I went and just started the official node docker container in the latest version and started node inside. It still didn’t work.

~# docker run -it node:latest /bin/bash
[email protected]:/# node
Segmentation fault

The kernel logs provide only very limited insights:

kern.info kernel: [   35.465909] device vethe7ba7a4 entered promiscuous mode
kern.info kernel: [   35.466145] IPv6: ADDRCONF(NETDEV_UP): vethe7ba7a4: link is not ready
kern.info kernel: [   35.466151] docker0: port 1(vethe7ba7a4) entered forwarding state
kern.info kernel: [   35.466160] docker0: port 1(vethe7ba7a4) entered forwarding state
kern.info kernel: [   35.467156] docker0: port 1(vethe7ba7a4) entered disabled state
kern.warn kernel: [   35.492075] cgroup: docker (1684) created nested cgroup for controller "memory" which has incomplete hierarchy support. Nested cgroups may change behavior in the future.
kern.warn kernel: [   35.492078] cgroup: "memory" requires setting use_hierarchy to 1 on the root
kern.info kernel: [   35.527037] eth0: renamed from veth1db87ca
kern.info kernel: [   35.538820] IPv6: ADDRCONF(NETDEV_CHANGE): vethe7ba7a4: link becomes ready
kern.info kernel: [   35.538838] docker0: port 1(vethe7ba7a4) entered forwarding state
kern.info kernel: [   35.538843] docker0: port 1(vethe7ba7a4) entered forwarding state
kern.info kernel: [   35.538861] IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes ready
kern.info kernel: [   36.539627] node[1930]: segfault at 10 ip 0000000000a6444d sp 0000750c38bd8550 error 4 in node[400000+13c7000]
kern.alert kernel: [   36.539643] grsec: Segmentation fault occurred at 0000000000000010 in /usr/local/bin/node[node:1930] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:1919] uid/euid:0/0 gid/egid:0/0
kern.alert kernel: [   36.539781] grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/local/bin/node[node:1930] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:1919] uid/euid:0/0 gid/egid:0/0

First, some network config changes done by docker, then docker complains about missing support for nested cgroups, then some more network config changes before node crashes with a segmentation fault. And eventually some error messages because GRSEC prevents node from writing a core dump.

After a lot of searching, there seems to be a whole list of problems in Alpine Linux that prevent it from reliably running Docker Containers, among them GRSEC, problems with LXC networking and missing cgroups hierarchy support. So basically all of the things that show up in the Kernel logs around the Segmentation fault.

By the way, according to the Alpine Bug Tracker, support for nested cgroups was added to Alpine 3.2 – but although I am running 3.3.1, it still isn’t working.

Interestingly enough, in the few days my post about installing Docker Daemon on Alpine Linux has been online, it has already helped somebody else fix the same problem but also lead him to the conclusion that Alpine Linux is not so great as a host for Docker containers.

As with all Linux distributions, if I can’t get them to run properly after a few hours of trying, I throw them away and won’t touch them again for a few years. Now I have added Alpine Linux to that list. Fortunately, there are enough alternatives out there.

But since I’m not really interested of trying out many different Linux distributions, I’ll just go back to Ubuntu. Yes, it is still a ~600 MB download instead of an 80 MB one but at least it works most of the time.

Another valid option for hosting Docker Containers of course would be CoreOS, but since they refuse to include support for CIFS, I cannot use it for my development environment.

Mounting CIFS (formerly Samba) volumes via fstab on Alpine Linux

For my local development environment, I store all my code on my Windows 10 PC. The web servers and everything else then runs inside Linux Virtual Boxes. The Virtual Box shared folders feature has a lot of problems, so I mount the directories using CIFS, which as formerly called Samba.

Naturally, I wanted the same setup I had for my old Ubuntu boxes in Alpine. The setup was, as usual, not as straightforward as I had hoped. First, I installed the cifs-utils. This may or may not be necessary.

apk add cifs-utils

Then, I simply copied over the entry for /etc/fstab from the Ubuntu Box.

//192.168.178.30/www    /mnt/www    cifs    uid=0,gid=0,user=janhapke,password=*****

In Ubuntu, this has been working fine for years and over several versions. On Alpine, I got an error message.

mount: can't find /mnt/www/ in /etc/fstab

Strangely enough, mounting directly from the command line worked, so CIFS was properly installed and the machines could talk to each other.

It turns out, Alpine parses the fstab a little more strictly than Ubuntu and absolutely needs 6 options per line whereas Ubuntu is OK with just 4. So to fix this, I just had to add the last 2 entries (for dump & pass, respectively).

//192.168.178.30/www    /mnt/www    cifs    uid=0,gid=0,user=janhapke,password=***** 0 0

Great, now I could easily mount the directory on Alpine with a simple mount /mnt/www

Until I rebooted the machine. I always poweroff the VMs when I don’t use them because like shared folders, suspending them doesn’t really work so well in Virtual Box. The problem is, it now tried to mount the folders on boot, before the network was even available. This resulted in a lot of error messages during the boot process.

mount error(101): Network unreachable
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
mount: mounting //192.168.178.30/www on /mnt/www failed: Network unreachable
 * Some local filesystem failed to mount
 * ERROR: localmount failed to start

This was then followed by even more error messages about system services that cannot start because not everything could be mounted. It was possible to log in, but the system was not really properly configured (e.g. keymap ws incorrect).

The fix was again adding the right thing to /etc/fstab, namely appending the option _netdev to the mount options (which again was not necessary on Ubuntu).

//192.168.178.30/www    /mnt/www    cifs    uid=0,gid=0,user=janhapke,password=*****,_netdev 0 0

This option tells Alpine Linux to only mount this directory after the network is available. With this entry, the VM can now finally mount files from the Windows Host machine.

Installing Docker (Daemon) on Alpine Linux

I’m thinking about overhauling the infrastructure behind this site (and others) a bit and want to place as many services as possible into Docker containers that can then run inside a small Virtual Machine on Amazon EC2. So I started playing around with some Virtual Machines locally and since Ubuntu would be too boring, I decided to give Alpine Linux a try. It promises to be really lightweight so that seams ideal as a container host running inside a VM.

After finally figuring out how to install Alpine (which is another story), installing Docker was relatively straightforward based on their Wiki:

  1. Add the Community Repository to the APK Repositories file:
    ~# vim /etc/apk/repositories

    Add the following line:

    http://dl-6.alpinelinux.org/alpine/edge/community

    Or, if you are using a mirror, use the URL of the mirror, in my case:

    http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/edge/community
  2. Update the list of available software:
    ~# apk update
  3. Install Docker:
    ~# apk add docker
  4. Configure docker daemon to start automatically on boot:
    ~# rc-update add docker boot
  5. Start the docker daemon:
    ~# service docker start
  6. Verify it’s running:
    ~# docker ps

    Output should look like this:

    CONTAINER ID    IMAGE    COMMAND    CREATED    STATUS    PORTS    NAMES

Great, now I have a running Docker Daemon. Time to build a simple container. I quickly created a very short Dockerfile and ran docker build. After downloading some layers, it failed with a very cryptic error message:

failed to register layer: ApplyLayer exit status 1 stdout:  stderr: chmod /bin/mount: permission denied

Searching for this message on Google lead me down many rabbit holes, but ultimately it takes only a single command to make it go away:

sysctl -w kernel.grsecurity.chroot_deny_chmod=0

This disables a security feature inside the Kernel, so it might not be safe for a production environment that runs containers but I think it’s acceptable for the machine that merely builds them.