Alpine Linux sucks for hosting Docker Containers

After setting up Alpine Linux to run docker containers and fixing a whole bunch of issues I encountered, I now came to the point where I tried to run a node.js app inside a Docker Container that runs on an Alpine Linux Host. But node immediately exited with a Segmentation Fault. After some experimenting, I went and just started the official node docker container in the latest version and started node inside. It still didn’t work.

~# docker run -it node:latest /bin/bash
[email protected]:/# node
Segmentation fault

The kernel logs provide only very limited insights:

kern.info kernel: [   35.465909] device vethe7ba7a4 entered promiscuous mode
kern.info kernel: [   35.466145] IPv6: ADDRCONF(NETDEV_UP): vethe7ba7a4: link is not ready
kern.info kernel: [   35.466151] docker0: port 1(vethe7ba7a4) entered forwarding state
kern.info kernel: [   35.466160] docker0: port 1(vethe7ba7a4) entered forwarding state
kern.info kernel: [   35.467156] docker0: port 1(vethe7ba7a4) entered disabled state
kern.warn kernel: [   35.492075] cgroup: docker (1684) created nested cgroup for controller "memory" which has incomplete hierarchy support. Nested cgroups may change behavior in the future.
kern.warn kernel: [   35.492078] cgroup: "memory" requires setting use_hierarchy to 1 on the root
kern.info kernel: [   35.527037] eth0: renamed from veth1db87ca
kern.info kernel: [   35.538820] IPv6: ADDRCONF(NETDEV_CHANGE): vethe7ba7a4: link becomes ready
kern.info kernel: [   35.538838] docker0: port 1(vethe7ba7a4) entered forwarding state
kern.info kernel: [   35.538843] docker0: port 1(vethe7ba7a4) entered forwarding state
kern.info kernel: [   35.538861] IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes ready
kern.info kernel: [   36.539627] node[1930]: segfault at 10 ip 0000000000a6444d sp 0000750c38bd8550 error 4 in node[400000+13c7000]
kern.alert kernel: [   36.539643] grsec: Segmentation fault occurred at 0000000000000010 in /usr/local/bin/node[node:1930] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:1919] uid/euid:0/0 gid/egid:0/0
kern.alert kernel: [   36.539781] grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/local/bin/node[node:1930] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:1919] uid/euid:0/0 gid/egid:0/0

First, some network config changes done by docker, then docker complains about missing support for nested cgroups, then some more network config changes before node crashes with a segmentation fault. And eventually some error messages because GRSEC prevents node from writing a core dump.

After a lot of searching, there seems to be a whole list of problems in Alpine Linux that prevent it from reliably running Docker Containers, among them GRSEC, problems with LXC networking and missing cgroups hierarchy support. So basically all of the things that show up in the Kernel logs around the Segmentation fault.

By the way, according to the Alpine Bug Tracker, support for nested cgroups was added to Alpine 3.2 – but although I am running 3.3.1, it still isn’t working.

Interestingly enough, in the few days my post about installing Docker Daemon on Alpine Linux has been online, it has already helped somebody else fix the same problem but also lead him to the conclusion that Alpine Linux is not so great as a host for Docker containers.

As with all Linux distributions, if I can’t get them to run properly after a few hours of trying, I throw them away and won’t touch them again for a few years. Now I have added Alpine Linux to that list. Fortunately, there are enough alternatives out there.

But since I’m not really interested of trying out many different Linux distributions, I’ll just go back to Ubuntu. Yes, it is still a ~600 MB download instead of an 80 MB one but at least it works most of the time.

Another valid option for hosting Docker Containers of course would be CoreOS, but since they refuse to include support for CIFS, I cannot use it for my development environment.

10 thoughts on “Alpine Linux sucks for hosting Docker Containers

  1. Pingback: What OS for Docker host? (Part 3) – From the disheveled desk of Nathan Bak

  2. Jason

    FYI this is happening because Alpine doesn’t use GlibC but rather uses Musl so a lot of things need recompiling – switching to a glibc based Alpine will do the trick. Alpine does have a few quirks for new comers which can make it frustrating

    Reply
  3. Sumit Khanna

    I’m about to use Alpine Linux for a router, so I found this post looking for problems with Alpine before I dove in. I’m glad the author is talking specifically about Docker containers.

    So this isn’t really a problem with Alpine more as it exposes some interesting things about Docker. Docker doesn’t run naively on Mac/Win and probably never will. Even with the newest Docker releases that use much better OS-based hypervisors (instead of the old VirtualBox based Docker Machine); they are still running under a hypervisor. I’d even argue that Docker doesn’t really work on Linux. It works on a really customized Linux kernel. For most people this doesn’t matter because installing Docker also installs all the correct kernel modules for it. If you’re running Gentoo/Funtoo, you typically have to go into your Kernel config and manually configure a lot of docker specific options (FYI: I run Gentoo and ran into this). Interestingly enough, Docker does seem to work naively on FreeBSD (although I haven’t tested it to see how/how well it works).

    If you’re going to run Docker dev environments, I’d suggest a distro where it’s prepackaged with all the kernel dependencies. If you’re doing a larger deploy, I’d suggest looking into stuff like Kubernetes, DC/OS (Mesos) or other management layers for running large mappings of containers.

    Good post. 🙂

    Reply
  4. Natanael Copa

    The problem you are having with node here has nothing to do with musl libc. It is the hardened kernel (PaX) that has a security feature that will prevent applications to allocate memory that is both writeable and executable at the same time (W^X). This improves security since it makes it much harder to inject code to be executed with a security vulnerability. However, this also breaks anything that uses JIT (just-in-time) compilers. Node, java, luajit are examples that will break. In Alpine we set attributes on the affected binaries that disables this security feature, which is why those works in Alpine. The node:latest image has not those attributes set so kernel will deny node to do its JIT thing.

    There are multiple ways to work around this:
    – you can use the linux-vanilla kernel package, which is a normal, non-hardened kernel
    – you can disable the PaX feature systemwide with: echo 1 > /proc/sys/kernel/pax/softmode
    – you can set the file xattr in your node container to disable mprotect:
    FROM node:latest
    RUN apt-get update && apt-get install attr && attr -s pax.flags -V m /usr/local/bin/node

    I think none of those are optimal. I would like to find a solution where you can disable PaX features per container. But we are not there yet.

    Reply
  5. Benoit

    It’s not alpine who sucks it docker that uses kernel extension that are not allowed by grsec kernel, just run the vanilla kernel install if you’re planning to use docker !

    Reply
  6. Avery Freeman

    Alpine can be good for some things, but its documentation is lacking – things are missing, etc. I just tried to setup LXD/LXC and got a bunch of errors working through their official wiki trying to make a debian container. I apk-added all the lxc packages available and it still didn’t work. Then I noticed debootstrap was complaining about not having perl, so I installed that and it worked. We’ll see how it holds up.

    Honestly, I got Alpine working pretty good as a hypervisor with qemu-kvm, libvirt, and ZFS, so that’s saying something. Don’t try and install kimchi, though … (shudder)

    Reply
  7. Benoit Beausejour

    I hit on the same issue you had in this post and debugged this a bit more. The problem in my case had nothing to do with musl and everything to do with grsecurity and pax.
    Node, specifically, will load data in memory pages to execute (JIT), this is prevent / stopped by pax.

    For now my workaround (which is not ideal) is to set pax on the alpine container host in softmode, which will prevent segfaults in the containers. Another fix is to install paxctl in your container images and whitelist the ELF themselves (paxctl -cm /usr/bin/node). I found this too cumbersome and impractical in my case since my alpine docker hosts are used for development only.

    Here are the grsecurity and pax flags I set on my docker hosts:

    kernel.grsecurity.chroot_deny_mknod=0
    kernel.grsecurity.chroot_deny_chmod=0
    kernel.pax.softmode=1

    Reply

Leave a Reply

Your email address will not be published.