Work Around Docker Devicemapper Busy Filesystem

Posted on Jul 18 2017 in Ops

If you are using Docker on Red Hat Enterprise Linux 7 (or one of its derivatives, such as CentOS 7), you may run into an error when you attempt to remove a container:

# docker rm my-container
Error: Error response from daemon: Driver devicemapper failed to remove root filesystem 4f872301a939db6eb8dd14d296bc152dbb97774e0da77e0b4585c06f7d4f5f1e: Device is Busy

I hit this and I went googling, to discover this GitHub issue, where people are working on a fix. The problem appears to be some sort of file or mount leakage out of docker into other processes running on the host. A little alarming!

It seems there is a temporary fix (though with some unclear impact). You can configure the docker daemon to run with an alternate mount configuration by placing MountFlags=slave in /etc/systemd/system/docker.service. It also sounds like they are trying to have this fixed in RHEL7.4 and/or the next Docker release. We'll see!

However, if you don't want to do that, there is a handy workaround:

Step 1: Find file hash

Inspect the container that refuses to be removed:

docker inspect 4f872301a

Look for the section that mentions the GraphDriver, which is likely to look something like this:

"GraphDriver": {
    "Name": "devicemapper",
    "Data": {
        "DeviceId": "23",
        "DeviceName": "docker-9:127-303967-c32dd5e9af3c0c15dc8c6ec476759a701619186845fde7638d0ffcfd93fb05ad",
        "DeviceSize": "10737418240"
    }
}

You are interested in the long hash at the end of the DeviceName attribute. In my example, that's c32dd5e9af3c0c15dc8c6ec476759a701619186845fde7638d0ffcfd93fb05ad.

Step 2: Identify Processes Hanging On

You need to find the processes that think they have that file open.

find /proc/*/mounts | xargs grep -E "c32dd5e9af3c0c15dc8c6ec476759a701619186845fde7638d0ffcfd93fb05ad"

Hopefully, you'll see a few lines like this:

/proc/23007/mounts:/dev/mapper/docker-9:127-303967-c32dd5e9af3c0c15dc8c6ec476759a701619186845fde7638d0ffcfd93fb05ad /var/lib/docker/devicemapper/mnt/c32dd5e9af3c0c15dc8c6ec476759a701619186845fde7638d0ffcfd93fb05ad xfs rw,seclabel,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota 0 0
/proc/23008/mounts:/dev/mapper/docker-9:127-303967-c32dd5e9af3c0c15dc8c6ec476759a701619186845fde7638d0ffcfd93fb05ad /var/lib/docker/devicemapper/mnt/c32dd5e9af3c0c15dc8c6ec476759a701619186845fde7638d0ffcfd93fb05ad xfs rw,seclabel,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota 0 0
/proc/23009/mounts:/dev/mapper/docker-9:127-303967-c32dd5e9af3c0c15dc8c6ec476759a701619186845fde7638d0ffcfd93fb05ad /var/lib/docker/devicemapper/mnt/c32dd5e9af3c0c15dc8c6ec476759a701619186845fde7638d0ffcfd93fb05ad xfs rw,seclabel,relatime,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota 0 0

Step 3: Restart or Kill the Processes

For each of the PIDs you find (the PID is just after /proc/ in the output), use ps aux | grep <PID> to figure out what's running at that PID, then restart or kill the processes as you deem appropriate. In my case, I had to restart a few random processes that had nothing to do with docker or the container that was refusing to be removed.

After that, another docker rm should do the trick.

Happy troubleshooting!