Docker / device mapper - out of space issue - Appsembler

Help & How-to

Your comprehensive knowledge base for Open edX, Tahoe LMS, and Virtual Labs. Dive into in-depth documentation, tutorials, troubleshooting guides, and best practices to help with your integration of Appsembler's products.

Docker / device mapper – out of space issue

Docker&devicemapper initialize a 100GB sparse file (`/var/lib/docker/devicemapper/devicemapper/data`) to store images and containers. To check the available space inside that file you can run the following commands:
  • `du -h /var/lib/docker/devicemapper/devicemapper/data`
  • `ll -s /var/lib/docker/devicemapper/devicemapper/data`
  • `docker info` -> Data Space Used/Data Space Total
Because there is a possibility that under certain conditions the space used by deleted images/containers is not reclaimed[1], we might need to enlarge the data file when it’s full.
Steps:
  1. Make sure there’s enough free disk space to:
    1. create backups of device mapper files in step 6 below
    2. enlarge the data file in step 7 below (the additional space will initially be unallocated/sparse, but we still need to make sure there’s enough space to actually use that space)
  2. Check which files are used by docker/devicemapper: `docker info` -> Data file, Metadata file, Data loop file, Metadata loop file
  3. `cd /var/lib/docker/devicemapper/devicemapper`
  4. Check that the mappings are set up: `losetup –all`, `losetup –associated data`, `losetup –associated metadata`
  5. Stop the docker daemon (`service stop docker`) and make sure that the mappings are gone (re-run the commands from step 2 above)
  6. Backup the files: `cp data data-backup`, `cp metadata metadata-backup`
  7. Enlarge the data file: `truncate –size=+50G data`
  8. Ensure that the operation has been successful: `du -h data`, `ll -s data`
  9. `service docker start`
[1] Related issues/articles: