Disk full but free space left on linux server — inode limit

Servers with many small files

We are running some web servers at the institute which serve tons of files. The files are images of protein structures, and many meta data files for the images. The meta data is very small, and many of the files contain less than 500 bytes of data.

…may lead to hard disk problems due to exessive inode usage

An automated update script was scheduled to add more data over the last weekend, but all I got on Monday morning was tens of thousands of error messages from the rsync process that should have copied the source data:


rsync: recv_generator: mkdir "/srv/www/PDB/zz/3zzu" failed: No space left on device (28)
*** Skipping any contents from this failed directory ***
...

A quick look with df claims that there was still space left on the device though!

fileserver13:~ # df -h /srv/
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 589G 451G 134G 78% /srv

But the problem is the sheer number of files, as you can see from looking at the inodes instead of disk space:

dhcp210:~ # df -i /srv/
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda3 39299904 39299904 0 100% /srv


So what is an inode?

On Unix file systems, an inode exists for each file (file in the broader Linux sense, so also directories) and stores its meta data, e.g., user permissions, last accessed date, etc.


2 possible solutions: delete useless files or get more inodes

So we need more inodes. There are some posts on the internet which suggest you delete small, useless files to free up inodes and give advice on how to find candidate files like temporary data (here or here). In my case, the millions of small files are needed though, so this is not gonna help.


Getting more inodes on a partition requires reformating it

It turned out that the number of inodes a file system has is determined at file system creation time (when you format the partition with a file system using mkfs). It is computed based on the partition size, and the interesting parameter is the inode_ratio. You can set it when creating the file system, and it is easiest to choose one of those ratios/setups pre-configured in /etc/mke2fs.conf. On our SuSE 12.3 server, the profile small results in the following settings:

small = {
blocksize = 1024
inode_size = 128
inode_ratio = 4096
}

This means it is optimized for many small files on the disk: smaller block size (to waste less space with small files) and more inodes.

So I grabbed a new disk, partitioned it with the profile and used that disk for the data.

The complete partition command will be given below, but. WARNING: Creating a file system on a disk DELETES ALL DATA which may be on that disk! Be 100% sure you choose the correct device! Backup your data!

To delete all data on the partition and re-create it with the EXT4 files system and new inode settings small, use this command:

mkfs.ext4 -T small /dev/yourpartitionhere

You need to run this as root. And it will delete everything on /dev/yourpartitionhere!

For my 750GB partition, this resulted in 183,144,448 inodes, which is enough for me. (With the default setting, the 500GB disk had 39,299,904 inodes, which was NOT enough.)

Advertisements

About dfspspirit

PhD student in bioinformatics, interested in photography, level design, digital image manipulation, architecture and, of course, bioinformatics.
This entry was posted in bash, IT and computers, linux, supercomputing, troubleshooting and tagged , , , , , , . Bookmark the permalink.

One Response to Disk full but free space left on linux server — inode limit

  1. Jojo Addison says:

    Thank you!!! You saved me 🙂

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s