File Descriptors on Linux

Jul 25, 2026 ·9 min read

Sooner or later every one of us meets it: some service that has been running fine for months suddenly starts logging Too many open files, and most of the internet’s advice is to raise some limit to a million and move on. That works, in the same way that turning up the radio fixes a rattling engine. So before reaching for the volume knob, let’s actually look at what a file descriptor is, because it is a Unix concept that manages to be both trivially simple and responsible for a remarkable share of 3 AM incidents.

Just a number

A file descriptor is an integer, and as far as your process is concerned, that integer is the whole data structure.

When a process calls open(), the kernel does all the real work of locating the file, checking permissions, and setting up its internal bookkeeping, and then hands the process back a small number. From that point on, the process refers to the file exclusively by that number: read(3, ...), write(3, ...), close(3). The number is an index into a per-process table that the kernel maintains, and the process never sees anything behind it. Every process starts life with three of them already populated: 0 for stdin, 1 for stdout, 2 for stderr.

All of shell redirection is built on this one property. 2>&1 means “make entry 2 in the table point where entry 1 points”. That is the whole mechanism, and it turns the usual >/dev/null 2>&1 incantation into something you can read off the page.

Because this is Unix, the same table holds a good deal more than files. Pipes, sockets, terminals, devices, they all live in it and answer to the same read() and write(). Linux then ran with the idea to its logical extreme: timers (timerfd), signals (signalfd), process handles (pidfd), event counters (eventfd), even anonymous chunks of memory (memfd) are all file descriptors now. If a kernel developer needs to hand userspace a reference to anything, the answer is a file descriptor.

Descriptors, descriptions, and inodes

That per-process table is a simplification, though. Between the integer and the data on disk there are three layers:

the file descriptor, the per-process integer
the open file description, a kernel object holding the file offset and status flags
the inode, the actual file on disk

The open file description is created by open() and sits between the other two. It can also be shared. When you dup() a descriptor, or when a process forks, you get two descriptors pointing at the same open file description, sharing its offset and flags. If the parent seeks, the child’s position moves too, because there is only one position. Two separate open() calls on the same file, on the other hand, produce two independent descriptions with two independent offsets into the same file.

The distinction shows up in practice quickly. Two processes appending to the same log file after a fork interleave cleanly because they share one offset and O_APPEND is atomic. Two processes that opened the file separately without O_APPEND will overwrite each other. A log file with lines mysteriously chopped in the middle is usually this layer of the diagram showing up in production.

Deleted files that keep their disk space

The sharing model also produces my favorite classic of the genre: the full filesystem with no big files on it.

A file on Linux is deleted when two conditions are met: no directory entry points to it, and no process holds it open. rm only handles the first part. If a process still has the file open, the inode and every block of its data stay where they are, invisible to ls and du but very much visible to df.

The usual version of this incident is a runaway log file. Something logs gigabytes, the disk fills, someone deletes the log, and the disk stays full, because the service never reopened its log file and is happily writing into a file that no longer has a name. du claims the disk should be half empty, df insists it is full, and the two will keep disagreeing until the process lets go.

The debugging move is one command:

lsof +L1

which lists open files with zero remaining links, meaning deleted-but-held-open. The fix is to make the process let go, either by restarting it or, more politely, by sending whatever signal makes it reopen its logs. This is also the entire reason log rotation tools have copytruncate and postrotate mechanisms: rotating a log out from under a process that keeps its descriptor is this same problem, scheduled daily.

As a bonus party trick, as long as the process is alive, the “deleted” file is fully readable through /proc/PID/fd/N. I have recovered a config file this way more than once, from a process that was still running with the only remaining copy open. It is a good reason to think twice before restarting something in a panic.

Watching progress from the outside

That /proc/PID/fd/ directory has a sibling I reach for just as often: /proc/PID/fdinfo/. For every descriptor a process holds, there’s a file in there showing the kernel’s bookkeeping for it, and the first line is the one we care about:

$ cat /proc/12345/fdinfo/3
pos:    2147483648
flags:  0100000
mnt_id: 29

pos is the current file offset of that descriptor, live, updated as the process works. The offset is all you need to calculate progress.

Say you started a cp of some enormous file and, as is tradition, it prints nothing while you sit there wondering if it’s working or wedged. The answer is sitting in /proc:

$ ls -l /proc/$(pgrep -x cp)/fd        # find the source file's fd, say 3
$ cat /proc/$(pgrep -x cp)/fdinfo/3    # pos: how far in it is
$ stat -c %s /path/to/source           # size: how far it has to go

Divide one by the other and you have a percentage. Sample it twice a few seconds apart and you have a throughput rate and an ETA. You get all of that from the outside, without the copying process knowing or cooperating, using two files in /proc and some arithmetic.

This works on anything that chews through a file sequentially: cp, tar, gzip, dd, md5sum, a database import, that mystery process that has been “almost done” for an hour. There’s a small tool called progress that packages this trick, scanning for known commands and printing live percentages, and it is well worth having in the toolbox. But the mechanism is so simple that a shell one-liner in a watch gets you most of the way, which is what I do on machines where installing things isn’t an option.

The kernel already tracks the offset and /proc already publishes it, which saved anyone the trouble of adding a progress API to forty different tools.

Too many open files

Which brings us back to Too many open files, or as the kernel calls it, EMFILE.

Every process has a limit on how many descriptors it can hold, the nofile ulimit. Historically this defaulted to 1024, a number chosen in an era when a thousand open files was an extravagance. Modern systemd raised the ceiling considerably, but the soft limit often still sits at 1024 for compatibility reasons, mostly because the ancient select() syscall physically cannot handle descriptors above 1023 and nobody wants to find out the hard way which programs still use it.

When a process hits the limit, raising it should not be the first move. First ask whether the process is supposed to have that many descriptors open. A busy proxy legitimately holding fifty thousand sockets is one thing. A small web app that has crept from 200 descriptors to 900 over two weeks is leaking, and raising its limit just reschedules the incident for next month.

The inspection tooling is nothing fancy:

ls /proc/PID/fd | wc -l         # how many
ls -l /proc/PID/fd              # what exactly
cat /proc/sys/fs/file-nr        # system-wide count

That second command tells you what the descriptors actually are: files, sockets with their connection info, pipes, or the anonymous kinds. A leak usually has a signature. Nine hundred sockets in CLOSE_WAIT means the application isn’t closing connections the peer already hung up on. Hundreds of descriptors pointing at the same file mean someone is opening it in a loop and forgetting the close().

When the limit does need raising, remember that for anything under systemd, ulimit in a shell and /etc/security/limits.conf are both the wrong place. Services get their limits from LimitNOFILE= in the unit. Plenty of hours have been lost to editing limits.conf and wondering why nothing changed.

Inheritance

One more property rounds out the picture: file descriptors are inherited. fork() copies the table, and exec() preserves it. That is how the shell wires up pipelines, how your program has stdout at all, and how a descriptor can be handed to a child process on purpose.

It is also how descriptors get handed to child processes by accident. A daemon that spawns a helper passes along everything it forgot to close, which is how a listening socket ends up held open by some long-lived grandchild, and why the “restarted the service but the port is still in use” mystery usually ends with lsof -i :PORT pointing at a process you did not expect. The fix has been around for ages, the O_CLOEXEC flag, which marks a descriptor to be closed automatically on exec. Modern runtimes set it by default on everything they open. The bugs live in the code that predates that habit, which, this being our industry, is most of it.

For completeness I’ll mention that processes can also deliberately send descriptors to each other over Unix sockets, SCM_RIGHTS, which is how systemd hands a pre-opened listening socket to a service. The kernel duplicates the entry into the receiving process’s table, and suddenly “just a number” is a number that traveled between processes.

That is the whole picture: an integer, a table, and three layers of indirection. Most of the weird behavior on a Linux box, from truncated logs to full disks with no files to ports that won’t free up, is one of those layers doing what it was designed to do.

Processes leak, hold on to disk space and squat on ports; the descriptor just takes the blame for it.