File locking in Python

Fri 06 March, 2015

Two threads or processes want to modify the same file at the same time. Here’s a Python solution that covers Mac OSX and Linux.

A lock is a way to tell other threads or processes that a resource (like a file) is in use. Unfortunately, file locks on Unix are advisory only. This means that, by default, processes won’t care if you’ve locked a file. Rule number one of file locking is that you need two processes that want to work together. Thankfully, if you’re the developer, you can force them both to play nice.

You lock and unlock files on Unix using the system call flock. You can see the documentation by running man flock.

Python wraps flock. Here’s how we can use it to lock a file:

import fcntl
x = open('foo', 'w+')
fcntl.flock(x, fcntl.LOCK_EX | fcntl.LOCK_NB)

Unlocking is just as easy:

fcntl.flock(x, fcntl.LOCK_UN)

If a second process tries to get a lock on the file while the first process has the lock set, it will raise an exception, like this:

BlockingIOError: [Errno 35] Resource temporarily unavailable

The file will stay unavailable until it is unlocked (released).

That second process could keep on trying to get a lock forever until the first process unlocks it, or it could exit with an error or go do whatever else it needs to be doing.

Here’s an example of waiting for the lock to be released:

while True:
        fcntl.flock(x, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except IOError as e:
        # raise on unrelated IOErrors
        if e.errno != errno.EAGAIN:

This file locking-unlocking dance has been extracted into a decorator.

Cheating your way to a cross-platform solution

Instead of using a operating system feature, programs can negotiate their own system for signalling when a file is in use. They use lock (sometimes called shadow) files. Vim does this. So does Emacs.

They check for the existence of a lock file before you open the file for writing, if it doesn’t exist, create it, if it does exist, wait for it to be deleted. There’s plenty of Python modules for doing this on Pypi.