File locking in Python

Two threads or processes want to modify the same file at the same time. Here’s a Python solution that covers Mac OSX and Linux.

A lock is a way to tell other threads or processes that a resource (like a file) is in use. Unfortunately, file locks on Unix are advisory only. This means that, by default, processes won’t care if you’ve locked a file. Rule number one of file locking is that you need two processes that want to work together. Thankfully, if you’re the developer, you can force them both to play nice.

You lock and unlock files on Unix using the system call flock. You can see the documentation by running man flock.

Python wraps flock. Here’s how we can use it to lock a file:

import fcntl
x = open('foo', 'w+')
fcntl.flock(x, fcntl.LOCK_EX | fcntl.LOCK_NB)

Unlocking is just as easy:

fcntl.flock(x, fcntl.LOCK_UN)

If a second process tries to get a lock on the file while the first process has the lock set, it will raise an exception, like this:

BlockingIOError: [Errno 35] Resource temporarily unavailable

The file will stay unavailable until it is unlocked (released).

That second process could keep on trying to get a lock forever until the first process unlocks it, or it could exit with an error or go do whatever else it needs to be doing.

Here’s an example of waiting for the lock to be released:

while True:
    try:
        fcntl.flock(x, fcntl.LOCK_EX | fcntl.LOCK_NB)
        break
    except IOError as e:
        # raise on unrelated IOErrors
        if e.errno != errno.EAGAIN:
            raise
        else:
            time.sleep(0.1)

This file locking-unlocking dance has been extracted into a decorator.