✏️ safer 2: a better safer file writer ✏️

Part 1.5 of the Coroniad

4 min readApr 23, 2020

Introduction

This is a follow-up to this article about the safer Python writer from this series of articles aimed at any Python reader past the beginner level.

safer is a tiny, single file Python library for safely writing to files where either the entire file is written, or nothing is changed.

There was a lively discussion on Reddit which found a race condition (fixed here), alerted me to the fact that you can use open() with an integer value representing a file descriptor (which I disallow here), and to the fact I wasn’t handling file modes properly (fixed here).

What changed everything was this innocuous comment:

Is there a way to recode this so it’s just an additional flag passed to the standard steam open() instead of a whole new object?

I had already experimented with that earlier in development but I had abandoned it as “too hard”. But reading this comment, I thought, “Something like that would be so much better, maybe it’s worth the time.”

And thus… ✏️ safer 2: a drop-in replacement for open()! (and also backward compatible with oldsafer ).

New safer.open()

You can drop safer.open() right in in place of open() in Python 2 or 3:

# dangerous
with open(filename, 'w') as fp:
    json.dump(data, fp)  # File might be partially written# safer
with safer.open(filename, 'w') as fp:
    json.dump(data, fp)  # All of the file is written, or none

safer.printer() is still there and useful.

safer.writer() is deprecated in favor of safer.open(mode='w') (though still there for backwards compatibility).

What makes this new version interesting?

It was surprisingly tricky

I had all the machinery written, so why was delivering it in a different way non-trivial?

Calling open returns one of five different classes depending on whether it’s Python 2 or 3, and depending on the mode argument.

safer needed to add some behavior on __exit__() and close() and unfortunately the only way to do this is inheritance — which means five new classes.

There’s also the issue that open() itself has a different signature on Python 2 and Python 3.

Dynamically constructing classes — use type()?

I didn’t want to write five derived classes, even if I used a mix-in class to avoid duplication, because the logic involved in selecting which classes to use would have been incomprehensible — I sketched it out and quailed.

So I decided to dynamically create the classes using type(). It came out quite neatly.

One of the few unfortunate design decisions of Python is that the built-in functiontype() is in fact two unrelated functions — one-argument type(object) that returns an object’s type, and three-argument type(name, bases, dict) which creates brand-new classes!

I had never used three-argument type() before, and I came up with an amusing technique to construct the dictionary of class members from the locals() in this little function.

There was a bit of a panic when I realized that I was creating a brand-new class for each open()! Surely that’s not cheap.

I quickly dropped an lru_cache on it, discovered it didn’t work on Python 2, and then because my blood was aroused, wrote my own cheap version. It all worked

Dynamically constructing classes inline!

And then the next day , I came back and threw away the locals() trick, and the use of type() and used a new idea: an inline class. (I also dumped the lru_cache in favor of a simple dict.)

I didn’t know that dynamically defining a class to inherit from a variable parent would work — but it did and in Python 2 and 3. Points for Python!

Dynamically selecting an implementation at runtime

A different version of open() are selected at runtime here depending on whether we’re using Python 2 or 3.

Both versions eventually delegate to the same actual implementation function, _open(), which is defined here, but there is a a little delicate work validating parameters here in order to entirely replicate what built-in open() would do.

Thanks for reading!

If you want to read more:

Click on Watch in the Coroniad repository (which only gets this series)
Or follow me on Medium (which gets all my posts)