One missed opportunity in Black's algorithm is that it currently treats the maximum line length as a literal hard limitation in number of characters. Here is a trivialized example:
to_add = [item for item in data.new_items if item not in data.old_items]
to_remove = [
item for item in data.old_items if item not in data.new_items
]
Although the constructs are nearly structurally identical, they can be formatted very differently, which sometimes hinders understanding them.
A different approach would be to instead normalize all words to a certain fixed width. So, "to_add" and "to_remove" would have the same virtual width.
A related issue is that leading indentation counts towards the width limit. This causes refactorings which simply move code around (changing its indentation level) to change the code's shape, even when the code hasn't otherwise changed. This is exacerbated by that one often needs to mold code in such a way that Black formats it in an agreeable way, but this is generally not done during refactorings, so the readability of the code suffers.
I had the opportunity to write a formatter (for SQL, also unconfigurable/opinionated); it seems to successfully avoid these problems: https://github.com/CyberShadow/squelch
Line length is there for a reason, it's to fit everything on the screen. Ignoring leading spaces/indentation or giving a 'fudge factor' doesn't help keep everything on the screen.
Ultimately there has to be a hard limit and it's silly to argue over special cases that should exceed it (because that's just another thing to bikeshed over in code reviews). Black sets a hard limit and enforces it--done, no more discussion.
IMHO it's a code smell to have a bunch of long lines of code that are all visually similar but vary in a tiny and easy to miss way. Here's another way to think about the code you wrote that boils it down to an even smaller and more focused intent:
new = set(data.new_items)
old = set(data.old_items)
to_add = new - old
to_remove = old - new
It's not exactly the same as what you wrote but you get the idea, and it can be made simpler if you're using set types to start with. It's kind of a nudge that if your intent is to do set-like operations like difference, intersection, etc. then you might want to use the right tools for the job instead of banging out more procedural code.
To reiterate, the example is trivialized; there isn't always an elegant solution that you just can't see because you didn't think hard enough.
Regarding the hard line width, "fit everything on the screen" is a poor goal to aim for; a more useful goal is to make the best use of the two dimensions you have at your disposal. Squishing things across either axis will make for a poorer experience than occasionally requiring some scrolling for some setups.
What I like to do is configure black for a maximum line length of 100 characters (although I prefer keeping docstrings at a maximum of 72 characters). That's not as restrictive as black's default of 72 characters, but still compact enough to be readable and to fit at least two files next to each other.
If a line is longer than that it's most of the time either something which should be wrapped into multiple lines anyway (like a dictionary with multiple keys) or a sign for code smell.
One of the most common code smells I see leading to longer than necessary lines (and which might make the code harder than necessary to read) is not using early returns.
Also using a trivialized example:
to_add = []
for item in data.new_items:
if item not in data.old_items:
# code spanning multiple lines with further indentation levels
…
to_add = []
for item in data.new_items:
if item in data.old_items:
continue
# code spanning multiple lines with further indentation levels
…
A different approach would be to instead normalize all words to a certain fixed width. So, "to_add" and "to_remove" would have the same virtual width.
A related issue is that leading indentation counts towards the width limit. This causes refactorings which simply move code around (changing its indentation level) to change the code's shape, even when the code hasn't otherwise changed. This is exacerbated by that one often needs to mold code in such a way that Black formats it in an agreeable way, but this is generally not done during refactorings, so the readability of the code suffers.
I had the opportunity to write a formatter (for SQL, also unconfigurable/opinionated); it seems to successfully avoid these problems: https://github.com/CyberShadow/squelch