Oh joy. While writing the question (and coming to the directory history part) I stumbled upon the solution myself. Far provides a folder history as well (Alt+F12), so doing a quick Alt+F12, ↵ goes back to the previous folder.
I have now bound this to Ctrl+⇧+\.