Finally found the problem, RealVNC.
I had to install it for some reason a few months ago to get the viewer. Apparently it doesn't ask or anything if you want to also install the server, so it did. The (non-working) RealVNC server hence listened on all the ports instead of TightVNC or UltraVNC.
Disabled the RealVNC server service and fixed all my problems