Windows and Linux do require an eject, even though they have certain protections. They're just not necessarily as conscientious about user education. (I.e. warning you about it). An unmount / "eject" lets the OS ensure it is not "transferring data" any more.
It may be that your copy application doesn't return until all data has been written to the device. (How much do you trust your understanding of the copy app?) However one example scenario is that your file manager shows image thumbnails, and starts writing a thumbnail cache on the device without you realizing it.
Removing the device during a write will corrupt the filesystem. I don't know how the other OS's handle uncleanly unmounted FAT filesystems on removable devices, but Linux doesn't detect them at all. So the next mount will not trigger a filesystem repair, and the filesystem will remain corrupt.
How likely a corrupt FAT is to lead to data loss over time, I'm not sure. There's likely reasonable attempts to protect against it on well-developed kernels like Windows and Linux, because of how often people do this. (In a similar vein, linux has a mount option which tries to match Windows removable FAT behaviour... it does fsync() on close(), or something like that).
However it's good hygiene not to let your filesystems become corrupted. If nothing else, it means that if you're forced to shutdown uncleanly (e.g. by power failure), then you can scan the filesystem for corruption afterwards, and you'll only be faced with whatever happened recently.
As for the worst case? Your USB device is a very cheap SSD. The expensive SSDs fail hilariously when they lose power, including the loss of all data on the device. Sleep well and tend your backups. https://www.usenix.org/conference/fast13/understanding-robustness-ssds-under-power-fault
Even if you're lucky, you could lose the data on a whole 2Mb erase-block - when you were only writing to a 512 byte file on that EB. With flash storage, the device has to erase that whole 2Mb EB before it can rewrite the data. And the device will remap sectors for a couple of reasons, so the 2Mb of lost data could be from several places anywhere in the filesystem. My reference here would be kernel devlepor Pavel Machek's distress, at realising the linux ext filesystems (even with journalling) wasn't protect his data against this known failure mode. (FAT isn't really any better).
[I'm not a good example here, because I have a laptop with an SD card, I let it run out of battery quite frequently, and it accumulates errors to the point where fsck.vfat is not always able to fix them. OTOH there's nothing on the card that I don't have a copy of elsewhere :). I've had 3 unexpected failures of SD card hardware in the past].