You may be able to point MX records to an IP address, but I do not recommend it. The last time I checked into this topic, I didn't locate any RFC standards that prohibit having an MX record point to an IP address. However, I did note that anti-spam solutions will commonly dislike this. So, unless you want automated systems to be prone to flag your domain as being more questionable, your best bet may be to follow common practice, which is to have MX records point to a domain name that has either an AAAA record, or an A record, or both. (I do not mean to suggest that you should have only one AAAA record or A record. Multiple are fine.)
So then when someone sends an E-Mail to someone@example.com, the client software should look up the MX record and find that mail should be handled by the machine at mail.example.com. Therefore, mail does go through mail.example.com, but that isn't part of the E-Mail address that people need to share. It is the mail.example.com address (which the MX records are pointing at) that will have the AAAA record(s) and/or A record(s) that ultimately get used by mail software.
Whether example.com has any AAAA record(s) and/or A record(s), and what the values of those records might be, have no effect on mail (with the vast majority of E-Mail software which is compliant to the standards of appropriately using MX records.)