Some ways,
- Signature based detection.
A good and updated anti-virus suite (yes, I know 'good' will be debated)
will help track most of the malware before it starts engaging with your system - Anomaly based detection.
A track of outbound communication from individual applications
(this is also done by most AV/AS software)
will help identify unexpected 'mothership-calls' from applications.
Note that I do not mean analysis of communication. I mean attempts of communication be applications that are not expected to do that (say editor applications for example). Analysis of communication (say from a chat application you downloaded) might also be done, but would be quite a complex problem.
I'll quote a personal example of a good malware detection case.
One of the standard AV/AS suites on a Windows machine of mine was active when,
I tried to open a 'sample' (and malware scripted) HTML file from one of our work servers.
It was immediately caught by the suite.
Then, I tried a Cygwin scp
fetch of the same HTML file now renamed as TXT on the server.
The suite did not let the scp
land on my host disk. It was deleted as soon as it was fetched.
The detection was based on recently updated signatures for a new 'script-based-attack'.