EDIT: I recall vaguely something about it being a problem with Linux guest VMs because it aggressively uses RAM as a file system cache. So it'll just fill up RAM to the max when reading from disk, and overwrite this if it's later needed by applications.
Maybe memory compression helps, but it doesn’t appear so to me right now.