The basic reason is that I see no reason to engage in a morally dubious practice when it doesn’t even yield any pragmatic benefits. For example, if I take the wheel of a runaway bus, and I can swerve to kill 3 people instead of the 30 in the path of the bus, then maybe I go ahead and do it. Sure, I feel bad about killing 3 people who otherwise would have lived, but I understandably could “pick the lesser of two evils.”
Yet that’s not what happens if you vote for someone you think will violate property rights and order the deaths of innocent foreigners. Here, your vote is not reducing the number of people killed (unlike in the bus story). Whether or not you vote for the “lesser evil,” the same person will become president and will go about doing a whole bunch of evil. So there is no practical reason for you to join in, and thus you might as well save yourself the inconvenience of waiting in line at the polling booth.
I deal with the obvious retorts here.