Well, this is just my off-the-cuff response, but let’s think how a free market society might deal with the problem of a sociopathic serial killer without prisons or the death penalty.
Since punishment would amount to rectification of injustice, rather than inflicting further wrongs as a punishment for past wrongs, the most people could do in terms of prevention is to pretty much refuse to associate with someone who was known to be a violent murderer. Now keep several things in mind: in a totally free society, there would not be any public property. This means there’s nowhere he could stay (not even the streets) without having to acquire the property first, or getting the property owner’s permission to stay. If he had acquired a reputation for being a vicious killer, any large transaction, especially something that required a formal contract such as buying a house or renting and apartment, would probably become almost impossible, as no one will want to accommodate, or even get physically close to, this person. As is also discussed in the Tannehill’s book, one more thing to realize is that the “underground” world of society would be much smaller and more impotent than in ours today. All the organized crime organizations depend on black market sales in things like drugs, prostitution, and gambling. These three things would be legalized in a free society, so anyone wishing to sell them would have to go into business legitimately. I doubt any large-scale crime network for selling these things would remain if there were above-ground alternatives, since the latter would be subject to much more consumer scrutiny. With no way for crime networks to have funding, they would fall apart. What this means is that the murderer would likely have little hope for permanent housing. Consider too that it would be much harder for them to move in secret. Once someone gained a reputation for killing, that would travel with them much more strongly than we can probably imagine today. It would be advantageous for insurance companies and defense agencies to work together to closely monitor such a person. If they tried to enter a new town, it’s possible that notice of this person’s arrival precedes them into town. The owners of the private roads and sidewalks could have preemptive policies that they do not want to let anyone who has such a reputation pass on their roads, and police could be ready to stop the person on the outside of town. Without any way to travel in, how could someone like this even enter the town?
So a serial murder would risk essentially the complete ostracism of society. It would be much harder living underground today. Daily life would become a huge problem without the cooperation of other people.
Lastly, I would say, justice in a free society is administered on a case-by-case basis, and in the long run the rules conform to what the majority of society would want. So in the case of a third man who kills a known serial murderer in retaliation, several things would happen. First off, rather than the third man being rounded up by the cops and put on trial for homicide just like anyone else (which could itself carry the death penalty in many places in our country), the murderer’s family or friends would have to decide to prosecute this case against whoever killed the murderer, which is not a given. It’s not to say that the family would never do anything at all, but I think the remorse and guilt would change their behavior toward this third man. Second, I suspect courts would generally go much easier on such a vigilante killer. It is likely that the judges and arbitrators would not practice the belief that two wrongs do make a right, so retaliatory crimes in general would be punishable just as regular crimes are (consider how you would want the legal system to react to a man who got back at someone else who reneged on a contract by say burning their car or vandalizing their property–you would want this behavior discouraged). However, in the case of someone who killed a serial murderer out of vengeance, I have to imagine that people’s inborn feelings in this case would override much of the punishment. Very few people if any at all would feel sympathy towards the dead serial killer or their family. So the debt amount agreed upon would likely be a lot lower, considering that the first person has such a loss which is not being repaid. Even if a court awarded a huge repayment to the serial killer’s family, the family would still have to find an agent willing to attempt to collect this debt. Unlike our justice and punishment system, every person in the chain overseeing this process is responsible for their own actions. A judge would always be answerable in the future to how they acted in this case. The debt collector would have to explain why he agreed that the third man owed such a large debt to the serial killer’s family. No one could hide behind “the system,” and not have to worry about consequences for their own decisions ever affecting them.
Please read the Tannehill’s chapter for more discussion of this.