Once you give the government the power to tax things that it calls "negative externalities," it's got an excuse to tax almost anything, and will probably use it eagerly. Personally, I've noticed a near-perfect correlation between stuff that economists think is bad and stuff that they think has negative externalities.
But my fundamental objection is #2 [see below]. And contrary to Mankiw, you don't have to be a believer in small government to take this position. Anyone who thinks that speech should not be taxed no matter how offensive it is takes this position. Anyone who thinks that extremely unpopular religions should not be taxed takes this position. Indeed, if you think that you should be able to marry whoever you like, no matter how much your parents hate them, you don't really belong in the Pigou Club.--Bryan Caplan
#2 You recognize the externalities but you don’t think the government should try to respond to them. You are such a believer in small government that you are willing to live with inferior economic outcomes, such as pollution and congestion.