Alderian, there's a certain, I don't know exactly how to phrase it - moral hazard, I guess, in taxing people so that the already enormously wealthy and powerful can become more wealthy and powerful. Don't you see a certain perversity there? Even if your economic argument was correct, which I don't believe it is, I would reject subsidies on the grounds of immorality. If the government wants gas prices to be lower for poor people, which it shouldn't, then it should either pay poor people a stipend for their gasoline needs or it should nationalize the oil industry.
But your argument, I think, doesn't hold water. For most people, gasoline costs are elastic -- there is a certain minimum amount you have to purchase, but beyond that you can reduce your costs. Gasoline prices can only rise so much before people won't buy it. Then, the oil industry can't afford to pay its CEOs those kinds of salaries anymore. Eliminate the indirect subsidies like zoning laws that encouragement car-friendly roads, and the margin decreases even further.
The benefit to the common man, whoever he might be, is obvious: our government is in enormous debt. They're spending trillions of dollars they don't have, and somebody is going to have to pay for that.
Also, when you say that not everyone gets taxed, it's true that not everyone gets income taxed, but virtually everyone pays some sort of taxes, either directly (property, sales, etc) or indirectly (corporate taxes or tariffs that raise prices, for example).