Well, that's complicated -- it depends exactly who feels it has been broken. Congress occasionally could redress such a situation through legislation, for example.
But I'll forego further hairs-splitting and agree with that premise. As you seem to admit, though, that's irrelevant in determining whether the court is _right_. Simply because there is no practical recourse when they get it wrong does not make citing them a good argument against somebody who disagrees.
If you want to make the argument, "The Supreme Court justices are eminent legal scholars, so I find their opinion more convincing than yours," that is one thing; but you were making the argument, "Nobody has power to overturn the Supreme Court, so you can't disagree with them," which is just fallacious.
"I think my point still stands though, that when one random person claims the Feds are breaking the constitution and the majority of judges disagree, it's not unreasonable to side with the judges."
Maybe, but that doesn't really describe the situation. There were dissenting judges in the first place, after all -- some of these decisions were 5-4, and there are plenty of scholarly arguments on both sides as to whether they were or weren't right. The "one random person" might be well-read in those arguments and have an actual reason he believes the court got it wrong.
I almost certainly don't go as far as GF or krellin in thinking the Court has gotten these issues wrong, but I do think they have gotten some very big ones wrong, and there are good arguments for that, some of which have even been laid out carefully in dissents and concurrences. If you don't want to get into all those details, that's absolutely fine -- but don't make a weak appeal to authority in the face of an actual argument.