@Draug,
The difference between 5th amendment and 1st amendment religious freedom has been, by some arguments, that the 5th amendment reads, "No person shall be held to answer for..." and "nor shall any person be subject for..." etc, etc. It says pretty clearly that these things won't happen in the US.
The argument hinges on the more specific language in the first amendment religion clause that begins, "Congress shall make no law respecting..."
They argue that while the fifth amendment says people cannot be held [by anyone] to answer for certain crimes, the first only specifies that *Congress* shall make no such law, and that the first does not, therefore, prohibit the *States* from making such a law, whereas the fifth does.