I think that is true. To be religious, you must worship something. Typically people worship things with greater power than themselves, and if that does not manifest itself in physical form, then it does so in moral form, meaning that what they worship is the source of morality for them.
The non religious defy the idea that there is something more powerful than humans that should be worshipped, and thus morality is ours to choose.
Essentially, it is absolute monarchy v absolute anarchy.