So, for instance, a black A on red paper?
Well, it's definitely easier to see the colors on grayscale tones. For letters that look red, being written in black on a red background just makes them look black. For non-red letters, being written in black on a red background doesn't seem to remove their color, but makes it less obvious.
For some characters, the association is stronger. For instance, the letter K is especially green. 3, 4, and 5 are especially orange, brown, and yellow.
a is the same color as A. In fact, α (greek alpha) is also. I'm almost certain that capitals and lowercase letters have the same color in all cases. But in some cases the colors have slightly different shades. For instance, lowercase L (which is blue) looks a lot like uppercase I in most fonts. So lowercase L is a darker shade of blue than uppercase L. Also, for whatever reason, e is considerably oranger than E.