"In your analogy, I'd say you remove colours by adding layers."
You don't remove colors by adding layers. You remove wavelengths (or spectra to be more precise) by adding layers. Neither spectra nor layers are the same thing as colors so you can say whatever you like, but it's still arbitrary.
If someone parks their car in a parking lot, have they added an occupied parking spot or removed a free one? The answer depends entirely on what you happened to be counting, not on what actually happened.
Having been professionally involved in an optics research group in a project that involved calculations concerning colors I can assure you, the choice of how to add colors is similarly arbitrary.
Saying that black is the absence of color is exactly like saying that the West is to the left. Sure, if you're facing North it is, but you might as well have chosen to face South in which case the West would have been to the right.
The thing is, colors aren't numbers. So in order to add them you can't just use addition as we know it from numbers. You have to define what adding colors means and whatever definition you chose is arbitrary. Although a particular application might make one choice more sensible than another, i.e. if you're building a monitor you would chose a system (such as RGB) where black is the absence of colors, while if you're building a printer you would chose a system (such as CMYK) where white is the absence of color.
For discussing skin color, which like printing is determined by absorption of light, a system where white is the absence of color would make the most sense.