I definitely consider myself a Swedish-American. As to African American, it's been my experience that that was merely a way to classify people of the negroid race without being offensive. (Negroid, Mongoloid, and Caucasian were the terms I learned in my 7th grade Social Studies class in 1979, they might be offensive now too).
Growing up in Atlanta, there was the n-word which was clearly offensive and derogatory. Then there were several other words that were clearly meant (at varying times) to refer to the same group without giving offense. These were primarily negro, colored, black, and African-American. It seems right now that referring to people as whites, blacks, hispanics, native americans, and asians works pretty well.
Basically most people will get the idea if you're trying to be offensive or not.