Before I married I dated a few guys from my group of friends from university. This group included a great variety of people in terms of health, wealth, race, looks, etc. The only common point was that we were all similar age.
As I was lucky enough to be popular, I was choosy. I never wanted to date a smoker, because I find this a filthy, disgusting habit: I never wanted to date a fat man because I felt physically repelled; and I never wanted to date a black man because I find them physically unattractive.
I liked all of my friends as friends, but when it comes to physical attraction I just cannot bring myself to something more physical with a black guy.
Nobody would call me "smokeist" for not wanting to date a smoker, or call me "weightist" for rejecting a fat man; so why should I be labelled "racist" for finding black people unattractive?
Our views of what is esthetically pleasing are personal and we are entitled to them; we should no be forced to like or dislike certain characteristics. Furthermore we are entitled to openly express whether we like something or not.
An example of something impersonal is in the field of art: some people pay lots of money for works by, say Tracey Emin, but I find her works absolute and utter rubbish. In the same way I find the paintings of Van Gogh totally appealing, but other people don't like them. We can express our opinions on these works of art openly and freely. Why can we not express our opinions when we are talking about human beauty, or lack of it, according to our own tastes?