Does racism need to be legitimately harmful in order to be considered morally objectionable? Suppose that black men incite an admittedly irrational fear in me, so that whenever I see a black man in public I cross the street -- should I feel compelled to correct this phobia? Or how about this: I find black men unattractive, so I don't date them.

I'm not sure what you mean by "legitimate harm," but it strikes me that any failure to accord others the dignity they are due as human beings causes significant harm to oneself and to others. If you agree that racism is a failure to respect human dignity, you ought to recognize it as morally objectionable and ought work to correct that failure in your own life by, for example, striving to overcome the racist fears you describe.