It's not clear to me which of two questions you're asking: (a) Is it always racist to use the word "niggardly"? (b) Can it be racist to use the word "niggardly"? I'd answer "no" to (a). It's not racist, and it's accurate, to describe Ebenezer Scrooge (before his conversion) as a niggardly character. But suppose someone uses "niggardly," perhaps mistakenly thinking that it's related to the N-word, in order to express racial hatred. I think that counts as a racist use of "niggardly," so I'd answer "yes" to (b).

As a white male myself, I guess I'm answering your question in the affirmative by even presuming to post an answer to it at all. Surely the question you asked is so broad that no one could reasonably answer it in the negative. Racism exists: some people or practices are racist. Sexism exists: some people or practices are sexist. There: I've said it, and I defy any reasonable person to deny my assertions or call them "illegitimate." Now, it's a harder and more interesting question exactly how much a white male can say about racism or sexism without losing credibility on those issues, but I'm inclined to think that a white male could, in principle, become the world's foremost authority on racism and sexism, and the burden of proof would rest with anyone who said he couldn't speak legitimately on this or that particular aspect of those issues: we'd be owed an explanation why not.