Fascists are extremists. Anti-fascists are people who oppose Fascists, and come from all political walks of life.
My grandfather fought the Fascists and spent years in a Fascist PoW camp. He was politically conservative (though of course nothing like contemporary Republicans - I can't imagine what scathing names he would have had for Trump!)
I live in a very liberal city now, but one where tens of thousands of our citizens were dragged off to concentration camps to die. We are still a city of anti-fascists.
I am not a Communist and never have been, and yet I am anti-Fascist.
Is this all clear now? Fascists == extremists. Anti-fascists == everyone else.