Racism is racism. It can arise through different situations. But it is still hatred of a race. Defined as "poor treatment of or violence against people because of their race". The situation that gave rise to that has nothing to do with if it counts as racism or not.
Also, if you want to get into "colonial roots". What do you call this?
http://en.wikipedia.org/wiki/Judea_(Roman_province)You are kind of right though that anti-semitism usually has different characteristics than other types of racism. It is a lot more closely related to racism against Romani or Armenians I think. A weird mixture of exiling them so that they have to work in niche fields in order to survive and then resenting them for it.
I am honestly fascinated by Anti-Semitism. I mean I am from Saskatchewan which has huge racial problems. But as many terrible things as I have heard said about Natives, I have never heard anyone accuse them of ending the gold standard, or making bread out of children. Anti-Semitism stands alone in its batshit insanity.
Edit: one more thing about Anti-Semitism (I took a class that went through Jewish history and it is something I really noticed). The Jews are always the exact opposite of what you want to portray yourself as, even if it is a total contradiction. Throughout history people have said the were they force behind communism AND capitalism, they have been cowards and warmongers. They control the liberal media and the far right lobbying. They are the hippies and the bankers.
Don't misunderstand me: there are certainly specific Jewish people involved in the left AND the right. Sometimes in very important positions. But this assumption of being the shadowy control behind every movement is something unique to the Jews