If I were to wear a buddhist swastika would people think I was a Nazi?

I'm a little concerned, would they make me take it off in school because they associate it with Nazi's and not buddhism.