In the grimiore it says they are female I know but they are called Wizards in game and the grimiore cards still classy them as Wizards so call them what they are they are Wizards not witches
-
I know for me it was because the first time I heard a Wizard scream it reminded me of the witch from Left 4 Dead, and it kind of stuck.