I think you hit this issue squarely at its head, Emily.
I've come to the conclusion that, although not all things are black and white, people like things to be black and white. It just makes life simpler; it reduces a spectrum of possibilities to a "us" versus "them" thing again. 'What really bakes your noodle' is how my conclusion pits me against everyone who believes there is only black or white, see? An evil web is weaved.
There are, I think, some reasonable dichotomies. The human sex genes are an example. The human species has just two chromosomes to play with, X and Y, and these determine sex (I hope the biology peeps will help out here). There are four mathematical unique possibilities (XX, XY, YX, YY)
but only two combinations of our chromosomes (XX, XY) occur naturally, right? (Actually wrong; I'm oversimplfying things. Where are the biologists?? Last I heard, chromosomes have variances that mutate the sex gene but I don't think we should go down that road. Most (or all) sex mutations cannot reproduce, which is only to say that I value the person's varied sex but that variance cannot biologically contribute in the species' evolution). IF right, then that's it; there's no spectrum of sex, at least not "bio-genetically".
It sounds to me like some people are linking the biological makeup of our species to gender, ie. male = man and female = woman. Anthropology and sociology have exposed me to the
possibility that gender is a cultural and/or social construct independent of biology. If it is, then I would have to say there are many (infinitely) possible genders. But I don't see much meaning to the concept "gender" if every individual constructs their own gender, see? A universal definition would be easier.
Personally speaking: I am male, bio-genetically, but I identify with a few genders, including adrogyny. How's that for confusing?