No competitive athlete is going to want to participate in games that allow substance abuse. Most of the non gender specific individuals just want to be treated as the gender they identify with, no more no less.
Hermis are a whole other topic altogether. You're right in saying if you're born with a dick you're a guy and vice versa. You can identify as a gender if you want, just don't expect to be able to compete against whichever gender you choose. Especially when it's physical combat or strength tests. Female vs Male ? Fine, but fuck off if you think that a man should be able to fight women just because he doesn't want his hog anymore.