I've sperged about this long ago, maybe back on KF, and for the most part I agreed with this.
Another aspect of it that I focused on was the potential 'seriousness' of the joke and I think what I boiled it down to was that it's (mostly) ok to joke around like that with people that you would not be attracted to. If someone is a genuine lesbian making those lesbian jokes, I think it's pretty weird. Same with gay guys making those gay jokes. Or straight people making those jokes about the opposite sex. Edit: And this is not 'unicorn-based', this is 'it would be weird to have serious sexual flirting about your coworkers''.
My older thoughts were more detailed than that, but I forget what other factors I talked about...