And it's prejudice, if you ask me. A small minority of fans painting their entire upper body the colors of their favorite football team when it's 10° below doesn't lead to any media depiction of that team's fan base as "rabid." But I'm not bitter.
You're also not alone in noticing the distinction in treatment. It comes up in Henry Jenkins'
Textual Poachers, which is the go-to text if you're interested in academic discourse on media fandom.
The standard argument that you hear from folks is that sports are "real" whereas media fandom is built around a fantasy or fictional world, but that all goes to hell when you start thinking about sports fans participating in fantasy leagues, and there actually is fan fiction based on sports figures.
When I taught my freshman composition course centered on fandom, the department asked me to include more material about sports fans. Other than articles on their buying habits or on soccer riots in Europe, it was very difficult to find anything that treated sports fans as oddballs that I could use in the course reader.