Well, it is. Or, at least, it used to be.
This catchphrase, widely used as an interlineation, originated with a story by Wallace Macfarlane, "To Watch the Watchers," in the June 1949 issue of Astounding. It concludes: "It is a proud and lonely thing to be a man."
We haven't located the first use of the obvious pun, which appeared in fanzines throughout the 1950s, but it became indelible in fannish memory after Robert Bloch used it in a story called "A Way of Life" (see FIAWOL) in the October 1956 Fantastic Universe, a heavily Tuckerized post-Apocalyptic tale in which all of humankind evolved from fandom, closing with: "It is a proud and lonely thing to be a fan."
The phrase poignantly resonated with fans in the 20th century, who often heard their favorite genre dismissed as "that Buck Rogers stuff," and not uncommonly grew up as friendless, outcast nerds among nonreaders until their contact with fandom, which for many was through the mail only or at then rare conventions. Today, when science fiction is ubiquitous, the internet is full of sf discussion, and most weekends offer multiple choices of cons attended by thousands, the phrase is often used ironically.
See also: "Fans are slans."