Oh! Cult following doesn't necessarily mean worship or anything at all to do with religion. From Wikipedia:
"A cult following is a group of fans devoted to a specific thing, usually a media work, such as a book, film, television or radio program, though some comic books, video games, musicians, writers and others also gain cult followings. Some non-media items also have what could be considered cult folowings (such as Tab).

These dedicated followings are usually relatively small and pertain to items that don't have broad mainstream appeal, though there are exceptions. Cult followings are often dedicated enough that many people of similar interest are familiar with one another due to convention gatherings, concerts, Internet chat rooms, or shops featuring the item(s) in question."

See, that describes us. But it doesn't say we're worshipping Tolkien or Sauron or Illuvatar or anything.

Also-
"'Cult following' is also used to describe the more obsessive fans of established mainstream performers. For example, many persons have been interested in Michael Jackson's music or in Disney films, but some fans take their interest to extreme levels, hoarding vast amounts of collectables. Some such "cult fans" occasionally veer into obsessive-compulsive disorders or stalking; however, cult followings do not necessitate that individuals partake in cultish activities."

There you have it. The former is what people usually associate with the word, I think! And LotR fans are certainly that!

However, I did find this link a few years ago-
http://www.uib.no/people/hnohf/greven.htm
I found it disturbingly intriguing.

I think LotR is a cult movie and a cult book, and that's a good thing. I'm actually quite proud of it!