I think this often results from people not understanding what you do at all, so when they see you do something unfamiliar at a computer, you seem like a hacker to them. This seems to happen a lot when I pop open a shell and start working at the command line: it just seems like voodoo to certain people (like my parents or girlfriend). If you can get the same affect by showing something off to the folks who read hacker news though, you're probably a hacker in the sense it's used around here.

Oh, and I don't know Clojure either, and I'm not losing any sleep over it :)