I think 10 recommendations seems excessive. Good programmers wouldn't waste their time working for a company that required them to bother that many people, just so you could figure out if they were a good programmer.

Think of it as a barrier to entry. Good programmers want good opportunities that don't make them bend over backwards to get. Path of least resistance.

If you make it hard for them to work for you -- they won't.

Seriously, who wants to call up 10 people and ask them if they mind if some random person calls them?

It's a waste of time and any good programmer is going to know this and have plenty of better places to work that don't require 10 recommendations.

Also, what a programmer works on doesn't say anything about how smart (s)he is. If you want to know how smart they are, ask them about as many random things as you can. How broad is their knowledge base? Smart programmers know a lot of stuff outside the world of programming.

Ask her about philosophy, beer, or cards. Ask about poker, pizza and travel.

Smart people have big minds that expand way beyond personal projects and portfolios. If they have a portfolio /at all/ then they are spending more time on marketing themselves than they are on programming and that's not such a good sign. I know you don't mean they have a binder with photos of work they did, or maybe you do, but either way, they should be spending more time on programming than marketing their programmer skills. If they harp on how awesome they are at programming, they probably don't know much about programming.

I'd say irrelevant projects done for fun are almost a better sign of smartness than those that were only relevant.

Ask the potential candidate about how the projects they have worked on could be improved. What went wrong, where were the bottlenecks, what kind of hardware was it, did they find any limitations with the software, programming languages, or environments they worked in?

Ask them how to build world peace and solve the financial crisis. See how rational they are. See how honest they are.

Introspection, Honesty and Rationality are probably the three most important characteristics I look for in programmers. If they can communicate with the "real world" around them as well, that's a plus.

Most people in the real world can't handle the amount of thought, blunt truth, and rationality that programmers dish out, so it becomes the job of the programmer to figure out how to communicate with others, so that just brings me back to the 10 recommendations thing... Probably not a good idea. I hope that was hyperbole.

Oh I totally agree. I just said 10 recommendations to make a point. If you can easily gather recommendations from individuals in multiple sectors, than this validates your importance and potential impact. Reputation is crucial, because as we all know there is a huge difference between excellent programmers and average programmers.

As for your second comment about judging "smartness", I would have to politely disagree. What you describe sounds more like "emotional intelligence." Can this individual relate to others on a human level? Do they seem like a "jack-of-all-trades"? This is actually crucial as well. If you do not have employees that can communicate well, and distill high-level thinking, then the transfer of those ideas is very difficult. As an example, we had a student in our group who was wicked "smart" but could not communicate their ideas seamlessly. Thus, when that student graduated, we were left with a bunch of beautiful code that none of us could understand.

I still stand by my original statement. If a prospective hire brought forth a solid portfolio of accomplishments, they will have defended their cognitive abilities and adept understanding of difficult concepts...assuming those projects were non-trivial.

smart and gets things done.

http://www.joelonsoftware.com/articles/fog0000000073.html