About 20 years ago, Jeff Atwood made the claim that 199 out of 200 programming applicants have no coding ability whatsoever. I have already written about it to death, including here, and my conclusion stands even though I have received criticism: Jeff Atwood’s number does not really come from anywhere. It’s derived from a strictly hypothetical blog post about hiring the top 0.5%. A lot of people argue that the SPIRIT of the claim is essentially true, and that the vast majority of coding candidates cannot code – but he stated “199 out of 200” like it was a fact.
Atwood’s strictly technical content is often great, but his editorials can sometimes be reductive.
The Camel Has Two Humps
There is a paper that made the rounds several years ago, which Jeff Atwood promoted. It argues that the researchers 20 years ago found a reliable test for determining whether or not someone would make it as a programmer.
The paper is…confusing, at least to me. Their test was to essentially administer a series of coding questions to people who had never touched coding, and to then assess their mental model for solving them. A comment explaining it is here, and the original paper is here.
But they never actually show you the whole test, just a single question from the test:
Then the paper was retracted, and they explained the retraction here. They were unable to replicate the results.
Honestly, the most interesting thing to come of all of this may be Adam Kay’s thoughts.
Full quote:
Notion 1: Good science can rarely be pulled off in an environment with lots of degrees of freedom unless the cause and effect relationships are really simple. Trying to assess curricula, pedagogy, teaching, and the learners all at once has lots of degrees of freedom and is *not* simple.
So for example we’ve found it necessary to test any curriculum idea over three years of trials to try to normalize as much as possible to get a good (usually negative) result.
Notion 2: Most assessments of students wind up assessing almost everything but. This is the confusions of “normal” with “reality”.
For example, in our excursions into how to help children learn powerful ideas, we observed many classrooms and got some idea of “what children could do”. Then I accidentally visited a first grade classroom (we were concerned with grades 3-6) in a busing school whose demographic by law was representative of the city as a whole. However, every 6 year old in this classroom could really do math, and not just arithmetic but real mathematical thinking quite beyond what one generally sees anywhere in K-8 [kindergarten and grades 1 through 8].
This was a huge shock, and it turned out that an unusual teacher was the culprit. She was a natural kindergarten and first grade teacher who was also a natural mathematician. She figured out just what to do with 6 year olds and was able to adapt other material as well for them. The results were amazing, and defied all the other generalizations we and others had made about this age group.
This got me to realize that it would be much better to find unusual situations with “normal” populations of learners but with the 1 in a million teacher or curriculum.
I found Tim Gallwey, who could teach anyone (literally) how to play a workable game of tennis in 20 minutes, and observed him do this with many dozens of learners over several years.
I found Betty Edwards who could teach (again literally) anyone to draw like a 2nd year art student in one intense week.
And so forth, because what the exceptional teaching is doing is actually allowing assessment of what general human beings from a typical bell curve can learn from crafted instruction.
Coding in the days of AI
Now that we have widespread generative AI, the gate-keeping can either come crashing down, or accelerate. Now it may be more important than ever to administer whiteboard tests…or less important than ever, idk.
Motivation
So…I think I can understand some of the subtext (if I am using the word correctly) behind my initial reaction to Atwood’s blog posts, and the criticism I received in turn.
Let’s draw a dichotomy. On one side, there are the people who think coding is a unique ability that some can never learn, and that very few can ever learn to do well. On the other side, mine, you have those of us who believe virtually anyone can code.
So to someone like me, the notion that the vast majority of software engineer candidates cannot code at all, regardless of university or work experience, was absurd. What did they think these people were doing? Cheating? Copying from StackOverflow? You CREATED StackOverflow, Jeff, this one is on you.
Now we have AI, and maybe the entire thing is just moot. Candidates absolutely may be getting through university with generative AI. Tech workers, too…many companies do not just allow generative AI, but actively encourage it. So the LeetCode style interviews can continue, I just wish the interviewers cut candidates more slack.
You have 45 minutes. You are asked a question that people in the industry typically never solve. Imagine the pressure of having years of experience compressed into the result of a single hour-long test, topped off by the idea that maybe your judgmental interviewer will BLOG ABOUT YOU afterward and how silly it was you had the audacity to pass yourself off as a coder.
Closing Thoughts
But…yeah. Chip on my shoulder, clearly. Before getting a full-time job, I applied to hundreds of companies.
You have experiences, knowledge, your own unique brand of trials and struggles and triumphs. If you already worked but were recently laid off then you have systems you maintained, colleagues you built bonds with, industry knowledge. And now, in a difficult market, you come face-to-face with an interviewer who judges you as unfit based on the result of some trivia?
The idea I hear is always the same: These candidates must be lying in their resumes.
And maybe it’s true. Maybe I’ve spent so much time on one side of the table that I have no notion of how it feels to vet candidates.
But how does the quote go? You are a human being, damn it! Don’t let one judgmental interviewer bring you down and make you question your own worth as a developer, let alone as a person.