« Back to home

Inside Interview Code Tests

Posted on

Introduction

Code tests are an often debated interview step. Depending on the company, the team, the job market and other factors, code tests may or may not be the right choice for your interview process. Some engineers swear by them (See: Joel on Software), some hate them or think they're useless. If done well, they can be another tool in your interview toolbox.


What makes a good interview step?

A code test is like any other step in the interview process, and you should apply the same widely regarded “best practices” about interviews to the “code test step”.

Keep The Goal In Mind

It's very important to keep the goal in mind during an interview and during a code test.

You aren't asking questions to fill the time, or because they're the questions you think are typical of interviews. Each question should be accomplishing something and each answer should have one of two outcomes: either the person is someone to hire, or someone to not hire.

You're making a determination of whether someone will be excellent (not competent) in a specific role. Understand the role, and what the role requires before the interview. What does the candidate need to be excellent at? What can they be “just OK” at? What are they allowed to be bad at? Remember that excellence is the bar, but at the same time nobody is perfect. Ask questions - or invent a code test - appropriately.

Respect The Candidate's Time

Candidates are people too. Most candidates are still working, have full time jobs, family responsibilities, not to mention other interviews. A few hours is a perfectly reasonable request, and manageable by the candidate. A few days is not. The goal is not to push candidates to prioritize their interviews or judge us before you've even interviewed them.

Because code tests are the most time intensive step, you should do them later in the process, and only if they are necessary to make a final decision on the hire. You should be transparent with a candidate before asking for such an investment. Some key phrases:

  • “We won't ask you to invest the hours into this if you're not a strong, promising candidate.” *

  • “You've passed our interviews so far, but we need this to make a final decision”.*

Put The Candidate At Ease

It's important to make the candidate feel at ease. Nervousness is normal. Anxiety about interviews is normal. These aren't always indicators of poor performance or poor communication, and sometimes indicate poor interviewing skills! The candidate should feel comfortable, should be able to be themselves and be honest. It is the interviewer's responsibility to create that environment. Likewise for a code test, you should make sure the candidate is at ease with the problem and empowered to do their best with the time, setting, help and tools that they would use on a daily basis.

Leave a Good Impression

There's an element of recruiting within the interview process. In the software profession, there is a lot of demand for quality talent, and a relatively small supply. If you've attracted a promising candidate, it's also important that you leave them with a positive and competent impression, so that they'll want to work with us. The programming test should not seem like a waste of time or a chore, it's up to us to leave a good impression about this step.

Use The Interview To Make Up Your Mind

A good interview means you've made up your mind, with strong logical reasons that you can articulate to the rest of the interviewers and your team. Coming out of an interview without either a strong negative conviction or a strong positive conviction might mean that you lacked the correct questions, or were unwilling to press someone to get to the bottom of a topic. This doesn't help us hire! For a code test, you should be 100% confident that the test will be a valuable part of your decision, and be able to articulate why and how.

Take A Positive Approach

Instead of presuming everyone is “a liar” or “trying to fool us” you should presume candidates are good people and let them demonstrate their talent and fit for the team. Ask yourself if you want to work for a company that fundamentally treats hiring as conflict or a battle. Imagine if that was how people treated dating! An interview is just a “first date” and the code test might be a “second date”. Presume the best and then see where it goes.

Remember The Other Side Of The Coin

You can't have good interviewing - a process for adding people to the team - without a solid process for removing people from the team. Hiring is not a lifetime commitment and if someone doesn't work out, they should be let go. There is a cost involved in hiring and firing but at the end of the day if someone “fools” their way through the interview, this isn't an excuse for ruining the whole team. This is why you “look for excellence” rather than “rooting out” bad candidates.


Why ask for a code test?

It Allows Us To Look For Excellence

All programmers can build a program, and make it run. Likewise, meeting a few simple requirements for a problem that can be solved in a couple of hours isn't a given - but it doesn't demonstrate excellence.

Most programmers, if asked “what makes code excellent” know the words to say, or some of them. You do a code test to see this in practice.

Excellence in a simple project means that the developer thought about more than just making it run. Excellence means that they think about, and have invested in things like code quality, performance, readability and maintainability, style or structure. Not every engineer is excellent at all of these things, especially with limited time, but a good engineer will be excellent at some of these no matter the budget.

Giving a simple problem helps ensure that there is “time to make it good” not just “time to make it work”.

It Lets Us Find Finishers

There's making it work, and then there's making it complete. You are looking for candidates to take projects and tasks from inception to completion, delivering something that they could demo, explain, and be “proud of”.

Finishing something is the most difficult part; the “80/20” rule explains that the first 80 percent is easy and the last 20 percent is hard.

You need to setup the code test to allow and encourage people to finish, but then strictly look for finishers.

It Checks Ability To Work Remotely

If you're a remote-friendly development team, being able to handle a remote exercise is even more important than many co-located engineering teams. Unlike co-located development teams, it's harder for remote teams to communicate and you more often rely on written requirements. You also put a greater emphasis on the individual's ability to plan, to mange their own time, to hold themselves to a high standard, to do more than the minimum and to get things to completion alone. As part of the exercise you should see signs that the individual enjoys remote work, can translate written requirements into good programming, can manage their own time, can hold themselves to a high standard without micro-management, and above all an ability to drive something to completion.

It Helps Communicate Expectations

The problem description, and making the candidate write the code gives them a “small taste” of what it would be like on the job at the company. For example, if you require unit tests, and the candidate does not believe in writing unit tests, the experience they may have solving the problem may help them understand the company's expectations more concretely and help both parties decide on the fit.


When to do a code test?

The best time to do a code test is after:

  • You've answered the candidate's basic questions
  • You've determined that (at least on the surface) they might be a good fit
  • You've determined that (at least on the surface) they are technically qualified

Code tests are expensive. The candidate has to spend a lot of hours - arguably the biggest part of their interview - to create and deliver a solution for a job that's just one of many they may be considering. For the company, evaluating a code test takes valuable engineering time. Finally, because code tests are seen as a “painful” part of the interview process, they somewhat cost the company in terms of impression and reputation.

If you think someone's a qualified engineering candidate but want to evaluate their ability to deliver excellence, to be a finisher, or to work independently, then code tests can be a great follow-up tool.

Finally, I do recommend a follow up call for every code test just to discuss the solution. Not only does this provide a basis for concrete questions and discussion, but it also vets that the candidate understands their own solution and didn't end up copy-pasting it.


What are the wrong reasons for a code test?

Don't Know How Else To Interview

Code tests can be used as an excuse for poor interview skills or poor interviewers. It's arguably easier to “just email them a problem” than really dive into a discussion and ask challenging questions. Interviewing is a skill but a necessary one, and a code test shouldn't be a “substitute” for a real technical or role-based interview.

Can't Tell If The Candidate Can Code (Or Suspect They Can't)

If you find that you've gone through a technical interview and don't have a strong positive impression of whether someone has experience coding, has familiarity with a programming language, has familiarity with development tools, or whether they seem to be programmers at all - you need to immediately improve the technical interview.

If you suspect, after a *good *technical interview from a skilled interviewer, that a candidate cannot code well, does not understand fundamentals, or simply cannot effectively communicate these things, you should not proceed with the interview process or conduct a code test - as they have already failed.

Only candidates who pass the technical interview should be taking a code test. As such, you should be confident that - so far - they'd make an excellent addition to the team.

Internal Gate-keeping

Only an engineer can really evaluate a code test. If you are using this code test to “put the power in the hands of the engineers” about who can and can't be hired, you should revisit your whole interview process and make sure that the right people are making the decisions that matter.

Checking The Boxes

Programming is more about learning and creativity than it is about repetition. The most excellent developers will be life-long learners, who can and do pick up new skills, technologies and even languages as needed. Measuring this is not the purpose nor strength of code tests - but neither should the code test focus on a checklist of language or technology bullets.

Only Smart People Work Here

Big tech companies have especially gotten into this habit - of asking really difficult problems, theoretical problems, or riddle-like problems, and explaining that “a smart person can get this done”.

The proven truth is that solving those kinds of problems is a skill that comes from practice, and that there are patterns to these kinds of problems that both can be learned - and are not map-able to day to day developer activities.

Looking for “smart people” by some arbitrary definition of being able to recognize and solve a twisty problem focuses on the wrong thing - not excellence at the job or in the role. It simply self selects for a specific type of mentality and culture.


What makes a good code test?

Low Friction

If you're being honest about code tests, they are not universally loved and many excellent engineers refuse them, avoid them or do not agree with the practice. Although you have made a choice that code tests are useful for the decision-making process, you should make the test as low friction as possible. Build a code test that someone can't complain about and be as accommodating as possible, while still meeting the goals.

Small, Simple Problem

You are not testing the candidate's work ethic or whether they are intimidated by big problems. You are not testing their raw speed or even their ability to produce under extreme pressure. You cannot reasonably test the candidate's ability to handle highly complex problems with a “short code test”. For these reasons, a simple test is the best test.

If they fail this test, it should be because they lack the skills or do not meet the company's standards - not because of any deficiency in the problem!

A good rule of thumb is “ask for a solution, not a whole application”.

Excellence Possible In A Short Time

It should be possible for a good candidate to not only solve the problem, but do a complete and excellent job within a short amount of time. Remember to estimate time for all of the kinds of practices you expect from excellent contributors.

Give Unlimited Time

Although the problem should be solvable in a small amount of time, you aren't trying to put pressure on the candidate, and you want to be respectful of their time and lives. On a remote team, flexibility of time is a strength not a weakness. You need to find out if someone is a finisher, versus finding out if they are fast.

Do the Problem Yourself

Don't ask someone to do a problem you haven't done yourself. Don't estimate it given only your own biases and experiences. Don't make assumptions that the requirements or problem are clear without proving it. Do the test yourself, get a feeling for how to solve the problem and how long it takes. Are you really too busy?

Have someone else review your code. Look over someone else's solution. Be one with the code test.

Be Domain Friendly

Remember that you might be knowledgeable about some subjects and have context in certain business or problem domains. You might be great with lotteries and games, or know data problems. Most candidates will not have experience in the same specific areas. Design a problem either with a domain familiar to almost everyone, or a domain that anyone can understand quickly.


What makes a bad code test?

Expectations Unclear

The requirements should be clear. This is like writing a good story or a good ticket. It should include examples or even use cases or scenarios. Clear description of Inputs and of Outputs. Constraints. Stakeholders. Priorities. Everything!

Excellence Unexpected

There is no reason to hide the fact that you expect excellence. You will be telling the candidate what you expect from them before you hire them right? So tell them before the code test what the company's expectations are in terms of things like excellence, quality, polish, QA, etc. Include a list of expectations for either both role or the task rather than leaving it up to chance.

Results Create A Mystery

If you can't tell if the candidate is good but ran out of time, or if they weren't good, or simply gave up, or if the problem was misunderstood, or too hard, then there is a problem with the problem or method for the code test.

There should be no possible way that the problem isn't clear and doable. The solution should be a clear yes and a clear no, and not because of time constraints or any debate about the problem itself.

Too Much Overhead

The candidate has to spend a lot of time setting up their environment, credentials, frameworks, libraries, or building an “empty project” rather than working on their own original code. This isn't a good use of time and often is a sign that the problem could be simplified.

A One Tech Pony

If the exercise depends on the candidate learning or being experienced or highly expert with one specific technology or tool, without which they cannot succeed, this is probably a bad exercise for determining general development ability.

It's A Real Task!

Don't use a real problem from your backlog.

If you find there is any bias in the problem domain - or if the exercise would be arguably easier for an expert in a specific business domain, find another problem. It can be VERY difficult to recognize your own biases and see things from an outsider's point of view. For that reason, ask a an outsider to vet the problem or maybe get a consensus on the domain.

There are many kinds of bias - gender bias, racial bias, and so on. You should try to exclude these biases as well, and it can sometimes seem like a problem is “normal” even when it's highly rooted in your own culture.

Eureka!

With some problems, a leap of understanding, or “eureka” moment separates success from failure. Perhaps you expect them to “get stuck” then “make a breakthrough”... or implement one specific algorithm that is optimal. The candidate either had it or did not. This is a hit and miss way to evaluate developers as even a good developer might miss that one solution one time. In reality, developers can and do collaborate on “hard” problems like this!

Theoretical Mathematicians Only

Unless you are hiring for theoretical expertise - the problem should not rely upon a university education with courses in math or computer science theory. You're welcome to require these things from candidates, but don't test them using the code test step. Although there are exceptions to this rule, in general most companies agree that for much of modern software development, abstraction and tooling make day to day tasks less about theory and more about execution and design.


How do you evaluate a code test?

An example evaluation might include scoring a candidate from 1 to 10 in the following areas that focus on excellence, self-management, and finishing for a simple problem.

☑ Delivered Working Solution

☑ Precisely Meets Or Exceeds Requirements

☑ Time Well Managed, Plan Followed and Priorities Balanced

☑ Solution Seems Polished and Finished To 100%

☑ Readable, Good Style, Good Naming

☑ Well Structured, Short, Simple, Decoupled, Composed

☑ Edge Cases, Errors, Exceptions Considered and Handled

☑ Documented, Presented, or Demo'd Well

☑ Tested Well (Positive AND Negative Cases)

As a rule of thumb, it's OK to forgive a bug or two. You wouldn't fire a developer for one bug, so you shouldn't discount a developer for one bug. Consider first whether it was an understandable oversight, a mistake, or if it was an indication of being sloppy.

In addition to computing a score, you should look at your own test (your team did the test yourselves right?) and grade on a curve. If your own developers are getting on average 80% scores maybe you don't want to hold candidates to 90% or higher. As much as you tell yourself that you do these things well, you probably don't do everything perfectly, and it probably averages out across different people with different skills.


Where do you look to find inspiration for code tests?

I find a great resource for simple code tests is to look for curated collections of code katas. Code Kata is a term coined by Dave Thomas, co-author of the book The Pragmatic Programmer. The idea of a Kata is a simple problem you can do many times, each time getting better at it and learning something new. The concept is borrowed from martial arts, and good code katas tend to have a lot of room for improvement and thought, but likewise can be solved easily. A code kata is also a good way to learn a new programming language, and is how I originally learned to code with both Ruby and Node.js.

Because Katas are popular, you can find solutions on the web. However, there's a difference between a solution and delivering excellence, and if you include a follow up discussion about the code, I've rarely seen an example of plagiarism paying off.

Some of my favourites include:

If you decide to go with a code kata, I recommend adapting it and clarifying it into a “user story” that's clear and to the point, rather than open ended. I also recommend doing it yourself before sending it to candidates.


Final notes

If a developer has publicly available sample code available, that was clearly developed by them and demonstrates the qualities you would look for, why not accept that instead of a code test? Or if a developer has tons of experience working remotely, or a great reference, maybe you don't need to ask for a code demonstration.

As a final word, you should explain to the candidate why your company uses a code test, and even sell them on how good a practice it is. If you can't make an elevator pitch that works, you shouldn't do it.