SWE interviews should be paid, part 2

Post Navigation:

1. $500 developer interview - with Thomas Yopes, co-founder & CTO at Cloudthread.io (YC S21)
2. Hiring by building product - insights from YC founders & Algora.io users
3. The status quo - with David Zeng, co-founder & CTO at Beacons.ai (YC S19)
4. Economic naturalist - explaining our observations using economics
5. Closing remarks - making it easier to contribute

Disclaimer: Algora.io simplifies interviews using bounties - visit documentation - view jobs - try it out.

When engineers change jobs as frequently as restaurant workers, while startups take 3-6 months to hire a new contributor, does any work actually get done half the year?

More broadly, in a low retention labor market with slow hiring, rising wages and updated preferences on flexible & remote work, how should modern organizations grow their teams?

Restaurants figured it out: they bring their most promising candidates for a day on the job. A paid shift, doing real work.

Super sensible and productive way to conduct hiring. But it's not just for brick-and-mortar businesses.

$500 developer interview

Thomas Yopes, co-founder & CTO @ Cloudthread.io (YC S21) is doing something creative.

He calls it a "Ride Along": top candidates spend a day of coding with his team and receive $500.

I had a call with Thomas recently where he shared his team's experience from their latest Ride Alongs.

1. What was your rationale for doing these Ride Alongs?

"My experience hiring non-ideal people; I want to avoid it. Bad hires are so epically painful. They are negative 10Xers."

2. Did you see anyone else doing this?

"I did take-home assignments before, this seemed like a reasonable extension. I didn't see anyone else doing this."

3. What was the experience like?

"Very exhausting. But we're very satisfied with the outcome. First we did screening & technical interviews to arrive at our 5-10 most promising candidates. We then did the Ride Alongs with them. We did 4 of them last week, have also done multiple in one day for different positions. We're looking for self-motivation and ownership, we want to see whether candidates can run with minimum input. We sent the offer letters a few days later."

4. Did you share your entire codebase or just part of it?

"Depends on the position, but we're liberal with sharing our code. People sign NDAs in advance, that's good enough for us.

5. Could you walk me through a Ride Along?

"Sure. We have a 1-hour meeting in the morning to provide more context and tech onboarding for the task, cover questions and have everyone get a sense of the culture. People work on their task throughout the day. They submit pull requests, we review their work, give feedback and then notify them with our decision."

6. What about the payment?

"That's the most trivial part. We just ask people their preferred method and make the transfer."

7. Any major takeaways or areas for improvement?

"By the end of it, we realized that the introductory part was the same for everyone. We could make an intro video instead of those first 15 minutes with every candidate. Better documentation for getting them started and FAQs would also help."

8. Would you recommend this to other founders?

"Definitely. I really think more companies should be doing this. The process is consuming, but it's an important decision, can't do it lightly."

Thanks once again to Thomas @ Cloudthread.io for sharing his story!

Hiring by building product

1. Kevin Hale hiring remote engineers at Wufoo (YC W06) - YC How to Start a Startup 2014: Lecture 7 [44:20-45:34]

Question from audience: "How did you hire people that you felt would be able to work remotely in this kind of environment that is not standard?"

Kevin Hale: "So, pretty easily, you have them work on a side project for you, so you contract them out, and they have to work remotely as such. Usually the project I like to have them work on is about a month long, they can do things much faster for a week, but usually you get a good sense of how well people manage themselves and work on things from a project like that. So that was always the first assessment, we never did anything just by interviews.

2. Hiring Engineers with Ammon Bartram from Triplebyte (YC S15) - YC podcast 2017 [27:23]

Ammon Bartram: So, as I mentioned earlier, the core problem is there’s this tension between the skills that can be measured in an interview, solving small problems quickly, and the skill that matters as a programmer, solving big projects over a long period of time. And so the first approach you can take to interviewing is just say, okay, we’re gonna not do it. We’re gonna do like trial employment. Or something like that. And that totally works. If you’ve worked with someone for a week, you have a far better read of their skill than I think anyone can get during a three to four hour interview. The problem is that there’s a pretty strong bias in who’s willing to do trial employment. And it’s an adverse bias. Many of the best programmers have lots of options. And if your company requires that everyone do this trial employment period, most of them are not gonna do it, so they’re gonna say no. And, obviously, anyone who currently has a job - [they can’t leave for a week, can't do it]...And so I think in the end we’re left with a thing kind of like the famous, “Democracy is the worst form of government, except for all the others.” I think that interviews are the worst way to evaluate engineers, except for all the other options.

3. Elliot Schrock, CTO at LithoByte, and Calvin Collins, graduating senior at Williams College

Elliot met Calvin on Algora, interviewed him asynchronously using a paid coding challenge ($300), did 4 months of contract work ($13,000) using bounties and then hired Calvin full-time. Elliot himself called this 'trial employment'. In fact, we credit Elliot with the whole idea of screening/interviewing using bounties, he was the first person to use Algora in this way. He even requested this neat feature called 'screening repositories' to interview all candidates in a single repository using a single bounty + anonymize candidates' activity (commits, comments, pull requests etc.) to avoid biases when doing code review. Elliot was a fantastic early user to have and we're really grateful.

4. Zaf, co-founder & CTO at Algora.io, used a coding bounty to hire summer interns.

After posting our internship & reviewing a couple dozen resumes, we shared this paid challenge ($200) with the 10 student engineers whose resumes we liked the most. We gave each of them a week to complete it. A couple of them didn't touch the challenge, a few started it but dropped it half-way, 5 of them submitted pull requests, and three of them (1, 2, 3) were awarded the bounty & summer internship.

The challenge required on average 12 hours to complete (pick up Typescript if not already familiar, code up a table with React & Tailwind, fetch data using our API and sort the table), while the candidates described the process as fun, engaging and instructive.

We also observed that the most promising candidates on paper (4.0 CS GPA, personal projects, past startup experience etc.) were the ones who actually solved the challenge. That's the opposite of the adverse bias Ammon Bartram talked about. However, we can't generalize further on this observation at this stage.

We reported this on our first SWE interviews should be paid blog post and got a lot of diverse reactions on HackerNews - thanks to everyone for sharing your feedback, opinions and insights, we really appreciate it. We do acknowledge that the first post was a little thin, edgy and even click-baity, but hey it helped us learn all these new perspectives from you :) This new blog post is an effort to provide some more substance, context, examples and perspective on this hiring methodology.

The status quo

This is certainly not how big tech companies hire, and what university students experience technical hiring to be like. At the moment, the majority of startups are also unfamiliar with this methodology.

Still, it is perfectly reasonable that only a subset of startup founders will find this to be right for them.

Here's what David Zeng, co-founder & CTO at Beacons.ai (YC S19), thinks about this and how his team approaches hiring (emphasis provided by the post author):

"It definitely is novel. I prefer to make an ultra-concise interviewing process. We typically do two technical questions, and then two or three 30-minute chats. My approach is that it's a numbers game for both sides and I would personally dread spending an entire day with a candidate that might not even accept, when there exist sufficient solutions that require less time. It's great if your process works though!"

"We've spent a lot of time developing a rich library of interview questions that test engineers on real-world scenarios to assess their technical proficiency in a single 60 minute interview. We constantly revamp these questions. If you can't decide within the 60 minute technical interview on a key aspect of the engineer's technical proficiency, that's a bad question on our side"

"It's a numbers game, and the number is super hot right now. As a startup, the two best things you can offer are agency and ownership, something that money can't replace compared to the bigger companies."

"It takes a couple of months from posting to engineer because we have exceedingly high bars for hiring. We are typically hiring the first engineer of their kind for our company and are only looking for Senior+ (L3+) as the first hires. Once we get past that, it gets a lot easier. The main thing I think about is management ratio. For every hour I put into the report, how many hours of my own equivalent labor do I get back? For me, that ratio is almost 40 for my first-line reports because our corps just takes the specs and runs for almost the whole week."

Thanks again to David @ Beacons.ai for the valuable insights!

Economic naturalist

The most important thing for founders is their time. And we have seen different founders choose to use their time very differently when hiring engineers.

In this last section we try to understand what's going on using economics.

Job Market Signaling 1973 - Michael Spence - 2001 Nobel Laureate in Economics:.

HIRING AS INVESTMENT UNDER UNCERTAINTY

"In most job markets the employer is not sure of the productive capabilities of an individual at the time he hires him. Nor will this information necessarily become available to the employer immediately after hiring. The job may take time to learn. Often specific training is required. And there may be a contract period within which no recontracting is allowed.

The fact that it takes time to learn an individual's productive capabilities means that hiring is an investment decision. The fact that these capabilities are not known beforehand makes the decision one under uncertainty.

This essay is about the endogenous market process whereby the employer requires (and the individual transmits) information about the potential employee, which ultimately determines the implicit lottery involved in hiring, the offered wages, and in the end the allocation of jobs to people and people to jobs in the market."

A CRITICAL ASSUMPTION

"It is not difficult to see that a signal will not effectively distinguish one applicant from another, unless the costs of signaling are negatively correlated with productive capability.

Signaling costs are to be interpreted broadly to include psychic and other costs, as well as the direct monetary ones. One element of cost, for example, is time."

Screening/Interviews aim to produce reliable signals about the candidates' productive capabilities. Producing these signals comes with costs, both to the hiring managers & the applicants.

Applicants

A university degree is a signal (modeled by Spence). Succeeding at a big tech technical interview is a signal. Triplebyte performance is a signal. LeetCode performance is a signal.

Job candidates can spend exorbitant amounts of time going through these hoops, but better engineers presumably need less time (lower cost) to transmit any given signal. The fact that these signals are costly to produce is what makes them reliable (otherwise they would be imitated).

The underlying assumption is that low productivity candidates opt-out of signaling (cost>benefit).

This costly signaling model explains our own observation that the candidates with the strongest grades & resumes (appearing to be more productive) were the ones who actually decided to start, and successfully completed, our screening bounty.

It would also explain Ammon's claim of adverse selection in trial employment if he had illustrated why a highly productive engineer (lowest cost) would ever refuse trial employment for the highest paying jobs (largest benefit). "The best programmers have lots of options" is not a sound economic argument for proving adverse selection.

Hiring managers

We now have a better understanding of candidates' signaling costs when it comes to screening & interviews, but what about hiring managers' & companies' (transaction) costs incurred for observing these signals and making a hiring decision?

David does not believe that spending a day working with a potential employee is an efficient use of his time (a view shared by most founders indeed) and prefers to use technical interviews, while Thomas, Kevin, Elliot & Zaf prefer to hire by building product.

Fundamentally, a founder's belief about the predictive validity of a hiring method, what they believe constitutes an efficient use of their time and their risk-aversion all affect their choice of hiring methodology.

The predictive validity (signal strength) aspect is exemplified by founders asking candidates for a real contribution (build a feature from the backlog) in order to evaluate them accurately.

The time aspect is exemplified by David's "management ratio", explaining his dislike towards the managerial overhead & sunk costs of administering interviews that involve real work.

The risk-aversion aspect is exemplified by Thomas' motivation towards avoiding those "negative 10Xers", making his ride along almost seem like buying insurance against making a bad hire (or making an investment towards his team's capacity to onboard new contributors).

Of course, human behavior is much more complex, but the above is a fair attempt towards explaining our observations.

Closing remarks

There's a job to be done -> People are needed to perform the job -> Hiring managers need to answer:

1. Can this stranger get the job done? (Performance)
2. Is this someone we want to work with & vice versa? (Culture)
3. Can we help them contribute immediately? (Onboarding)
4. Is our hiring accurate, consistent and scalable? (Process)

A day on the job gets the job done.

Pulling it off means minimizing the time & cost of turning a stranger into a contributor. That's why we're experimenting with bounties (you can see how they work in our documentation).

Whether you're hiring full-time or looking to collaborate flexibly, making it easier to contribute can go a long way.

Such a long way, that one day it may even make the SWE hiring process just a little more fun and productive for everyone involved.

What do you think?