Barney J Smith

build notes · mocha

what i got wrong first

the first version of mocha could hold a conversation. it just couldn’t tell you whether you’d done well.

that sounds like a small gap. it isn’t. a model that asks good follow-up questions feels impressive for about four minutes, and then a student asks the only question they actually came for: was that answer good enough to get me the offer. an llm will happily answer that question. it will also answer it differently every time you ask, which is worse than not answering at all, because a student practising against inconsistent feedback learns nothing and trusts it anyway.

so the real problem was never generating interview questions. it was making a score mean the same thing on tuesday that it meant on monday.

why consulting and finance first

the obvious move was to build one general interview coach for every student and every industry. i built for consulting and finance instead, and it’s the decision the whole product rests on.

general coaching produces general feedback: be more specific, add more structure, quantify your impact. true for everyone, useful to nobody. consulting and finance have something rarer, which is published, stable, widely-agreed hiring rubrics. a case interview is scored on structure, hypothesis, quantitative comfort and synthesis. those are not my opinions about good answers. they are the criteria the firms use, which means a score can be defended rather than merely asserted.

the cost of that choice is real and i want to state it plainly. every new track is not a config change, it’s weeks of rubric work: reading how a function actually evaluates candidates, encoding it, then testing whether the model applies it consistently. it means mocha is narrower than the tools it competes with, and there are students it can’t help yet. i took that trade because a narrow tool that is right is worth more to a student than a broad one that is plausible.

the part that took longest

evaluation, not generation. getting a model to produce a coherent mock interview took days. getting it to score two similar answers similarly took months.

the honest summary is that most of the engineering went into constraining the model rather than expanding it: giving it fixed criteria, worked examples of what a 6 and an 8 look like against those criteria, and a narrow job on each pass instead of one broad instruction to “evaluate this answer.” free-form judgment was the enemy.

what students actually did with it

the thing i under-weighted at the start was that practice is social. students didn’t adopt mocha one at a time, they adopted it in clusters, which is why 22 universities looks less like a marketing funnel and more like a series of small rooms where one person brought it to everyone else.

what i’d do differently

i’d build the scoring layer before the conversation layer. i built the impressive half first because it was the half i could demo, and then spent much longer than i planned making the unglamorous half trustworthy enough to ship. the conversation was never the product. the score was.

mocha is live at usemocha.app