#AI-assisted development questions
AddedThis page is for the last few days before an interview, when you no longer need to learn a topic but need to say it well, quickly. Every question here is one an interviewer for a senior full stack role that "requires hands-on experience with AI-assisted development tools such as GitHub Copilot and MCP servers" might ask, or a follow-up they'd reach for when your first answer was good. The first paragraph of each answer is the version you'd say out loud: 30 to 90 seconds, with the decision first and the reason straight after. The walkthrough under it is the reasoning you need when the follow-up is "why?", and the link at the end goes to the topic that teaches it properly.
Think of it as a deck of flashcards with the working shown. Read the question, close your eyes, answer it in your own words, then open the answer and compare: did you name the decision, the evidence or mechanism behind it, and the limit? If one of the three is missing, that's the card to go back to. A few cards have small runnable models where behaviour or arithmetic matters; their output was produced on Node 26 (the MCP ones with @modelcontextprotocol/sdk 1.32.2) and is re-checked every time the site is validated. Vendor facts are the ones the chapters checked, with their dates; nothing here is anyone's real experience. Where a question asks for a story, the card gives the structure and what the interviewer listens for, and the story itself has to come from your own work, written in My notes.
This first group covers how assistants work and where they help, prompting for code, and leading a team that uses them (chapters 1, 3 and 8).
Show answer
Three shapes, completion, chat and agents, which differ in how much they do before you look. "Completion predicts the next few lines from the code around my cursor, and I see every token before I accept it. Chat answers questions or drafts code from what I select or mention, and it only knows what's in the conversation. An agent takes a goal and runs a loop: the model proposes a tool call, read a file, edit, run the tests, the harness runs it if the permission rules allow, and the result goes back into the context until the model says it's done. They run in the editor, the terminal or the cloud; the further from me they run, the more I rely on tests and review instead of watching."
- The mental model is three levels of delegation: a typist reading over your shoulder, a colleague you paste things to, and a contractor you hand a ticket and a sandbox. The more you delegate, the more your job becomes specifying the task and checking the result.
- The model only turns text into text. The harness (editor extension, CLI, cloud runner) decides what the model sees and what its output may do: context assembly, instruction files, tools, permissions, compaction. That's what a team configures, which is why "which model is best?" is the wrong first question.
- Where they run, as of the chapter's check on 27 October 2026: editor (Copilot in VS Code and JetBrains, Cursor, the Claude Code IDE extensions), terminal (Claude Code's CLI, Copilot CLI, generally available since 25 February 2026), cloud (Copilot cloud agent, Claude Code on the web).
- A good follow-up to be ready for: "which would you use for X?" The rule from chapter 1's quiz: the less you'll watch, the more the task needs a check the agent can run by itself, such as a test suite that defines "done".
- Then name the tools you actually use and one concrete task per mode. Don't list tools you haven't used.
Learn more: What AI coding assistants do.
Show answer
It sends the whole context to the model, receives one proposed step, checks its permission rules, runs the tool if allowed, and appends the result to the context for the next turn. "The model is stateless: it never touches a file, it only produces a request, and it learns what happened only from the result the harness puts back. So the context grows every turn and is re-sent every turn, which is the cost, and every tool call passes the harness's permission gate, which is the safety. A denial is just another tool result the model has to cope with."
- In chapter 1's agent-loop model the task took five tool calls and an answer, and the final context was 11 messages, all re-sent on every turn. Anthropic's context-window docs say each turn's input contains all previous history plus the new message.
- The permission gate is where writes and shell commands ask the human. A terminal agent runs commands as you, with your SSH keys, cloud credentials and push rights, so "auto-approve everything" turns any mistake, or any instruction hidden in a file it reads, into an action.
- A well-built agent reports what it couldn't do instead of claiming success: when the model's edit was denied, the stand-in answered "Not changed: the edit … was denied."
- Cost follows the same loop: chapter 2's credits model priced one chat question at 1.90 credits and a 30-call agent task at 61.16, about 32 times as much, because each call re-sends a growing context.
- The interview point: agents are a loop you configure (tools, permissions, the check that defines done), not a smarter model.
Learn more: What AI coding assistants do and Copilot features at a glance.
Show answer
The model predicts one token at a time from a probability over its vocabulary, and it can only see what's in its context window for that request. "A token is a chunk of text, roughly four characters of English. The context window is everything the model can look at in one request, including its own answer, and there's no memory between requests; chat history works because the harness re-sends it. What the model knows about libraries is frozen at training time and mixes versions. So anything newer or private, our codebase, our conventions, our device API, it only knows if it's in the context, and when something isn't there it reconstructs it from patterns, fluently and sometimes wrongly."
- The chapter's mental model: a very well-read autocomplete with a small desk. It remembers patterns from a huge library; for one task it works only with what's on the desk.
- The four facts: next-token prediction with sampling; only the context window exists; knowledge is frozen at a cutoff; nothing in generation checks facts, it scores plausibility.
- Everything competes for the same window: system prompt, tool definitions, instructions, files, history and the answer. In the budgeting model, pasting a whole CI log (18,000 tokens) pushed the types, the test and the history out, while pasting only the failing test's output left room for everything.
- Bigger isn't free: Anthropic's docs name context rot (accuracy and recall degrade as token count grows), and Liu et al. (2023) found models did worse when the relevant information was in the middle of a long input.
- Practical rules: include the types, the test and the exact error; leave out whole logs and unrelated files; start a new session for a new task.
Learn more: How they work, just enough to use them well.
Show answer
Because it generates the most plausible continuation, not a checked one, and no: temperature changes how probability is spread, not which names the model thinks are plausible. "After fs. the model has seen readJson from fs-extra next to plenty of fs code, so it's a plausible token even though node:fs has no such function. Low temperature makes the top-scoring name more likely; if the top name is wrong, it's wrong every time. The fix is a check in the loop: the type checker against the installed declarations, the tests, and the real docs or .d.ts in the context for anything version-sensitive."
- In chapter 1's next-token model, at
T=1.0the two nonexistent names held 21.8% of the probability; atT=0.5, 8.5%; atT=1.5, 27.1%. Lower temperature means less variety, not more knowledge. - Training mixes versions: in the course's installed packages,
z.string().email()is marked@deprecated Use z.email() instead(Zod 4.6) andserver.tool(...)is@deprecated Use registerTool instead(MCP SDK 1.32.2). Older-style code often still runs, so nobody notices. - Private APIs are invisible unless they're in the context: your internal SDK or a device's REST API gets guessed in the shape of a common public API.
- Package names too: Spracklen et al. (USENIX Security 2025) found at least 5.2% of packages recommended by commercial models, and 21.7% by open-source models, didn't exist; 205,474 unique invented names.
- "Plausible, existing and correct are three different things":
fs.readFileexists but takes a callback, soJSON.parse(fs.readFile(...))is still a bug. TypeScript reports both kinds (TS2339 for a missing member, TS2554/TS2345 for the wrong shape).
Learn more: How they work, just enough to use them well and Spotting hallucinated APIs and subtle bugs.
Show answer
That a big window makes it possible to include more, not free: quality drops as the context fills, and the one constraint that matters ends up in the middle of hundreds of thousands of tokens. "I'd give it the context the change needs: the file, what it imports, what calls it, and its tests, plus the exact error. Agents that search the repository do a version of that themselves; with chat it's my job. And a new session for a new task, so old turns aren't competing for attention."
- Anthropic's docs (checked 27 October 2026) say accuracy and recall degrade as token count grows, "context rot", so curating context matters as much as the space available.
- Chapter 3's import-graph model found the relevant set for a one-file change: the types it imports (two hops, including a
Currencytype a model would otherwise guess), the helper, the caller to check and the test file. Two unrelated files were left out. - Leave out: whole logs (paste the failing test and the top frames), irrelevant open tabs (completion tools may include them), old conversation (two failed attempts are context too) and secrets and personal data, always.
- GitHub's prompt-engineering page says Copilot weights instructions at the beginning and end of a prompt more, and to start a new conversation for a new task. Claude Code's best-practices page recommends
/clearbetween unrelated tasks. - Standing context, like the test command and versions, belongs in the instruction file rather than in every prompt.
Learn more: Give the model the context it needs and How they work, just enough to use them well.
Show answer
They help most where the answer is a well-trodden pattern and checking is cheap; I'm careful, or write it myself, where correctness depends on facts the model can't see or a near-miss is expensive. "Good fits: boilerplate that follows the repo's pattern, a first draft of tests that I review, mechanical refactors with types and tests as a safety net, explaining unfamiliar code as a starting hypothesis, one-off scripts I can dry-run. Careful: auth, input handling and file paths, money and time zones, concurrency, retries, and anything using our private APIs unless I've given it the real types. The mental model is cheap to generate, expensive to verify: the value is the time saved generating minus the time spent verifying and fixing."
- Chapter 1's path-traversal model is the "almost right" case:
path.join(ROOT, name)let../../../etc/passwdout, the common fixstartsWith(ROOT)still let../uploads-old/keys.pemout, and only checkingROOT + "/"refused both. Every version passed the happy-path demo. - The DST model: adding 24 hours moved a daily 09:00 Toronto meeting to 08:00 after daylight saving ended on 1 November 2026, and passed any test written in October.
- Security studies point the same way: Pearce et al. found about 40% of 1,689 programs from 2021's Copilot vulnerable in security-relevant scenarios; Perry et al. (CCS 2023) found participants with an assistant wrote less secure code and were more likely to believe it was secure.
- Stack Overflow's 2025 survey: 66% name "almost right, but not quite" as their top frustration.
- The rating question to ask of any task: how expensive is it to verify, and how bad is an almost-right answer?
Learn more: Where they help and where they hurt.
Show answer
It depends on the task, codebase, developer and measurement, and the careful studies disagree, so I don't quote one number. "Controlled, self-contained tasks show big speed-ups: GitHub's 2022 experiment, published as a paper in February 2023, had developers write an HTTP server 55.8% faster with Copilot. But METR's July 2025 randomised trial of 16 experienced maintainers on 246 real issues in their own large repos found tasks took 19% longer with AI allowed, while the developers believed they'd been about 20% faster. METR's February 2026 follow-up with newer tools leaned towards a speed-up, but METR itself calls that data unreliable. And DORA's 2025 report found AI adoption going with higher throughput but lower delivery stability, calling AI an amplifier of an organisation's strengths and weaknesses."
- Peng, Kalliamvakou, Cihon and Demirer (arXiv, February 2023): 95 developers, one greenfield task, run by GitHub researchers; the authors didn't examine code quality.
- METR (July 2025): developers predicted 24% faster and afterwards still believed 20% faster while measured 19% slower. Self-reported speed-ups aren't evidence.
- METR (24 February 2026): estimates of 18% and 4% less time with wide intervals (−38% to +9%, −15% to +9%); developers declined to work without AI, withheld tasks and ran several agents at once, so METR is redesigning the study.
- Other points on the map: Paradis et al. (Google, 2024) about 21% less time on one enterprise task; Cui et al. 26.08% more completed tasks across 4,867 developers at three companies; DORA 2024 estimated −1.5% throughput and −7.2% stability per 25% more AI adoption.
- The senior answer ends with what you'd measure on your own team, not with a study.
Learn more: Where they help and where they hurt and Measuring whether it helps.
Show answer
The same people as for any other code: the author who opens the pull request and the reviewers who approve it. Ownership doesn't move; volume does. "So the habits have to scale: I specify done up front, ideally as a failing test; I verify with checks rather than the assistant's own claim of success, and I ask for the actual command output; and I review the diff like a colleague's, hardest on security, edge cases and tests that might assert the wrong thing. If I can't explain a line, I don't merge it. 'The AI wrote it' isn't an answer anyone accepts in an incident review."
- The developer's job is specify, review, verify, and then own: you can explain every line and you'd defend it at 2 a.m.
- The vendors say the same: GitHub's best-practices page says "you should always validate the code it suggests"; Claude Code's says that without something to run, "that makes you the test suite", and that "'All tests pass' on its own tells you very little".
- Evidence, not claims:
# pass 42 # fail 0from the branch you'll merge is evidence; "I've fixed it" is a claim. - Check the diff's shape before reading it: chapter 1's scope-check model flagged deleted tests, a new dependency that didn't exist on npm, a CI change and a reformat outside the ticket, all without reading code.
- What stays with people regardless: correctness and security of what ships, licensing and data decisions, design trade-offs, and understanding.
Learn more: The developer's job with an assistant.
const median = (xs) => xs.sort()[Math.floor(xs.length / 2)]; passes [1,2,3], [3,1,2] and [5]. Would you merge it?
Show answer
No. I'd check it against a slow, obviously correct oracle on generated inputs, which finds what the hand-picked examples share a blind spot with: two bugs in the sort and a missing case for even lengths. "sort() with no comparator sorts as strings, so [9, 10, 3] becomes [10, 3, 9] and the 'median' is 3, and it sorts in place, mutating the caller's array. It also never averages for even lengths. The fix is to copy, sort numerically and average the middle pair."
- In chapter 1's model, 500 seeded random arrays found 348 wrong answers and 357 calls that mutated the input.
- The smallest failures are the useful bug reports:
[9,10,3]→ got 3, want 9;[0,3]→ got 3, want 1.5. - The oracle is deliberately boring:
[...xs].sort((a, b) => a - b)and the middle element or the average of the two middles. ES2023'stoSorted((a, b) => a - b)copies and sorts in one call. - Hand-picked examples share your blind spots; an oracle plus generated inputs doesn't. For functions that are easy to state and easy to get subtly wrong, a property check is about ten lines.
- The trap to name: if the same assistant writes the function and its tests, a misunderstanding goes into both and they agree. Agreeing tests aren't independent tests.
Learn more: The developer's job with an assistant.
Show answer
Like a ticket for a contractor who can't ask questions until they hand in the work. "The goal in one line, the file to change, and the context it can't guess: the types it must use, the existing helpers, the callers that depend on it. Then the decisions only I can make: valid inputs and error behaviour, where rounding or ordering happens, what must not change, no new dependencies, our versions. One worked example pins details better than a paragraph. And I end with the definition of done, the test and type-check commands, and ask for the diff and their output as evidence."
- Whatever you leave out, the model fills with the most common version from its training data: a fair guess about the average codebase, a poor one about yours.
- Chapter 3's billing prompt shows each part: types (integer cents, so no floats), helper (
roundCents, so no new rounding code), caller (sopctis validated as untrusted input), rules (0–100 orRangeError, round once per line), an example (3998 × 0.85 = 3398.3 → 3398) and "done when". - Claude Code's best-practices page: name the file, describe the symptom and say what "fixed" looks like up front, and you usually save a round of corrections.
- A response that edits
invoice.tsor adds a package has broken an explicit rule, which makes it easy to reject. - Things repeated in every prompt (test command, conventions) go in the repository's instruction file instead.
Learn more: Give the model the context it needs.
Show answer
Because the model fills every gap with the most typical behaviour it has seen, and the typical behaviour is the happy path. "'Parse durations like 1h30m' leaves open whether 90m is valid, what 1.5h does, whether there are seconds, and whether bad input throws or returns NaN. So I write inputs and outputs with units, behaviour as examples including boundaries, exact error behaviour, and non-goals, and I often ask the model to list ambiguities as questions before writing anything. The examples become the test table, so the spec is also the check."
- Chapter 3's mental model: the model fills every gap with the average of the internet.
- In the spec-table model, a draft written from a one-line prompt passed 1 of 13 rows; one written against the spec passed 13 of 13.
- The dangerous row was
"1h30m15s": no crash, just 5,400,000 instead of 5,415,000, because the regex wasn't anchored. Silent wrong answers reach production; anchoring with^…$makes it fail loudly. - "Handle edge cases" isn't a spec: the edge cases and their behaviour (throw, return
null, clamp) are product decisions. Ask the model to suggest more cases; decide their behaviour yourself. - For bigger tasks, ask for a plan first and review it. Copilot CLI and Claude Code both document a plan mode for this reason; a plan is cheap to correct, a 600-line diff isn't.
Learn more: Spec first: describe behaviour and edge cases.
Show answer
Tests first, and I own them: they're a contract with a referee, and the assistant may change its side of the line, not move the line. "I write the tests from the spec, or have the assistant draft them and review them against the spec, commit them, and run them against a stub to see them fail for the right reason. Then the assistant implements with the tests read-only, iterating on the exact failures, and shows me the real output. In review I read the test diff first, because agents told to make tests pass sometimes edit or delete the test, weaken assertions, skip tests, mock the thing under test, or special-case the test inputs."
- The stub run matters: a test that passes against a stub that throws "not implemented" is testing nothing. In chapter 3's slugify loop, the stub scored 0/5, the first attempt 1/5 (its failures showed
"café-déjà-vu","a----b!!c"…), and the second 5/5 with the test file unchanged. - Split it into two prompts: tests only (with a stub), then implementation against approved, committed tests, with "if a test looks wrong, stop and tell me".
- Tests written after the code tend to assert what the code does, including its bugs. That's why they come first and are reviewed against the spec.
- Chapter 8's weakening check showed why counting assertions isn't enough: 3 removed and 3 added, every replacement weaker (
toBe(1)→toBeGreaterThan(0),toThrow("…")→toThrow(), atest.skip). - Green isn't the same as right: a passing suite proves the code matches the tests, no more.
Learn more: Tests first with an assistant and Reviewing AI-generated changes.
Show answer
Evidence, not adjectives: the failing input, expected and actual values, the exact error output, and the rule it broke. "'It doesn't work' gives the model nothing new, so it changes something at random. I narrow each turn to one problem and say what not to touch, and I ask it to explain the cause in one sentence before changing code, which tells me quickly whether it understood. If the approach is wrong, I ask for two or three alternatives with trade-offs before any code. After two failed corrections I start a fresh session with a better prompt, and if the fix is smaller than the explanation, I just write it."
- Chapter 3's mental model: a bug report to a colleague in another time zone, written so they can fix it without asking anything.
- In the error-to-prompt model, a double-booking check keyed by
room@startmissed a partial overlap; the correction prompt carried the input, expected, actual, Node's assertion diff, the rule ("same room and overlapping ranges") and the boundary decision (end is exclusive). - A rule you state in a correction should become a test, or it'll be broken again two iterations later.
- Restart rather than pile on: Claude Code's page says a third correction "mostly adds to a conversation that already contains two failed attempts". Paste what you learned, not the history.
- Stop and take over when the same mistake returns, when you no longer understand the diff, or when the problem is a decision, not code.
Learn more: Iterating and correcting.
bookings.forEach(async (b) => { await notify(b); }); await markBatchSent(batchId); All functions exist and the types check.
Show answer
markBatchSent runs before any notification has finished, because forEach ignores the promises its async callbacks return. "Each callback starts notify(b) and returns a pending promise straight away; forEach drops them and returns, so the batch is marked sent while the calls are in flight, and a failure becomes, at best, an unhandled rejection. I'd write await Promise.all(bookings.map((b) => notify(b))), or a for…of loop with await if they must go one at a time."
Array.prototype.forEachdiscards its callback's return value, so theawaitinside only pauses that callback.- Chapter 3's model printed
draft: done, sent 0, and 50 ms latersent = 3: nothing errors and the reminders still go out, so the bug hides. Promise.allwaits for all of them and rejects if any fails, so the caller can handle it before marking the batch.- No type checker flags this: every API is real and every type lines up. It's caught by review knowledge and by a test asserting the batch isn't marked sent when one notification fails.
- The hot-spot list to review in generated code: comparator-less
sort(), async inforEach, dates and time zones, money and floats, unanchored regexes, paths, SQL and shell concatenation, swallowed errors, boundaries, mutation, old API versions and new dependencies.
Learn more: Spotting hallucinated APIs and subtle bugs.
Show answer
That it exists, is the package we meant, is maintained and has an acceptable licence, before anyone runs npm install. "Assistants invent plausible package names, and if a name gets suggested repeatedly, someone can register it with a malicious install script; that's slopsquatting. So a 404 from the registry is a block, a package created three days ago with a dozen downloads a week and one letter off a popular name is a human check, and a copyleft licence is a question for legal, not a lint rule. I'd run that check in CI on every PR that changes package.json."
- Spracklen et al. (USENIX Security 2025): at least 5.2% of commercial models' and 21.7% of open-source models' recommended packages didn't exist, across 576,000 samples from 16 models.
- Chapter 8's dependency check classified four new names:
ok(MIT, old, widely used),ASK(GPL-3.0-only isn't on the allowlist),CHECK(new, little used, typo of a popular name),BLOCK(not on the registry). - Chapter 3's name checker went further: it imported every package a snippet named from the real
node_modulesand foundfs.readJsonandserver.addTooldon't exist, andserver.toolis deprecated in SDK 1.32.2. - Chapter 1's scope check would flag the new dependency before anyone read code: it wasn't in the ticket.
- Never install a package because a suggestion named it; prefer packages the team already uses.
Learn more: Spotting hallucinated APIs and subtle bugs and Security, licensing and data concerns.
Show answer
Short, organised around data and responsibility rather than products, because products change monthly. "Which tools: company seats only, plus MCP servers we've reviewed, with a quick path to get a new one reviewed. Which data: code and internal docs are fine with approved tools; customer logs, credentials and keys never go in, and we give people a one-command redaction script so the safe path is easy. Which work: anything, but device commands, auth and migrations get a second reviewer, whoever typed them. Ownership: whoever opens the PR owns every line and says what they verified. I'd write it down in the repo, give it an owner, and revisit it quarterly."
- Chapter 8's mental model is a traffic-light map keyed to data, not to tools: green for approved tools on company seats, amber for risky code areas, red for restricted data, whatever the tool.
- DORA's AI Capabilities Model (Google Cloud, 23 September 2025) lists a "clear and communicated AI stance" first among seven capabilities that amplify AI's benefits; it measures that people know the stance, not what it says.
- The policy-as-data model answered seven requests: an approved tool on a personal account was denied ("different data terms"), an unknown MCP server was denied with a way forward ("ask in #ai-tools").
- A ban moves the risk instead of removing it: people keep using assistants, on personal accounts, with nobody reviewing what goes in. Strict and clear beats vague and forbidden.
- Add a section for agents and MCP servers: where agents may run, which commands auto-approve, no production secrets in workspaces where agents run, approved servers only.
Learn more: Setting team guidelines for AI-assisted work.
Show answer
Visibility, routing by risk and the agent-specific rules; how to read the diff is the same checklist as for any PR. "The PR template asks how much was generated and what the author verified. Routing is by what the change touches, so device commands, auth and migrations get a second reviewer whoever wrote them. Size limits don't grow because an agent can produce a big diff. For Copilot cloud agent PRs, GitHub doesn't count the requester's approval and workflows don't run until someone approves them. And instruction files and MCP config are owned by the tech leads, because they change what every assistant on the team does."
- The mental model is air traffic control, not a second pilot: label each change, route it to someone qualified for its risk, and hold it until that kind of change's checks have passed.
- Chapter 8's routing model never used "was this AI?" to decide risk; it used AI involvement only for reviewer notes: missing "what I verified", tests that shrank, and the agent-approval rule.
- Copilot code review reads instructions from the PR's head branch, so a PR can change the rules its own AI review follows; CODEOWNERS on those files means a lead must approve any such change before it merges, though the AI review of that same PR still reads the edited version.
- Volume is the lead's problem: DORA's 2025 report (as summarised by InfoQ) warns AI can raise code volume without improving delivery. Hold the size limit, count review as work, automate the mechanical checks, and spread review owners across both sites.
- AI review is a first pass: its comments are suggestions; a human who owns the area approves.
Learn more: Reviewing AI-generated changes.
Show answer
Three separate questions: what goes out, what comes in, and what the tools can do. "Out: company seats so business data terms apply, content exclusion for sensitive paths, and no customer logs or credentials in any tool, with a redaction script that keeps placeholders consistent so the result is still useful. In: generated code gets the normal security review, new dependencies are checked for licence, existence and age, and we choose deliberately whether to block or reference suggestions that match public code. Tools: agents run with the developer's permissions and MCP servers with their own, so approved, pinned servers, least-privilege tokens, and a human confirmation for anything that writes to a device."
- The mental model is customs at a border: declare what leaves, inspect what arrives, and give the vehicles crossing (agents, MCP servers) passes for one trip.
- The most likely real incident is pasting a support log into a chat. Chapter 8's redaction model removed a password and a token first, then a meeting title by knowing the log's format ("Acme acquisition - board only"), then IPs, emails and phones with consistent placeholders (
<ip-1>twice), and flagged one free-text line for a human. .gitignorehides files from git and reviewers, not from an agent: the model showed.env, a private key and a call capture all readable in the folder. Chapter 2 adds that content exclusion doesn't apply to agent mode.- Copilot's code referencing checks suggestions plus about 150 characters of context against public GitHub code; matches are blocked or shown with the repository and licence, depending on policy.
- Device-platform data to name in the guideline: meeting metadata, call content, network details, device credentials, partner SDKs.
Learn more: Security, licensing and data concerns.
A comparison team that didn't change tools went from 18 to 15 hours over the same weeks. The pilot team's reverts went from 3 to 7 a month; the comparison team's stayed at 3.
Show answer
I can't say it's working yet: most of the improvement also happened to the team without the tool, and the pilot's reverts more than doubled. "The pilot's own before/after is −33%, which is what a launch slide would say. The comparison team improved 17% with no new tools, so the part we might attribute to the pilot is about 17 points, and with a few dozen PRs per period that's likely within noise. I'd report it that way, keep the pilot running, look at which PRs were reverted and why, and run the developer survey."
// "Is the Copilot pilot working?" Before/after vs a comparison team (difference-in-differences).
// The numbers are made up for this card: median hours from PR open to merge, and reverts per month.
const teams = {
pilot: { before: 18, after: 12, revertsBefore: 3, revertsAfter: 7 },
comparison: { before: 18, after: 15, revertsBefore: 3, revertsAfter: 3 },
};
const pct = (a, b) => ((b - a) / a) * 100;
for (const [name, t] of Object.entries(teams)) {
console.log(`${name.padEnd(10)} median ${t.before}h → ${t.after}h (${pct(t.before, t.after).toFixed(0)}%), reverts ${t.revertsBefore} → ${t.revertsAfter}`);
}
const own = pct(teams.pilot.before, teams.pilot.after);
const did = own - pct(teams.comparison.before, teams.comparison.after);
console.log(`pilot's own before/after: ${own.toFixed(0)}%`);
console.log(`difference-in-differences: ${did.toFixed(0)} points (the part the comparison team didn't also get)`);
console.log(`counter-metric: reverts ${teams.pilot.revertsAfter / teams.pilot.revertsBefore > 1 ? "rose" : "held"} for the pilot only`);pilot median 18h → 12h (-33%), reverts 3 → 7
comparison median 18h → 15h (-17%), reverts 3 → 3
pilot's own before/after: -33%
difference-in-differences: -17 points (the part the comparison team didn't also get)
counter-metric: reverts rose for the pilot onlypctgives each team's change in median merge time:(12 − 18) / 18 = −33%for the pilot and(15 − 18) / 18 = −17%for the comparison team.- The difference-in-differences subtracts what happened anyway (a release freeze ending, smaller tickets, the season) from the pilot's change:
−33.3 − (−16.7) ≈ −17points (the code subtracts the unrounded values). - Chapter 8's simulation is why "within noise" is the honest word: with 40 PRs per team per period, a real 15% speed-up was detected only 22% of the time, so "no significant difference" is the expected result even when the tool helps.
- The counter-metric is the reverts line: 3 to 7 for the pilot only. Faster merges with more reverts may mean lighter review, not better work.
- The layers to keep apart: adoption (seats used), activity (acceptance rate, "AI-written lines") and outcomes (lead time, change failure rate, survey). Only outcomes answer the director's question, and never make an activity metric someone's target.
Learn more: Measuring whether it helps.
Show answer
Treat the team's AI setup as code in the repository, shared by both sites, and provision everyone at the same time. "One set of instruction files with the build and test commands and our 'never' rules, prompt files for jobs we repeat, like adding an endpoint or writing the end-of-day handoff, and an approved, pinned list of MCP servers, including a team-hosted device-lab server where each person signs in as themselves. Tech leads at both sites own those files, and a CI check catches contradictions, literal secrets and unpinned servers. Then a short monthly session alternating which site presents what worked and what didn't."
- The setup can shrink the time-zone gap (same rules, same prompts, same tools) or widen it (private setups, and reviews across the overlap hour become arguments about style the assistants introduced).
- Chapter 8's CI check found nine problems in a messy repository, the first being
npm testin one instruction file andpnpm testinAGENTS.md: two sites' assistants verifying differently. - Unpinned servers bite at random:
@team/lab-mcp@latestupdated under everyone, and whoever restarted first saw the breakage. - Shared setup, personal credentials: a shared service account makes the audit log say "service-account rebooted board-1".
- Agents don't remove the gap, they move it to review: an overnight agent PR needs an owner at the receiving site, a spec with acceptance criteria (nobody is awake to answer its questions) and the normal size limit.
Learn more: Helping a distributed team use AI tools well.
Show answer
One specific time from your own work, AI-related or not, told as STAR with a learning; this card gives only the shape, because the story has to be yours. "I'd open with a one-line headline, then the situation: what was going wrong without guidance. Then what I did, with a 'because' for each step: how I found out how people actually worked, what I proposed, how I got both sites to agree, how I made the compliant path the easy one. Then what I measured afterwards, and what I changed when part of it didn't work." If you haven't led an AI rollout, say so, walk through how you'd do it, and use a practice you did introduce as the evidence.
- What the interviewer listens for: whether you started from how people really worked, how you got agreement rather than imposing a rule, whether "yes" was made cheap, and whether you measured and adjusted.
- Chapter 8's own note on this question: answer from something you actually did; don't borrow a story.
- Find candidates with the Tech Lead site's story prompts and write them in your notes; rehearse a two-minute version and a 30-second one.
- If the practice was AI-related, be precise about which tool you used (for example Claude Code rather than Copilot) and what you verified.
- An honest adjacent story scores far above an invented one, and invented ones rarely survive the third follow-up.
Learn more: Setting team guidelines for AI-assisted work, the Tech Lead site's Behavioural interviews chapter, and your own stories in My notes.