AI in the Team

# Using AI assistants on the device admin team

Owner: tech leads (Austin + Bengaluru). Reviewed every quarter; last review 2026-10-27.

## Tools
- Use Copilot through your company seat only. Personal plans have different data terms.
- MCP servers: only the ones in `.vscode/mcp.json` in this repository. Want another? Open an issue
  with what it can read and do; a tech lead reviews it within two working days.

## Data
- Never paste customer logs, call recordings, credentials or keys into any AI tool.
  Run `npm run redact -- <file>` first and check the output.
- Device lab data is fine with approved tools.

## Work
- You own every line you merge, including what an agent wrote. Be able to explain it.
- Device commands, auth, migrations and release config: second reviewer, AI-assisted label.
- Keep PRs under 400 changed lines. Agents make big diffs easy; reviewers' attention doesn't scale.

## Sharing
- Prompts and instructions that work go in `.github/prompts/` and `.github/copilot-instructions.md`,
  through a pull request.
- Bring one thing that worked and one that didn't to the monthly AI session.

// Routing pull requests by risk, not by who (or what) typed the code. ILLUSTRATIVE rules and PRs.
const RISKY = [
  [/^server\/src\/devices\/commands/, "device commands (reboot, factory reset)", "@api-leads"],
  [/^server\/src\/auth\//, "authentication", "@security"],
  [/^server\/migrations\//, "database migration", "@api-leads"],
  [/^(\.github\/(copilot-instructions\.md|instructions\/|prompts\/)|AGENTS\.md|\.vscode\/mcp\.json)/, "AI instructions or MCP config", "@tech-leads"],
];
const MAX_LINES = 400;

function route(pr) {
  const notes = [];
  const owners = new Set();
  let reviewers = 1;
  for (const f of pr.files) for (const [re, what, team] of RISKY) if (re.test(f.path)) { owners.add(team); notes.push(`touches ${what}`); }
  if (owners.size) reviewers = 2;
  const lines = pr.files.reduce((n, f) => n + f.added + f.deleted, 0);
  if (lines > MAX_LINES) notes.push(`${lines} lines changed (limit ${MAX_LINES}): ask for a split`);
  const src = pr.files.filter((f) => f.path.includes("/src/"));
  const tests = pr.files.filter((f) => f.path.includes("/test/"));
  if (src.length && !tests.length) notes.push("source changed, no test changed");
  if (tests.some((f) => f.deleted > f.added)) notes.push("tests shrank: check nothing was deleted to make them pass");
  if (pr.author === "copilot-cloud-agent") notes.push(`agent PR: ${pr.requestedBy}'s approval doesn't count; another reviewer must approve`);
  if (pr.aiAssisted && !pr.verificationNote) notes.push('AI-assisted but no "what I verified" note in the description');
  return { reviewers, owners: [...owners], notes };
}

const prs = [
  { id: 101, author: "priya", aiAssisted: true, verificationNote: true,
    files: [{ path: "web/src/rooms/RoomCard.tsx", added: 40, deleted: 12 }, { path: "web/test/RoomCard.test.tsx", added: 30, deleted: 0 }] },
  { id: 102, author: "copilot-cloud-agent", requestedBy: "sam", aiAssisted: true, verificationNote: false,
    files: [{ path: "server/src/devices/commands.ts", added: 60, deleted: 5 }, { path: "server/test/commands.test.ts", added: 4, deleted: 22 }] },
  { id: 103, author: "arjun", aiAssisted: true, verificationNote: true,
    files: [{ path: "server/src/rooms/list.ts", added: 380, deleted: 90 }] },
  { id: 104, author: "dana", aiAssisted: false,
    files: [{ path: ".github/copilot-instructions.md", added: 2, deleted: 9 }] },
];
for (const pr of prs) {
  const r = route(pr);
  console.log(`#${pr.id} by ${pr.author}${pr.aiAssisted ? " (AI-assisted)" : ""}: ${r.reviewers} reviewer(s)${r.owners.length ? `, owners ${r.owners.join(" ")}` : ""}`);
  for (const n of r.notes) console.log(`  - ${n}`);
}

#101 by priya (AI-assisted): 1 reviewer(s)
#102 by copilot-cloud-agent (AI-assisted): 2 reviewer(s), owners @api-leads
  - touches device commands (reboot, factory reset)
  - tests shrank: check nothing was deleted to make them pass
  - agent PR: sam's approval doesn't count; another reviewer must approve
  - AI-assisted but no "what I verified" note in the description
#103 by arjun (AI-assisted): 1 reviewer(s)
  - 470 lines changed (limit 400): ask for a split
  - source changed, no test changed
#104 by dana: 2 reviewer(s), owners @tech-leads
  - touches AI instructions or MCP config

// Spotting tests that were weakened to make a change pass. Reads a unified diff of test files.
// The diff is MADE UP, in the shape `git diff main...branch -- test/` prints.
const diff = `diff --git a/test/commands.test.ts b/test/commands.test.ts
--- a/test/commands.test.ts
+++ b/test/commands.test.ts
@@ -10,14 +10,12 @@
 test("reboot is sent once", async () => {
-  expect(device.reboots).toBe(1);
+  expect(device.reboots).toBeGreaterThan(0);
 });
-test("400 is not retried", async () => {
+test.skip("400 is not retried", async () => {
   await expect(send(badRequest)).rejects.toThrow("answered 400");
-  expect(fetchFn).toHaveBeenCalledTimes(1);
 });
 test("timeout error names the device", async () => {
-  await expect(send(slow)).rejects.toThrow("huddle-2 did not answer");
+  await expect(send(slow)).rejects.toThrow();
 });
+test("sends a command", async () => {
+  expect(await send(ok)).toBeDefined();
+});`;

const removed = diff.split("\n").filter((l) => l.startsWith("-") && !l.startsWith("---"));
const added = diff.split("\n").filter((l) => l.startsWith("+") && !l.startsWith("+++"));
const count = (lines, re) => lines.filter((l) => re.test(l)).length;
const findings = [];
const assertions = /expect\(|assert\./;
console.log(`assertion lines: ${count(removed, assertions)} removed, ${count(added, assertions)} added`);
for (const l of added) {
  if (/\.(skip|todo)\(|\bxit\(|\bxdescribe\(/.test(l)) findings.push(`test disabled: ${l.slice(1).trim()}`);
  if (/toBeGreaterThan\(0\)|toBeDefined\(\)|toBeTruthy\(\)|\.toThrow\(\)/.test(l)) findings.push(`loose assertion: ${l.slice(1).trim()}`);
}
for (const l of removed) {
  const exact = l.match(/\.(toBe|toEqual|toHaveBeenCalledTimes)\((.+)\);/);
  if (exact) findings.push(`exact check removed: ${l.slice(1).trim()}`);
}
console.log(findings.length ? findings.map((f) => `! ${f}`).join("\n") : "no weakening found");

assertion lines: 3 removed, 3 added
! loose assertion: expect(device.reboots).toBeGreaterThan(0);
! test disabled: test.skip("400 is not retried", async () => {
! loose assertion: await expect(send(slow)).rejects.toThrow();
! loose assertion: expect(await send(ok)).toBeDefined();
! exact check removed: expect(device.reboots).toBe(1);
! exact check removed: expect(fetchFn).toHaveBeenCalledTimes(1);

// New dependencies in an AI-assisted pull request, checked against a licence allowlist and the registry.
// The registry answers are a FAKE snapshot: every package name and figure here is made up, so the output never changes.
const today = Date.parse("2026-10-27");
const registry = { // name → what `npm view <name> license time.created` would tell you
  "room-time-utils": { license: "MIT", created: "2019-03-11", weeklyDownloads: 250_000 },
  "device-sse-helpers": { license: "GPL-3.0-only", created: "2023-05-02", weeklyDownloads: 4_100 },
  "expres-validator-utils": { license: "MIT", created: "2026-10-24", weeklyDownloads: 12 },
};
const ALLOWED = new Set(["MIT", "Apache-2.0", "BSD-2-Clause", "BSD-3-Clause", "ISC"]);

const before = { "express": "^5.3.0", "zod": "^4.6.0" };
const after = { "express": "^5.3.0", "zod": "^4.6.0", "room-time-utils": "^3.1.0", "device-sse-helpers": "^2.0.1",
  "expres-validator-utils": "^1.0.0", "express-rate-limiter-pro": "^3.2.0" };

for (const name of Object.keys(after).filter((n) => !(n in before))) {
  const meta = registry[name];
  let verdict;
  if (!meta) verdict = "BLOCK  not on the registry: a hallucinated name, and a squatter could register it tomorrow";
  else if (!ALLOWED.has(meta.license)) verdict = `ASK    licence ${meta.license} is not on the allowlist: needs a decision from legal`;
  else if ((today - Date.parse(meta.created)) / 86_400_000 < 30 || meta.weeklyDownloads < 1000)
    verdict = `CHECK  created ${meta.created}, ${meta.weeklyDownloads} downloads/week: new and little used (typo of a popular name?)`;
  else verdict = `ok     ${meta.license}`;
  console.log(`${name.padEnd(26)} ${verdict}`);
}

room-time-utils            ok     MIT
device-sse-helpers         ASK    licence GPL-3.0-only is not on the allowlist: needs a decision from legal
expres-validator-utils     CHECK  created 2026-10-24, 12 downloads/week: new and little used (typo of a popular name?)
express-rate-limiter-pro   BLOCK  not on the registry: a hallucinated name, and a squatter could register it tomorrow

// What an agent running in your project folder can read, versus what git (and so a reviewer) sees.
// Builds a throwaway project in a temp folder; the "secrets" are fake.
import { execFileSync } from "node:child_process";
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, readdirSync, statSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join, relative } from "node:path";

const dir = mkdtempSync(join(tmpdir(), "agent-sees-"));
const write = (p, text) => { mkdirSync(join(dir, p, ".."), { recursive: true }); writeFileSync(join(dir, p), text); };
write(".gitignore", ".env\n*.pem\ncaptures/\n");
write("src/index.js", "export const port = 3000;\n");
write(".env", "DEVICE_API_TOKEN=lab-9f2c41d7\nCLOUD_CLIENT_SECRET=fake-secret\n");
write("certs/device-ca.pem", "-----BEGIN PRIVATE KEY-----\nfake\n-----END PRIVATE KEY-----\n");
write("captures/huddle-2-call-4411.log", "09:03:05 call 4411 joined: title='Acme acquisition'\n");
execFileSync("git", ["init", "-q"], { cwd: dir });

const tracked = new Set(execFileSync("git", ["ls-files", "--others", "--cached", "--exclude-standard"], { cwd: dir, encoding: "utf8" }).trim().split("\n"));
const walk = (d) => readdirSync(d).flatMap((n) => { const p = join(d, n); if (n === ".git") return []; return statSync(p).isDirectory() ? walk(p) : [relative(dir, p)]; });
const looksSecret = /(TOKEN|SECRET|PASSWORD)=|PRIVATE KEY|title='/;
for (const f of walk(dir).sort()) {
  const text = readFileSync(join(dir, f), "utf8"); // what any file tool or shell in this folder can do
  console.log(`${f.padEnd(32)} git: ${tracked.has(f) ? "visible" : "ignored"}   agent: readable${looksSecret.test(text) ? "   ← sensitive" : ""}`);
}
rmSync(dir, { recursive: true, force: true });

.env                             git: ignored   agent: readable   ← sensitive
.gitignore                       git: visible   agent: readable
captures/huddle-2-call-4411.log  git: ignored   agent: readable   ← sensitive
certs/device-ca.pem              git: ignored   agent: readable   ← sensitive
src/index.js                     git: visible   agent: readable

// Summarising a short monthly developer survey per site, without over-reading it. Responses are MADE UP.
// Each answer is 1–5 (1 = strongly disagree, 5 = strongly agree).
const questions = {
  toil: "AI tools reduced the time I spend on repetitive work",
  confidence: "I'm confident in the AI-assisted code I merged this month",
  redo: "I often had to redo what the assistant produced",
};
const responses = [
  ...[[4, 4, 2], [5, 3, 3], [4, 4, 2], [3, 3, 3], [5, 4, 2], [4, 2, 4], [4, 3, 3]].map((a) => ({ site: "Austin", a })),
  ...[[5, 2, 4], [4, 2, 4], [5, 3, 4], [4, 2, 5]].map((a) => ({ site: "Bengaluru", a })),
];
const median = (xs) => { const s = [...xs].sort((a, b) => a - b); const m = s.length >> 1; return s.length % 2 ? s[m] : (s[m - 1] + s[m]) / 2; };
const agree = (xs) => `${Math.round((xs.filter((x) => x >= 4).length / xs.length) * 100)}% agree`;

for (const site of ["Austin", "Bengaluru", "all"]) {
  const rs = responses.filter((r) => site === "all" || r.site === site);
  const line = Object.keys(questions).map((q, i) => { const xs = rs.map((r) => r.a[i]); return `${q} ${median(xs)} (${agree(xs)})`; });
  console.log(`${site.padEnd(10)} n=${String(rs.length).padStart(2)}  ${line.join("  ")}${rs.length < 5 ? "  ← too few to report alone" : ""}`);
}

Austin     n= 7  toil 4 (86% agree)  confidence 3 (43% agree)  redo 3 (14% agree)
Bengaluru  n= 4  toil 4.5 (100% agree)  confidence 2 (0% agree)  redo 4 (100% agree)  ← too few to report alone
all        n=11  toil 4 (91% agree)  confidence 3 (27% agree)  redo 3 (45% agree)

// How many PRs does a pilot need before a real 15% speed-up shows up reliably? Simulated, seeded (mulberry32).
// Assumption (ILLUSTRATIVE): merge times are log-normal with the spread of the pilot-did model's data.
function mulberry32(a) { return () => { a |= 0; a = (a + 0x6d2b79f5) | 0; let t = Math.imul(a ^ (a >>> 15), 1 | a);
  t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; return ((t ^ (t >>> 14)) >>> 0) / 4294967296; }; }
const rand = mulberry32(7);
const SPREAD = 1.6; // same spread as pilot-did.mjs: hours = median × e^((u − 0.5) × 1.6), u uniform
const sample = (n, median) => Array.from({ length: n }, () => Math.log(median) + (rand() - 0.5) * SPREAD); // log-hours
const mean = (xs) => xs.reduce((a, b) => a + b, 0) / xs.length;
const varOf = (xs) => { const m = mean(xs); return xs.reduce((a, x) => a + (x - m) ** 2, 0) / (xs.length - 1); };

function onePilot(n, trueEffect) { // both teams speed up 25% (the freeze ends); the pilot gets trueEffect on top
  const groups = [sample(n, 20), sample(n, 20 * 0.75 * (1 - trueEffect)), sample(n, 20), sample(n, 20 * 0.75)];
  const [pb, pa, cb, ca] = groups.map(mean);
  const did = (pa - pb) - (ca - cb); // difference-in-differences on log-hours
  const se = Math.sqrt(groups.reduce((a, g) => a + varOf(g) / g.length, 0));
  return did + 1.96 * se < 0; // the 95% interval is entirely below zero: "the pilot got faster"
}
console.log("PRs per team per period | pilots that detect a real 15% speed-up | ...a real 0% (false alarms)");
for (const n of [20, 40, 80, 160, 320]) {
  const runs = 2000;
  let hit = 0, alarm = 0;
  for (let i = 0; i < runs; i++) { if (onePilot(n, 0.15)) hit++; if (onePilot(n, 0)) alarm++; }
  console.log(`${String(n).padStart(23)} | ${`${((hit / runs) * 100).toFixed(0)}%`.padStart(39)} | ${`${((alarm / runs) * 100).toFixed(1)}%`.padStart(27)}`);
}

PRs per team per period | pilots that detect a real 15% speed-up | ...a real 0% (false alarms)
                     20 |                                     13% |                        2.8%
                     40 |                                     22% |                        2.5%
                     80 |                                     34% |                        2.9%
                    160 |                                     62% |                        2.6%
                    320 |                                     87% |                        2.7%

# AI-assisted development: <month>

## Adoption (context, not success)
Active users: <n> of <seats> (Austin <n>, Bengaluru <n>). Agent PRs opened: <n>.

## Outcomes, with a comparison
| Metric | Pilot teams | Comparison teams | Note |
|---|---|---|---|
| PR open → merge, median / p90 | | | |
| Review time, median | | | |
| Reverts per 100 PRs | | | counter-metric |
| Change failure rate | | | counter-metric |

## What people said (survey, n = <n>)
Toil: <…>. Confidence in merged code: <…>. Top frustration: <…>.

## What we can't conclude yet
<e.g. "40 PRs per period can't show a 15% effect; see the pilot-size estimate".>

## Changes we're making
<one or two, each with an owner>

---
description: "Draft today's handoff note for the other site: what changed, what's blocked, what needs a decision"
name: handoff
agent: ask
tools: ['search/codebase', 'read/readFile']
---
Draft a handoff note for the team in the other time zone, from my work today on ${input:branch:current branch}.

Use only what you can see in the repository and the files I attach. Don't guess at status.

Sections, in this order, each at most five bullets:
1. Done: merged or ready for review, with PR links.
2. In progress: what state it's in and the next concrete step.
3. Blocked or needs a decision: the question, the options, and who can answer.
4. Watch out: anything that could break, such as a flaky test, a device in the lab left in a strange state, or a migration not yet run.

Plain, literal English. No idioms, no "ASAP": give dates and times in both IST and US Central.