Nine skills. Every rule sourced.

• 01

  ai-agent-engineering

  Workflow before agent, tools as interfaces, context as a budget, evaluator separate from generator.

  15 rules
• 02

  llm-evals

  Evals that catch real regressions, from 20 real-failure tasks to graded CI gates and production quality checks.

  15 rules
• 03

  system-design-scaling

  Capacity math first, then the cheapest fix for the component that breaks first, with sourced case studies.

  15 rules
• 04

  reliability-incident-response

  Mitigate first with the smallest reversible step, treat config as code, and close incidents with owned, triggered action items.

  15 rules
• 05

  performance-engineering

  Measure the tail, profile, fix the biggest share first, re-measure, then guard it in CI. Backend, build and LLM inference.

  15 rules
• 06

  data-ml-platforms

  Data contracts, online/offline parity, orchestration and backfill, experiment metrics and pipeline monitoring, from Zepto, Netflix, Uber and Airbnb posts.

  15 rules
• 07

  dev-productivity-ai-coding

  Agentic coding that ships: smallest change first, a verifier before scale, review and CI that keep up with agent volume.

  15 rules
• 08

  security-engineering

  Contain agents with sandboxes, outside-the-box credentials, scoped tools and tiered approval, plus sourced app-security lessons.

  15 rules
• 09

  frontend-mobile-engineering

  Budgets, safe releases, server-driven UI and peak-proof clients, from Airbnb, Uber, Discord and Zepto.

  15 rules

1. ai-agent-engineering 01

   Instead of reaching for an autonomous agent or a framework first

   Do try one LLM call, then a fixed workflow (chain, route, parallelize, orchestrator-workers, evaluator-optimizer), and grant agent autonomy only where it measurably beats that baseline

   Anthropic says "find the simplest solution possible" and OpenAI says maximize a single agent before splitting. S1 S17 S24

2. ai-agent-engineering 02

   Instead of adding subagents because the problem feels big

   Do stay single-agent unless the work is breadth-first, parallelizable and valuable enough to pay for the tokens

   agents use about 4x chat tokens and multi-agent about 15x, while Opus 4 lead plus Sonnet 4 subagents beat single-agent Opus 4 by 90.2% only on parallel research. S4 S23

3. ai-agent-engineering 03

   Instead of one-line tool descriptions written for humans

   Do design the tool interface like an HCI: namespaced names, clear parameters, 1-5 examples for ambiguous schemas, and poka-yoke inputs such as absolute paths

   tool use examples raised complex parameter accuracy from 72% to 90%, and absolute paths fixed edit failures after the agent changed directory. S1 S10 S5

4. ai-agent-engineering 04

   Instead of returning raw API payloads from tools

   Do cap, paginate and filter output and expose a response_format enum (concise or detailed)

   Claude Code caps tool responses at 25,000 tokens by default, and the Slack example returned 72 tokens concise vs 206 detailed. S5

5. ai-agent-engineering 05

   Instead of loading every tool definition upfront

   Do keep the 3-5 most used tools loaded and defer the rest behind tool search once definitions pass 10K tokens or 10 tools

   tool search cut definitions from about 77K to 8.7K tokens and lifted Opus 4 on MCP evals from 49% to 74%; tool choice degrades past 30-50 definitions. S10 S21

6. ai-agent-engineering 06

   Instead of routing intermediate data through the context window

   Do let the agent call tools from code in a sandbox and return only filtered results

   a Drive-to-Salesforce task fell from 150,000 to 2,000 tokens (98.7%), and Uber reports code mode cut tokens 55-71% on small SQL tasks. S9 S20

7. ai-agent-engineering 07

   Instead of filling the window and hoping recall holds

   Do budget context: just-in-time retrieval by file path or ID, clear stale tool results, compact near the limit, keep notes in memory outside the window

   recall degrades as tokens grow, and context editing plus a memory tool improved an agentic eval 39% over baseline while cutting tokens 84% on a 100-turn search. S6 S25

8. ai-agent-engineering 08

   Instead of letting subagents hand back everything they read

   Do give each one an objective, output format, tool guidance and boundaries, and demand a condensed summary

   subagents may burn tens of thousands of tokens but should return about 1,000-2,000, and work producing over 1,000 mostly irrelevant tokens belongs in isolation. S4 S6 S23

9. ai-agent-engineering 09

   Instead of standing up a vector index on day one

   Do start with agentic search over files, skip RAG under 200,000 tokens of knowledge, and when you need RAG add chunk context, BM25 and reranking

   contextual embeddings plus contextual BM25 plus reranking cut top-20 retrieval failures 67% (5.7% to 1.9%). S7 S14

10. ai-agent-engineering 10

    Instead of bolting on a dedicated think tool by default

    Do enable extended thinking and keep a think tool only for policy-heavy sequential tool chains where you measured a gain

    a 2025-12-15 update prefers extended thinking in most cases, and the original airline gain is reported as 0.370 to 0.570 in the text but 0.332 to 0.584 in the table. S3

11. ai-agent-engineering 11

    Instead of one endless session for multi-hour work

    Do split an initializer from a worker role, persist a progress file, a JSON feature list where agents only flip passes, git commits per feature, and a start ritual that runs an end-to-end test

    Anthropic's harness for a 200+ feature app used exactly this so each fresh session recovers state without trusting memory. S11

12. ai-agent-engineering 12

    Instead of asking the generator to critique its own output

    Do run a separate skeptical evaluator in fresh context against criteria agreed before the work, with a hard iteration cap

    tuning a standalone evaluator to be skeptical was "far more tractable" than self-critique, and a $9 solo run shipped a broken core feature that the $200 harness caught. S12 S22 S24

13. ai-agent-engineering 13

    Instead of trusting the model to refuse bad actions

    Do keep credentials out of generated code and put every tool behind one gateway with tool-level authz, PII redaction and owner approval for new tools

    prompt-injection success rose from about 0.1% on one try to 5-6% after 100 adaptive tries, and Uber reports its MCP Gateway fronts 800+ servers and 5,000+ tools this way. S29 S13 S19

14. ai-agent-engineering 14

    Instead of shipping when the demo feels 80% there

    Do start an eval of real cases before launch (10 golden examples is enough to begin), run it more than once, and budget the last mile explicitly

    LinkedIn reports 80% of the experience in the first month and four more months to pass 95%, and one regression showed in only one run of two. S18 S21

15. ai-agent-engineering 15

    Instead of running every step on the biggest model

    Do route by difficulty or run a cheap executor that consults a stronger advisor only on hard calls, capped with max_uses

    Haiku with an Opus advisor scored 41.2% on BrowseComp vs 19.7% alone, still cheaper than Sonnet, and Sonnet with an advisor gained 2.7 points at 11.9% lower cost per task. S27 S1

16. llm-evals 01

    Instead of waiting for a large benchmark before evaluating anything

    Do start with 20-50 simple tasks taken from real failures: bug reports, support tickets, incident and revert PRs

    Anthropic calls 20-50 real-failure tasks "a great start" and its code-review team turns past incident and revert PRs into a daily eval S1 S11 S12

17. llm-evals 02

    Instead of reading a 0% task as "the model cannot do this"

    Do write a reference solution that passes the grader, then audit the task, grader and environment before blaming the model

    0% pass@100 usually means a broken task or grader, and SWE-bench work found setup issues and patches applied twice S1 S13

18. llm-evals 03

    Instead of reaching for an LLM judge first

    Do grade with code wherever possible (tests, exact match, schema parse, end-state checks), use a model grader only for what code cannot check, and use humans to calibrate it

    Anthropic recommends deterministic graders where possible and calibrating LLM graders against human judgment, and calls LLM-as-judge "generally not a very robust method" S1 S9

19. llm-evals 04

    Instead of folding malformed output into general failures

    Do track format validity as its own metric and put a defensive parser in front of the grader

    LinkedIn reports cutting malformed YAML outputs from ~10% to ~0.01% with a defensive parser S14

20. llm-evals 05

    Instead of scoring only accuracy on should-answer cases

    Do balance positive and negative tasks (should abstain, should refuse, should not call a tool) and score an honest abstention above a confident wrong answer

    Anthropic asks for balanced positive and negative cases, and OpenAI argues accuracy-only scoring rewards guessing over admitting uncertainty S1 S18

21. llm-evals 06

    Instead of one trial per task

    Do run k trials and report pass@k for capability and pass^k for anything a customer relies on every time

    75% per trial is only about 42% pass^3, and one Anthropic regression showed up in one run out of two S1 S10

22. llm-evals 07

    Instead of one blended suite that gates nothing

    Do keep capability evals (expected to start low) apart from regression evals (expected near 100%) and block merges of prompt, model or harness changes on regression drops

    Anthropic says regression evals "should have a nearly 100% pass rate" and after its April 2026 postmortem runs a broad per-model suite on every system prompt change, where one verbosity prompt cost 3% in ablation S1 S7

23. llm-evals 08

    Instead of letting the generator grade itself or sharing its context with the judge

    Do run the grader in a fresh context that sees only the task, the rubric and the output, tuned to be skeptical, with a fixed iteration cap

    Anthropic found tuning a standalone evaluator to be skeptical "far more tractable" than making a generator critical of its own work S8

24. llm-evals 09

    Instead of comparing agent variants on score alone

    Do log tokens, tool calls, turns and wall time per trial and compare cost per passed task

    on BrowseComp, token usage alone explained 80% of performance variance and multi-agent runs used about 15x chat tokens S2

25. llm-evals 10

    Instead of trusting small leaderboard or A/B deltas

    Do pin and record resource limits, timeouts and infra config per run, rerun at different times, and treat gaps under 3 points as noise until configs match

    on Terminal-Bench 2.0, resource configuration alone moved scores by 6 points S5

26. llm-evals 11

    Instead of assuming the eval set is clean

    Do read transcripts for contamination and eval awareness, block benchmark material at the content level, and rescore flagged tasks

    Anthropic found 11 BrowseComp answers came from benchmark materials, and URL-level blocklists were not enough S6

27. llm-evals 12

    Instead of one global score over head queries

    Do report metrics per segment (new items, tail queries, task type) and measure retrieval separately from generation

    Zepto reports head-query benchmarks ignore ~80% of query space, and Anthropic tracks retrieval as top-20 failure rate, 5.7% down to 1.9% S20 S4

28. llm-evals 13

    Instead of counting how much code or how many tests an LLM generated

    Do pass every candidate through filters (builds, passes repeatedly, adds coverage or kills a mutant) and measure the rate humans accept

    on Meta's Reels and Stories, TestGen-LLM candidates went 75% built, 57% passed reliably, 25% raised coverage, and Meta reports 73% acceptance for both TestGen-LLM and ACH S15 S16

29. llm-evals 14

    Instead of one big prompt run for a bulk LLM migration

    Do loop run, collect failures, tune the prompt, sweep again, with the existing test suite as grader, and finish the long tail by hand

    Airbnb reports 75% of files migrated in four hours, 97% after four days of "sample, tune, sweep", and the last 3% by hand S17

30. llm-evals 15

    Instead of treating offline evals as proof after ship

    Do run continuous quality evals on production traffic, track a usefulness signal (marked useful, comment addressed), and turn every production miss into a new task

    an Anthropic routing bug reached 16% of Sonnet 4 requests in its worst hour, and the Sep 2025 postmortem fixes added continuous quality evals on production, and Uber reports 75% of uReview comments marked useful S3 S19

31. system-design-scaling 01

    Instead of picking an architecture from a diagram

    Do write the capacity estimate first: peak requests per second, read/write ratio, bytes per day, working-set size and total DB connections, then size the database before the app tier

    Zerodha notes a framework doing 500K rps in front of a DB doing 1K qps serves 1K rps S6

32. system-design-scaling 02

    Instead of sizing for average load or a smooth growth curve

    Do size for the burst ratio and the launch or event step, and load-test that number

    Zerodha treats 10x bursts over 10 s as normal [S6], Threads passed 100M users in its first five days [S13], and one Netflix live event peaked at 65M concurrent streams S10

33. system-design-scaling 03

    Instead of sharding or adding a cluster by reflex

    Do check whether the working set fits in RAM on one box and whether one process can hold the connections

    Zerodha reports a full trading day of data for millions of users fits in a few hundred GB of Redis, and one Go service holds hundreds of thousands of WebSockets S6

34. system-design-scaling 04

    Instead of giving every pod a generous connection pool

    Do set pool max about 20% above observed peak and add up connections across all pods against the DB limit

    Zepto found pools 10x oversized (20 active plus 70 overflow per pod) and set out to cut 16k active Postgres connections to 5k [S17]; Zerodha says 10-100 pooled connections serve 10K HTTP requests S6

35. system-design-scaling 05

    Instead of buying storage-tier cores for a read-heavy hot path

    Do put a shared cache inside the data access layer, with a measured hit rate, a written invalidation path and a client-side staleness metric (share of reads that disagree with the source of truth after N minutes)

    Uber reports one 6M+ RPS use case on about 3K Redis cores versus about 60K CPU cores from storage [S3], CacheFront now serves 150M+ reads/s [S4], and Meta says fixing cache consistency starts with measurement; its Polaris checker took TAO from six to ten nines within five minutes S23

36. system-design-scaling 06

    Instead of adding nodes when a big database cluster's p99 climbs

    Do put a data-services layer between clients and the database and measure it before swapping the engine

    Discord went from 177 Cassandra nodes to 72 ScyllaDB nodes with p99 reads 40-125 ms to 15 ms, crediting part of the gain to its Rust data services layer S1

37. system-design-scaling 07

    Instead of using Redis as a work queue that must survive backlogs

    Do use a queue with guaranteed delivery that can absorb a large backlog while consumers are down

    Discord's Redis indexing queue maxed CPU and dropped messages during Elasticsearch node failures, so it moved to Pub/Sub S2

38. system-design-scaling 08

    Instead of growing one cluster past a few hundred nodes

    Do split it into many smaller clusters behind a routing layer so each failure and each operation stays small

    Discord split 200+ node Elasticsearch clusters into 40 clusters; median query latency went from 500 ms to under 100 ms and p99 from 1 s to under 500 ms S2

39. system-design-scaling 09

    Instead of spreading every request across all availability zones

    Do silo services per AZ so a sick AZ can be drained by weight, and list the services that cannot be siloed

    Slack's design goal is draining an AZ within 5 minutes in 1% steps for a 99.99% SLA, but Vitess still needs a single backend per write S5

40. system-design-scaling 10

    Instead of letting each team hand-roll clients and schemas on raw stores

    Do offer a few shared data abstractions (key-value, time series, counter) with the store behind them

    Netflix built its Distributed Counter on its TimeSeries abstraction and reports close to 75K count requests/s at single-digit ms latency S7 S8 S9

41. system-design-scaling 11

    Instead of dual-writing to several stores from application code

    Do send the write through a durable write-ahead log that replays into each store

    Netflix built a WAL abstraction specifically against data loss, corruption and entropy between stores S11

42. system-design-scaling 12

    Instead of denormalizing a hot document so every read is one lookup

    Do count writes per business update and normalize when one change fans out to many writes

    Zepto cut 1,000+ writes per product update to one, moved from 7 x M40 to 3 x M60 (85% less infra) and cut API latency 50-70% S18

43. system-design-scaling 13

    Instead of letting a demand spike reach the origin

    Do admit traffic at the edge with priority rules you can change in seconds without a deploy

    Zepto's CDN visitor prioritization cut 5xx and 4xx errors by 80% during high-demand events S19

44. system-design-scaling 14

    Instead of retrying failed dependency calls immediately or on a fixed interval

    Do use randomized exponential backoff and put new code paths behind a feature flag

    Google Cloud's 2025-06-12 Service Control crash lacked randomized backoff, and us-central1 took up to about 2 h 40 min to recover as the quota DB overloaded S21

45. system-design-scaling 15

    Instead of rewriting in a new language because a service "is slow"

    Do prove the runtime is the bottleneck with a measured, repeating signature first

    Discord moved Read States from Go to Rust after measuring latency and CPU spikes roughly every 2 minutes caused by forced Go GC runs S16

46. reliability-incident-response 01

    Instead of root-causing while users are still failing

    Do take the lowest rung of the mitigation ladder (see Answer shape) that stops the bleeding, then investigate

    Anthropic ended its Aug 2025 output-corruption and compiler issues with rollbacks, and Slack built cells so a sick zone can simply be drained. S11 S13

47. reliability-incident-response 02

    Instead of pushing config through a global system that applies everywhere at once

    Do ship config like code: canary, percentage, then global, with a health gate and an automatic halt between stages

    Cloudflare's Dec 5 2025 change went out via the global config system with no gradual rollout and returned 500s to ~28% of HTTP traffic for ~25 min. S2

48. reliability-incident-response 03

    Instead of trusting files your own pipelines generate

    Do validate them before publish and again on load (schema, row count, size against hard limits) and keep the last-known-good to fall back to

    On Nov 18 2025 duplicate rows doubled Cloudflare's Bot Management feature file past a 200-feature limit (about 60 in use); Google's Service Control crashed on policy data with blank fields. S1 S7

49. reliability-incident-response 04

    Instead of unwrap(), asserts or null dereferences on a hot path fed by config

    Do handle the error by keeping the previous config, emitting a metric and serving traffic

    Cloudflare's FL2 proxy panicked with "called Result::unwrap() on an Err value" and Google's binary crashed on a null pointer. S1 S7

50. reliability-incident-response 05

    Instead of shipping a new code path switched on

    Do put it behind a flag that defaults off, enable it by percentage, and keep the flag as the kill switch until it has soaked

    Google's report says the new Service Control code "did not have appropriate error handling nor was it feature flag protected". S7

51. reliability-incident-response 06

    Instead of fixed-interval retries or a breaker built on an unmaintained library

    Do use capped exponential backoff with random jitter and a maintained circuit breaker

    Google's tasks lacked "randomized exponential backoff" and us-central1 took up to ~2h 40m to fully recover while its quota database was overloaded; Hystrix is in maintenance mode and Netflix points new work to resilience4j. S7 S9

52. reliability-incident-response 07

    Instead of letting a cleanup or delete job accept an empty or missing selector

    Do reject empty scope, dry-run and print the count, and abort above a blast-radius threshold

    On Feb 20 2026 a Cloudflare task passed pending_delete with no value, it was read as all prefixes, and 25% of 4,306 BYOIP prefixes were withdrawn for 6h 7m. S6

53. reliability-incident-response 08

    Instead of assuming a config change is safe because nothing broke when it shipped

    Do diff the intended and actual state at change time and alert on mismatches that have no live effect yet

    A Jun 6 2025 topology change wrongly linked 1.1.1.1 prefixes to a pre-production service and sat latent until a Jul 14 refresh withdrew them for 62 min. S4

54. reliability-incident-response 09

    Instead of reaching recovery tooling over the same network and auth path that is failing

    Do keep out-of-band, break-glass access with its own network and credentials, and rehearse it

    At Meta in 2021 an audit-tool bug let a backbone command through, then DNS servers withdrew their own BGP advertisements, so a single command took the backbone and the DNS routes down together. S8

55. reliability-incident-response 10

    Instead of hiding a critical dependency behind a single external store

    Do list each critical path's dependencies and give login and control-plane paths a fallback that does not share them

    In Cloudflare's Jun 12 2025 outage, Workers KV failed 90.22% of requests for 2h 28m and Access failed 100% of identity logins. S5

56. reliability-incident-response 11

    Instead of shipping new regexes or rules that run on every request untested

    Do use a linear-time engine or a CPU budget per rule, and canary new rules while watching CPU

    Cloudflare's 2019 WAF regex "backtracked enormously", pinned CPU near 100% and took the service down for 27 min. S3

57. reliability-incident-response 12

    Instead of one shared failure domain

    Do split traffic into cells (zone or region) that can be drained in small weighted steps

    In 2021 a single-AZ link fault still caused user-visible errors at Slack; its cell design targets draining a zone within 5 min in 1% steps. S13

58. reliability-incident-response 13

    Instead of trusting aggregate error dashboards to catch quality regressions

    Do run continuous correctness probes on production, per platform, plus soak periods for changes that trade quality for speed

    Anthropic's routing bug began at 0.8% of Sonnet 4 requests and peaked at 16% in one hour, and about 30% of Claude Code users saw at least one misrouted message. S11 S12

59. reliability-incident-response 14

    Instead of paging on vague thresholds or single spikes

    Do write numeric paging rules with a rate, a duration and known-deploy-window exclusions, and verify every mitigation against them

    Anthropic's on-call rule pages when "the error rate is greater than 2% for longer than 5 minutes AND it's not a known deploy window". S14

60. reliability-incident-response 15

    Instead of leaving kill switches and rollout flags in the code after launch

    Do give each flag an owner and an expiry, and delete stale ones with a tool

    Uber used Piranha to remove around two thousand stale feature flags and their code. S10

61. performance-engineering 01

    Instead of reporting mean latency or one benchmark number

    Do report p50, p95 and p99 at a stated load and concurrency, with the run config

    averages hide the tail; Discord reports Go GC forced "latency and CPU spikes roughly every 2 minutes", and Netflix judged ZGC on P99 as well as average. S9 S3

62. performance-engineering 02

    Instead of guessing the hot path from code reading

    Do take CPU and allocation profiles under production-like load before changing anything

    Zepto reports string split, trim and concat were 20% of CPU and 50% of allocations, and Meta's Strobelight post reports a one-character "&" fix worth an estimated 15,000 servers of capacity per year. S12 S2

63. performance-engineering 03

    Instead of fixing whatever is easiest

    Do rank by share of total time or cost and attack the largest share first

    Zepto reports 13+ of 20 build minutes were inside Terser, so swapping only the minifier took the build to about 2 min with no bundler migration. S13

64. performance-engineering 04

    Instead of moving to bigger nodes when p99 drifts

    Do profile the code and data path first

    Zepto reports c6i.2xl up to 12xl nodes and node isolation did not fix a 25 ms to 200 ms+ p99 drift, while code and cache fixes later cut peak pods by 50%. S12

65. performance-engineering 05

    Instead of watching container CPU usage

    Do alert on CPU throttling and set runtime parallelism such as GOMAXPROCS to the container limit

    Go saw node cores and the HPA averaged the app with sidecars; Zepto reports up to 95% lower latency on critical APIs after fixing requests, limits and GOMAXPROCS. S10

66. performance-engineering 06

    Instead of rewriting a service in another language for tail latency

    Do measure GC pauses and try a low-pause collector first, and rewrite only when the runtime is the floor

    Netflix runs over half its critical streaming services on JDK 21 Generational ZGC with pauses "usually under one millisecond"; Discord rewrote Read States in Rust after Go GC forced spikes every ~2 min. S3 S9

67. performance-engineering 07

    Instead of adopting virtual threads as a drop-in swap

    Do load-test with lock contention and at least as many blocked tasks as carrier threads, and keep thread dumps ready

    Netflix hit "a variation of the classic deadlock problem" where 4 virtual threads pinned in synchronized on a 4 vCPU instance exhausted the carrier pool and Tomcat stopped serving. S4

68. performance-engineering 08

    Instead of opening upstream connections per request or per worker

    Do share and reuse them, and size pools from observed peak

    Cloudflare reports Pingora lifted reuse for one major customer from 87.1% to 99.92% and used about 70% less CPU and 67% less memory; Zepto set pool max about 20% above observed peak after pools ran 10x too large. S1 S14

69. performance-engineering 09

    Instead of serving hot reads from the primary store

    Do put a cache tier in front and track hit rate and miss penalty per tier

    Uber reports 6M+ RPS at a 99% hit rate on about 3K Redis cores versus about 60K cores from storage; Zepto cut miss rate from 3% to 0.1% and miss penalty from ~200 ms to ~1 ms. S15 S12

70. performance-engineering 10

    Instead of trusting a microbenchmark win

    Do re-measure end to end at the same throughput and report pods or cost saved

    Zepto reports a goroutine finishing in microseconds still took ~39 ms end to end, and counted its pprof win as p99 down ~40% and pods down 35% at the same throughput. S12 S11

71. performance-engineering 11

    Instead of comparing runs from different machines or configs

    Do pin and document hardware and resource limits, repeat across times and days, and distrust small gaps

    Anthropic measured a 6 percentage point swing on Terminal-Bench 2.0 from resource configuration alone. S23

72. performance-engineering 12

    Instead of quoting LLM tokens per second

    Do report TTFT and ITL percentiles plus throughput at a fixed concurrency

    in Red Hat's llm-d test TTFT P95 fell from 745 to 272 ms while throughput went from 0.52 to 0.49 req/s, a trade one number hides; GuideLLM suggests TTFT under 200 ms for chat. S6 S5

73. performance-engineering 13

    Instead of resending a large static prompt on every call

    Do put a byte-stable prefix first, cache it, and keep the same model for the session

    caching took a 100K-token prompt from 11.5 s to 2.4 s TTFT, and cached tokens cost 10% of base input. S19 S20

74. performance-engineering 14

    Instead of piping raw tool output into the model

    Do cap, paginate and filter it in code before it reaches context

    Claude Code caps tool responses at 25,000 tokens by default (configurable), and code execution cut one MCP workflow from 150,000 to 2,000 tokens. S16 S17

75. performance-engineering 15

    Instead of profiling only during incidents

    Do run always-on fleet profiling and a CI perf budget that fails on regression

    Meta runs 42 profilers under Strobelight and uses its LBR profiles for FDO worth up to 20% fewer CPU cycles. S2

76. data-ml-platforms 01

    Instead of letting every producer write one shared event shape that consumers parse defensively

    Do version each event schema, give each version its own ingest path, and keep the source offsets on every row

    Zepto reports its 1B events/day ads analytics uses one ClickHouse materialized view per event-schema version and keeps Kafka _topic/_partition/_offset for audit. S1

77. data-ml-platforms 02

    Instead of trusting a generated feature or config file because your own pipeline built it

    Do validate its row count and size against the consumer's hard limit before you publish it

    At Cloudflare a ClickHouse permission change made a metadata query return duplicate rows, the Bot Management feature file doubled, broke a 200-feature limit (about 60 in use), and the proxy panicked. S17

78. data-ml-platforms 03

    Instead of assuming the stream delivers exactly once

    Do dedup at write time over a bounded window and apply the same dedup again in the read view

    Zepto dedups ad events with argMax over a 4-hour window and repeats it at read time in a view. S1

79. data-ml-platforms 04

    Instead of rerunning freshness-critical jobs over the whole table

    Do scope each run to keys with recent activity so the job finishes well inside its interval

    Zepto's budget-overspend job runs every 3 minutes, finishes in about 1 minute, and only touches campaigns with events in the last 3 hours. S1

80. data-ml-platforms 05

    Instead of storing every raw event to answer reach and audience counts

    Do keep probabilistic sketches when an approximate distinct count is enough

    Zepto estimated 100M+ events/day as about 3B records and 1.2 TB a month raw, and switched to sketches whose size stays constant. S2

81. data-ml-platforms 06

    Instead of computing ranking features live in the request path

    Do pre-materialize them and define each feature once so training and serving read the same definition

    Zepto says live feature compute "is incompatible with a sub-200ms latency budget"; Airbnb open-sourced its feature platform Chronon, co-maintained with Stripe. S4 S8

82. data-ml-platforms 07

    Instead of fanning one catalog change into many denormalized writes

    Do normalize the source of truth and batch downstream updates in windows

    Zepto cut 1,000+ writes per product update to one, moved from 7 x M40 to 3 x M60 (85% less infra) and cut API latency 50-70%. S3

83. data-ml-platforms 08

    Instead of training a separate recommender per surface on its own slice of behavior

    Do evaluate one foundation model on all interactions and adapt it per surface

    Netflix reports over 300M users at end of 2024 produce hundreds of billions of interactions, comparable to LLM token volume. S11

84. data-ml-platforms 09

    Instead of each team running its own cron scripts

    Do put batch and ML jobs on one orchestrator and measure the orchestrator's own overhead as a first-class metric

    Netflix Maestro runs about half a million jobs a day (around 2M on busy days), and a 2025 engine rework cut overhead from seconds to milliseconds, 100x. S12 S13

85. data-ml-platforms 10

    Instead of dual-writing to several stores from application code and hoping they agree

    Do put a durable write-ahead log in front so failed or partial writes can be replayed

    Netflix built a WAL abstraction specifically against data loss, corruption and cross-store entropy. S14

86. data-ml-platforms 11

    Instead of letting every dashboard chart re-run the same heavy virtual-dataset SQL

    Do materialize the dataset when it goes stale, with a timeout, and point charts at the result

    Zepto had 16K+ virtual datasets firing up to 12 identical queries per dashboard; materialization cut warehouse cost 30-35%. S7

87. data-ml-platforms 12

    Instead of reading one global metric built from head queries or the biggest segment

    Do report the metric per segment, including the long tail and new items

    Zepto found head-query benchmarks ignore about 80% of query space and now reports segment numbers such as retrieval efficiency for new listings. S5

88. data-ml-platforms 13

    Instead of a human triaging every failed job by reading logs

    Do classify failures automatically and route each class to an owner, with a human checking the classifier

    Netflix reports about 100,000 workloads fail on peak days and built Pensive for LLM-assisted diagnosis across Titus, Spark, Trino and Druid. S15

89. data-ml-platforms 14

    Instead of each team wiring its own LLM vendor keys and clients

    Do route all model calls through one gateway that speaks a single API shape

    Uber's GenAI Gateway mirrors the OpenAI HTTP/JSON interface and serves 60+ use cases for close to 30 teams across OpenAI, Vertex AI and in-house models. S9

90. data-ml-platforms 15

    Instead of routing every ad hoc question through an analyst writing SQL by hand

    Do offer text-to-SQL scoped to known tables and keep the generated SQL visible for review

    Uber reports QueryGPT cut query authoring from about 10 minutes to about 3, against roughly 1.2M interactive queries a month. S10

91. dev-productivity-ai-coding 01

    Instead of opening with a refactor, a bundler migration or a new abstraction

    Do profile, then climb the change-size ladder and stop at the first rung that holds

    Zepto found 13+ of 20+ build minutes inside Terser and swapped the minifier, taking builds to about 2 minutes with no bundler migration S22

92. dev-productivity-ai-coding 02

    Instead of letting the agent code straight from a chat prompt

    Do write a SPEC.md with done criteria, build it in a fresh session, and review with a fresh-context subagent that never saw the reasoning

    a reviewer without the author's reasoning catches what the author rationalized S1

93. dev-productivity-ai-coding 03

    Instead of a long CLAUDE.md of preferences

    Do keep it to what the code cannot tell the agent: the test and typecheck commands, pointers and critical gotchas, under 200 lines, with an owner

    the check command is the most useful line, and personal preferences belong in a gitignored local file S1 S8 S24

94. dev-productivity-ai-coding 04

    Instead of repeating a CLAUDE.md line the agent keeps missing

    Do move it into a command hook or deny rule, and run the test check from a Stop hook

    CLAUDE.md and skills are non-deterministic; hooks, deny rules and managed settings are deterministic, and exit code 2 blocks S1 S8

95. dev-productivity-ai-coding 05

    Instead of asking for the feature and its tests in one go

    Do write the failing test first and ask for just enough code to pass one test at a time, pasting the full error output on each failure

    Anthropic's org rollout guidance is tests first, one test at a time, full error output S11

96. dev-productivity-ai-coding 06

    Instead of scaling agents against a test suite you have not hardened

    Do make the verifier near-perfect first and make its output agent-readable: a few stdout lines, ERROR plus reason on one line, details in a log, a fast sampled mode

    Anthropic's 16-agent C compiler build found the verifier must be "nearly perfect, otherwise Claude will solve the wrong problem" S4

97. dev-productivity-ai-coding 07

    Instead of one marathon session on a long build

    Do run one feature per session with a progress file, a JSON feature list where the agent may only flip passes, a commit per feature, and a start ritual that reads git log and runs an end-to-end check

    JSON was less likely than Markdown to be overwritten, across a list of over 200 features S2

98. dev-productivity-ai-coding 08

    Instead of launching many parallel agents on day one

    Do start with two or three sessions in separate worktrees and add more only while review keeps up

    Anthropic's merges rose from about 2,500/day in May 2026 to over 30,000/day by September, and each bottleneck removed produced more code to review S12 S5

99. dev-productivity-ai-coding 09

    Instead of carrying one session across unrelated tasks

    Do start fresh per task, rewind failed attempts, and hand exploration of ten or more files, or three or more independent pieces, to subagents that return a summary

    noisy side work in the main context crowds out the task; same-file edits stay in one context S9 S10

100. dev-productivity-ai-coding 10

     Instead of hand-migrating thousands of files or chasing 100% automation

     Do run the LLM per file with a validator and retries fed by the error, tune the prompt on a sample of failures, and finish the tail by hand

     Airbnb reports 75% of ~3.5K test files in four hours, 97% after 4 days of tuning, last 3% by hand, 6 weeks against 1.5 years estimated S13

101. dev-productivity-ai-coding 11

     Instead of letting an AI reviewer post every thought

     Do measure the share of comments marked useful and addressed, cap nits per review and cap review rounds

     Uber reports 75% of uReview comments marked useful; Anthropic's 3-round cap moved one-round pass from 65% to 78% S14 S6

102. dev-productivity-ai-coding 12

     Instead of one approval rule for every PR

     Do classify risk and require 0, 1 or 2 human approvals by tier, never counting an agent approval as a human one

     Anthropic's Claude Code team runs exactly this tiering for agent-heavy review load S6

103. dev-productivity-ai-coding 13

     Instead of tracking average review time

     Do track P75 time-in-review and nudge diffs that sit

     Meta's NudgeBot cut average time-in-review 7% and diffs waiting over 3 days 12% S18

104. dev-productivity-ai-coding 14

     Instead of full CI and full clones on every agent commit

     Do gate main with a merge queue, run tests by impact, and use sparse checkouts that ask the server for diffs

     Uber reports SubmitQueue cut CI resource usage 53% and wait times 37%; Anthropic's sparse checkouts went from minutes to seconds and freed 25% CI CPU S16 S5

105. dev-productivity-ai-coding 15

     Instead of accepting LLM-written tests because coverage went up

     Do gate each one on builds, passes repeatedly, and adds coverage or kills a mutant, then let a human accept

     on Meta Reels/Stories TestGen-LLM tests built 75%, passed reliably 57% and raised coverage 25% of the time, and ACH targets tests at generated mutants S20 S21

106. security-engineering 01

     Instead of leaning on per-action permission prompts for safety

     Do run the agent in an OS-level sandbox with filesystem limits and default-deny network

     users approve roughly 93% of prompts, so prompts are weak oversight, while sandboxing cut prompts by 84% in internal use S2 S3 S1

107. security-engineering 02

     Instead of isolating only the filesystem or only the network

     Do enforce both boundaries together

     without network isolation a compromised agent exfiltrates files like SSH keys, and without filesystem isolation it can escape to the network S1

108. security-engineering 03

     Instead of passing API keys, git tokens or env secrets into the sandbox

     Do inject credentials from a proxy or vault outside it and scope what the proxy allows

     Anthropic keeps git credentials and signing keys outside the sandbox with a proxy that restricts pushes to the configured branch, and its managed-agent harness never sees credentials S1 S4 S16

109. security-engineering 04

     Instead of treating an allowlisted domain as safe

     Do treat every egress allowance as a capability grant and bind outbound calls to the sandbox's own identity

     allowing api.anthropic.com in Cowork let an attacker's API key carry data out, fixed by a proxy that only passes requests with the VM's own session token S2

110. security-engineering 05

     Instead of trusting model training or a single guardrail to stop prompt injection

     Do layer guardrails and keep hard limits in the environment

     Opus 4.7 injection success is roughly 0.1% on one attempt but around 5-6% after 100 adaptive attempts S2 S7

111. security-engineering 06

     Instead of showing an action-safety classifier the agent's reasoning and tool results

     Do give it only user messages and the proposed tool calls

     stripping assistant text and tool output is how auto mode resists injection, and its honest miss rate on real overeager actions is still 17% S3

112. security-engineering 07

     Instead of approving everything or nothing

     Do tier actions: auto-approve routine work, hard-deny destructive commands, route prod infra and outbound messaging to a human

     with auto mode the agent works 9x longer between interruptions, and Gusto saw a denial in roughly 10% of session transcripts S17 S3

113. security-engineering 08

     Instead of letting each team wire MCP servers straight into agents

     Do front them with a central gateway that enforces tool-level authz, redacts PII and keeps auto-generated tools disabled until an owner approves

     Uber reports the gateway hosts over 800 MCP servers and over 5000 tools S6

114. security-engineering 09

     Instead of giving the agent a human's credentials and the full tool list

     Do issue a single-purpose service identity with separate read and write credentials and drop dangerous verbs from the tool list

     revoking a dedicated agent identity ends its access everywhere, and a smaller verb set shrinks blast radius S18 S16 S14

115. security-engineering 10

     Instead of routing raw tool output and customer PII through model context

     Do filter data inside the sandbox and tokenize PII before the model sees it

     in Anthropic's example the model sees 5 rows of a 10,000-row sheet and emails become placeholders like [EMAIL_1] S5

116. security-engineering 11

     Instead of trusting internally generated config or feature files

     Do validate their size and schema like user input and roll them out in stages

     a Cloudflare feature file doubled past a 200-feature limit and panicked the proxy on 2025-11-18, and a global config push with no gradual rollout hit about 28% of HTTP traffic on 2025-12-05 S12 S13

117. security-engineering 12

     Instead of enforcing data purpose with point checks at each call site

     Do track data flows with labels checked at runtime across the stack

     Meta found point checking did not scale and reports 10x better computational efficiency with flow-based Policy Zones S8 S9

118. security-engineering 13

     Instead of forwarding every LLM or scanner security finding

     Do keep a THREAT_MODEL.md, re-check findings with an adversarial verifier in a fresh container and require a working PoC

     findings were exploitable 90% of the time with good threat models, and the verifier roughly halved non-exploitable findings S15

119. security-engineering 14

     Instead of patching internet-facing apps on a monthly cycle

     Do patch within 24 hours of a public exploit, ordering work by CISA KEV then EPSS

     Anthropic expects an order-of-magnitude rise in finding volume from AI-accelerated offense S19

120. security-engineering 15

     Instead of deferring post-quantum work

     Do turn on post-quantum key agreement where your edge supports it and inventory authentication for the next step

     Cloudflare reports over 65% of human traffic is already PQ-encrypted and targets fully PQ-secure, including authentication, by 2029 S10 S11

121. frontend-mobile-engineering 01

     Instead of a vague "make the app faster" goal

     Do set numeric budgets for median cold start, slow frames on your heaviest list, and memory, then track each per release

     Discord reports halving Android median startup in 2023 and up to 60% fewer slow frames in the chat list, plus ~12% less chat-list memory S3

122. frontend-mobile-engineering 02

     Instead of testing performance on flagship phones

     Do pick asset formats and render paths by what the low-end device holds at frame rate

     Discord reports animated WebP let low-end devices hit 60 fps S3

123. frontend-mobile-engineering 03

     Instead of budgeting only the client

     Do give each stage of a user-facing request a latency share and move expensive work out of the request path

     Zepto says live feature compute "is incompatible with a sub-200ms latency budget" and reads pre-materialized features instead S9

124. frontend-mobile-engineering 04

     Instead of shipping the biggest model or payload that fits

     Do shrink to the smallest one that meets quality inside the budget

     Zepto compared a 100M-param ModernBERT with smaller models and shipped a 4M-param ranker deciding in under 100 ms S8

125. frontend-mobile-engineering 05

     Instead of migrating the bundler when frontend builds crawl

     Do profile the build and replace the one hot stage

     Zepto found 13+ min inside Terser, swapped the minifier to esbuild and went from 20 min to ~2 min with no bundler migration S6

126. frontend-mobile-engineering 06

     Instead of pushing full JS bundles as OTA updates

     Do ship binary diffs generated against every live bundle

     Zepto's Phoenix cut updates from ~6 MB to ~500 KB and moved Android to 80%+ adoption in under 24 h, against 18-20% in 4 days before S5

127. frontend-mobile-engineering 07

     Instead of keying the OTA bundle version on the app version

     Do key it on native compatibility so one bundle serves every binary that can run it

     Phoenix ties the JS bundle version to native compatibility, not to native version S5

128. frontend-mobile-engineering 08

     Instead of flipping a config or flag for everyone at once

     Do roll it out by percentage with a tested off switch and a halt on error-rate rise

     Cloudflare's 2025-12-05 global config change had no gradual rollout and hit ~28% of HTTP traffic for ~25 min; Google Cloud's 2025-06-12 change was not flag protected S15 S16

129. frontend-mobile-engineering 09

     Instead of letting flags live forever

     Do give each flag an owner and an expiry and delete dead branches with tooling

     Uber's Piranha removed around two thousand stale flags across Objective-C, Swift and Java S17

130. frontend-mobile-engineering 10

     Instead of hardcoding layout per platform for surfaces that change every sprint

     Do let one backend response define layout, sections and actions over typed client components

     Airbnb's Ghost Platform drives web, iOS and Android from a single backend response S1

131. frontend-mobile-engineering 11

     Instead of one fixed grid for every query or category

     Do treat density and layout as data the server picks per intent

     Zepto uses 3x3 grids for "garbage bag" and 2x2 with bigger images for "T-shirt", picked by intent S11

132. frontend-mobile-engineering 12

     Instead of server-driving flows with deep state and business logic

     Do keep them client-owned and structure them as a tree of business-logic units

     Uber's 2016 rider app is "a tree of Riblets" whose routing is driven by business logic, not views S2

133. frontend-mobile-engineering 13

     Instead of clients retrying failed calls at once

     Do use randomized exponential backoff and honor a server-sent wait or fallback state

     Google Cloud's Service Control lacked randomized backoff and us-central1 took up to ~2h 40m to recover S16

134. frontend-mobile-engineering 14

     Instead of deploying code to survive a peak event

     Do put admission rules at the CDN edge, editable in seconds, that let priority users through and send the rest a wait screen

     Zepto reports 80% fewer 5xx/4xx and zero downtime in high-demand events; Zerodha treats 10x bursts over 10 s as normal S7 S14

135. frontend-mobile-engineering 15

     Instead of hand-migrating a test suite file by file

     Do run an LLM migration loop with validation, then sample, tune and sweep the failures, and finish the tail by hand

     Airbnb moved ~3.5K React test files in 6 weeks against a 1.5-year manual estimate: 75% in four hours, 97% after 4 days of tuning, last 3% by hand S4

• skill

  engineer-review

  Picks the skills your diff touches and lists findings as path:line: severity: problem. fix. with the rule each one breaks.

• skill

  engineer-postmortem

  Turns incident notes into a blameless postmortem, matched against a library of public outages. Every action item gets an owner and a revisit trigger.

• agent

  engineer-reviewer

  Claude Code subagent the review skill delegates to, so the checklist pass runs in its own context and returns only findings.

Criteria passed, summed over two questions per skill. Run in October 2026. Most of the gap comes from sourced specifics, such as the cause of a named outage or a company's measured figure, which a model without the skill rarely recalls correctly.