Summary
Self-taught AI evaluation and testing engineer building deterministic, replayable systems for measuring coding agents and LLM applications.
Merged upstream at promptfoo (by its co-founder), GitLab and TeaVM; an arXiv preprint; 30+ public repos, three PyPI packages, one npm package; a game shipped to Google Play.
I build verifiable-evaluation systems whose claims are checked rather than asserted: a public registry of 12 accepted evaluation claims, each hash-pinned to a commit and replayed weekly in CI (11 of 12 replayed green on 2026-10-19): deterministic, no LLM judges, failures published by design. Every one of these is listed with the defect, the proof and its live status →
Experience
AI/ML Engineer · Epix Devo LLC · Contract · Oct 2026 to Present
- Contract AI/ML engineering against approved requirements: model and framework evaluation, proof-of-concept prototyping, and integration with APIs, data stores and application backends.
AI Evaluation & Testing Engineer · Self-employed · Remote · Mar 2026 to Present
- Published evalmut, mutation testing for LLM eval graders, to PyPI (Aug 2026); two pull requests merged into promptfoo (26k-star LLM eval framework) by its co-founder (Sep and Oct 2026).
- Built a multi-agent LLM coding harness: a plan, build, review, ship loop with per-role model routing, a fresh-context LLM reviewer, an asynchronous evaluation role and a human on every consequential step; gates written as Claude Code hooks, later ported to pi-gates; property-based invariants over a native game engine and adb-driven on-device checks. Write-up (source private): exposure-watch/docs/HARNESS.md
- Operate model-drift, a public LLM regression tracker, in Python on a daily GitHub Actions schedule: it calls the OpenAI, Anthropic, Gemini, xAI and Groq APIs over REST, grades a frozen 35-task suite with deterministic checks, scores accuracy, latency, verbosity, reliability and refusal rate, and since Oct 2026 writes every run to PostgreSQL. Run daily since 2026-07-19 (one failed day); now 17 models / 5 providers.
- Diagnosed an apparent 66-point overnight drop (69% to 3%) as HTTP 429 rate limiting on Groq's shared 30-requests-per-minute tier, not model drift; shipped Retry-After-aware retries with exponential backoff, a per-host throttle and a reliability floor (runs under 50% call success leave the accuracy chart), and stopped failed calls scoring as wrong answers.
Indie Game Developer · SeraphLight Studios (self-employed) · Remote · Nov 2025 to Present
- Shipped Tap Dodge Rush to Google Play (Kotlin / Java, Android Views + Canvas, AdMob) after closed testing with 12+ testers and store review. A second, larger game (React / Capacitor), which the harness above was built on, was in active development through August 2026.
Line Cook to Kitchen Manager / Shift Lead · Big Island Brewhaus & other high-volume kitchens · 2016 to 2025
- Led shifts and trained staff; owned prep, inventory, and food safety under sustained pressure.
Skills
AI/ML engineering: LLMs · Generative AI (LLM application development) · OpenAI, Anthropic and Gemini APIs · Claude Code · Machine learning (LoRA fine-tuning with MLX)
Evals & AI: Deterministic grading · Mutation testing · Differential testing · Property-based testing · Regression gates in CI · Minimum-detectable-change floors · RAG (BM25, hybrid retrieval, nDCG@10) · Tool / function calling & guardrails · Safety (OWASP LLM Top 10, human-in-the-loop) · Prompt engineering · Context engineering · Model Context Protocol (MCP) · LangGraph · Multi-agent orchestration
Backend & data: Python · FastAPI · Pydantic · SQL · PostgreSQL (Neon) · SQLAlchemy 2.0 · pgvector · REST
Frontend / mobile: TypeScript · JavaScript · React · Next.js · Java · Capacitor · Android · WebAssembly · TeaVM
Ship & verify: pytest · Vitest · JUnit · jqwik · Playwright · Docker / Compose · CI/CD (GitHub Actions) · Prometheus metrics · Structured logging · PyPI / npm packaging · Cloudflare Workers · Test automation · QA for LLM systems
Open-Source Contributions & External Review
• Two pull requests merged into promptfoo (26k stars), both merged by its co-founder. #11278: a blank BLEU reference, such as an unset template variable, was scored instead of ignored; the fix shipped in promptfoo 0.124.0. #10256: the rouge-l / rouge-s assertion types, merged 2026-10-09.
• Three merge requests merged into GitLab: two CVE records whose stored score contradicted their own published vector (CVE-2023-5009 stored 9.6 CRITICAL; its vector computes 8.2 HIGH), and the security handbook's disclosure window, corrected from 30 days to 90.
• External audit of vac-protocol: an outside engineer falsified my own verifier,
cheapest forgery four bytes. Fix merged, then I found the same defect class four
more times (arXiv, below). As sole maintainer I merged four of his pull requests and answered his four specification issues (protocol v0.2); on two I showed his proposed fix would not close the hole and shipped a wider one.
• Outside critique, credited in the repo: Bartłomiej Nawara's missing-not-at-random critique of
model-drift's 429 handling shaped its reliability floor.
• Closed a bug report open in TeaVM since 2018 that its own suite could not catch: a one-character error in its GregorianCalendar emulation, found by differential-testing the browser port against the JVM. The patch re-enables the four commented-out assertions covering that line; merged same day by the maintainer.
• agent-certlab: Two real coding agents certified, every verdict regradeable from artifacts. Three seeded-defect task families with provable answer keys (incl. coordinated two-file defects where reverting either file alone still fails), a precondition prover (clean passes, every seed fails; it refused one of my own defects), and calibration agents the harness must separate before certifying.
• reference-fleet: An answer key for eval suites: six scripted reference models each broken one documented way at an exactly reproducible seeded rate; a naive suite detects 1 of 6 (paired protocol). A LoRA-trained native defect showed training's rate control is lossy: mixture 0.507 → realized 0.20 greedy, 0.31 at temp 1.0.
• vac-protocol: Verifiable Agent Claims: evidence-bundle format + offline verifier behind the registry above, with a public record of it rejecting my own evidence. Plus vac-gate: no verified contract, no green check.
• evalmut: Mutation testing for eval graders (built on its gradecore engine): inject known defects, report which checks stayed green. No LLM judge; every operator names its origin. Found 2 blind spots and 2 coverage gaps in gradecore's own graders and 8 in ports of a widely-used framework's assertions; separately fixed three other fail-open graders in gradecore 0.10.3. 325 tests.
• deadgate: a linter for CI checks that cannot fail in GitHub Actions workflows (PyPI). Across 275 repositories I hand-labelled all 653 of its findings and deleted two of its four detectors that the labels showed had never once been right: 36.1% false, measured rather than estimated. Found an unguarded required status check in heygen-com/hyperframes, fixed by my merged PR #5182. 228 tests.
• deadcheck: does an eval rubric catch what it claims? Run on three rubrics I did not write: OpenAI evals' Match accepts "10" for ideal "1"; a promptfoo example's refusal test passes a lowercased refusal-then-compliance reply. Seven probe rounds found 23 defects in deadcheck itself under a passing suite. 185 tests.
• rag-eval-lab: deterministic RAG eval harness that catches a planted hallucination in CI; from-scratch BM25 matching the published SciFact baseline (0.664 vs 0.665). 60 tests.
• eval-history: model-drift's run store on Neon PostgreSQL, written daily since Oct 2026 and archived daily as JSON: SQLAlchemy 2.0, Alembic migrations diffed against the models in CI, and a FastAPI API with Prometheus /metrics and JSON logs (tested, no longer hosted). 67 tests, coverage gate 88%.
• llm-gateway: a FastAPI LLM gateway that puts OpenAI, Anthropic and an offline mock behind one OpenAI-style endpoint: API-key auth, per-key token-bucket rate limits, a per-tenant cache, retries, SSE streaming, estimated cost accounting, Docker config. agent-graph can route through it. 29 tests, run against the mock.
• agent-graph: a LangGraph ReAct agent with
unit-testable multi-step tool use: a swappable mock/LLM policy runs it deterministically without a key; a
safe AST calculator (no eval); a max-step guard. 67 tests.
• harness-builder: More scaffolding scored worse, for 22x the cost. Three harness shapes on one 20-task suite, zero errors in 60 runs: one drafter 95%, planner plus drafter 90%, four-call panel with judge 80%. The panel beat the single drafter on 0 of 20 tasks, and the tool reported “this suite cannot decide” rather than ranking.
• tapdodge-engine: One rulebook, two runtimes, and a check that fails when they stop agreeing: a JVM-versus-browser (TeaVM) differential in CI. In a committed fault injection it was the only check that caught a runtime-supplied-RNG fault; the same-runtime determinism test and the golden file both passed it.
• Also public: exposure-watch (AI agents vs. plain code at known-exploited-vulnerability triage; pre-registered, 57 held-out pairs: a tie on correct answers, 47 to 45, p = 0.69, but the agent pipeline sent its 10 unsure cases to a person and got 0 wrong, vs. 12 wrong from code plus a keyword rule) · crashkit (live) · mcp-tools (MCP server, no SDK) · pi-gates / pi-eval (fail-closed commit and model gates, plus deterministic grading, for the pi coding agent) · llm-wire-stub (npm) · readme-test-count (fails CI when a README's test count disagrees with the suite; runs in the CI of six of my repos)
Publications
“Four Ways to Forge a Bundle My Own Verifier Calls Clean: Refusal-Site Mutation Testing of an Evidence-Bundle Verifier.” Erik Hill. arXiv preprint, 2026. arXiv:2608.26183
25+ articles on evaluation and testing failures. dev.to/agentdev9 · Drift Notes: erikhill.dev/notes
Education
Clark College, 2007 to 2009 (some college, no degree).