Build 11 / Local AI Engineering Lab
From local models on an NVIDIA DGX Spark to Caraval: a working lab for turning small, defined software tasks into code, test evidence, review, and a human decision.
Published 26 October 2026 · Build evidence through 25 October 2026
Greg's take
The Spark gave me somewhere to run local models. The more interesting question was what I could trust them to build.
A model can produce code that looks convincing, pass the checks it has been given, and still miss part of the job. It can also do useful work and then fail to hand it over properly. Both happened in this lab.
So the build became as much about the workflow as the models: define the task, constrain the work, test the result, review the exact files, and keep the failed attempts in the record. That is the thread from the first Spark experiments to Caraval.
The aim is useful software with evidence I can inspect. Every step towards more automation has to earn its place.
The system
This is the engineering environment behind the experiments: local inference, controlled execution, and a record of what actually happened.
Hardware
An NVIDIA DGX Spark with a GB10 GPU and an ARM64 Linux environment provides the local compute. Ollama serves the local models. Memory, temperature, context size, and model residency all affect what the lab can safely attempt.
Workflow
Task briefs define scope and acceptance. Separate workspaces contain changes. The workflow records coding, checks, review, and the exact candidate that is being considered for integration.
Operator
A work queue, status views, delivery records, and operator reports make progress inspectable. Codex-assisted engineering supports the lab; local model runs remain separately recorded and bounded.
Build journey
01 / Foundation
The September Spark harness used disposable Python projects, explicitly editable files, separate builder and reviewer roles, and Docker checks. Candidate branches and reports were retained for human review.
02 / Reliability
Structured responses, context limits, model identity logs, and explicit failure states made it possible to tell a bad candidate from an infrastructure problem. A failed attempt stayed failed.
03 / Caraval
Caraval introduced structured briefs, isolated coding and review, frozen checks, and an on-demand runner. Queueing and project dependencies added coordination without making a queued item an automatic deployment.
04 / Local tools
The local worker route gained declared file access, native test feedback, finite response and correction budgets, and an explicit finish step. Context continuation became part of the engineering work.
05 / Useful output
RepoScout grew into a repository inventory and comparison tool with saved snapshots and change reporting. Task intake and input preparation reduced the manual work needed before a controlled run.
06 / Evidence
Packaging, relocation, restored-source checks, and recorded adoption decisions extended the definition of done beyond a passing run in the original workspace.
Inside the harness
Caraval connects the task, selected source, local worker, real tests, and review record. Codex remains the externally hosted lead; local worker inference runs on the Spark. The overall workflow is hybrid.
Workers can inspect declared files, make permitted edits, run the fixed checks, and finish or escalate. They do not receive an unrestricted shell.
Real test failures return to the worker while the approved budget allows. An edit invalidates earlier test success; exhausted or uncertain work stops for inspection.
Passing checks and an explicit finish produce a candidate for review. Neither a local model verdict nor a passing test automatically merges or deploys the product.
Local models / recorded use
Routing below reflects the 25 October build records; the three Qwen installations were checked on 26 October. Each role is scoped to its observed work. These are development results, not a general model leaderboard.
Primary local coder
Ollama · Q4_K_M · 65,536 tool-worker context
Used for small, explicitly scoped implementation tasks. Its JSON-action route makes edits, checks, and completion visible to the controller.
Observed: A supervised three-task RepoScout batch completed with one worker correction and no lead implementation corrections. Combined source passed 101 tests plus three frozen acceptance checks.
Limit: Codex still reviews the actual candidate. Other tasks have needed lead corrections or stopped before completion.
Secondary coding candidate
Ollama · Q4_K_M · 65,536 tool-worker context
An explicitly selected alternative for bounded coding work; it is not an automatic fallback.
Observed: Completed a frozen RepoScout selection task in five responses with no corrections. Original and restored source passed 77 tests plus frozen acceptance; the lead accepted it without code edits.
Limit: A separate review trial falsely accepted known defective code. Coding success does not qualify it as the sole reviewer.
Advisory + worker experiments
Ollama · Q4_K_M · 32,768 tool-worker context
The default tool-worker profile in the configuration, also used in a separate small advisory lane. Practical primary coding currently goes to Coder Next.
Observed: A context-continuation repair enabled a bounded worker task to finish with passing checks. Earlier failed and context-blocked runs remain in the record.
Limit: The seven-case advisory evaluation completed but failed its full quality rubric: it missed a required failure mode and made incorrect claims about two correct controls.
Selected advisory reviewer
vLLM · NVFP4 model · 65,536 recorded context
Run separately for selected bounded reviews, with native tests and Codex adjudication. It is not an always-on stage after every edit.
Observed: Accepted a correct RepoScout selector and rejected a deliberately broken version with a witness that Codex reproduced. Both review results were accepted for that two-case trial.
Limit: Cold startup took about ten minutes and the run recorded memory pressure and swap activity. The defect was already detectable by native tests; added detection beyond tests is unproven.
Context figures describe these configured routes, not maximum model capability. Models are selected for an explicit run and used sequentially; this diagram does not imply concurrent GPU serving or automatic model switching.
Earlier experiments / 28 September
Tested as a different-family reviewer in the original Spark harness. It passed 3 of 5 small smoke checks; standalone coding and arithmetic failures were retained. Its review was kept secondary to executable tests and human judgement.
Tested for heavier reasoning in the early local-model setup. It passed 5 of 5 small smoke checks. Those results establish that narrow test run, not current routing or general coding accuracy.
Both are historical tests. Neither appeared in the current Ollama inventory checked for this update. No model was started or benchmark rerun to prepare this page.
Recorded evidence / 25 October
These are documented outcomes from the lab records. Keeping the distinction between worker performance and accepted product quality makes the results useful.
Delivered product
The complete 22-file RepoScout package passed 128 project tests and 14 CLI scenarios from a relocated copy in the isolated ARM64 Python runtime. Repeated packaging produced identical bytes; an altered archive was refused. This verifies that delivery on the Spark, not off-device recovery.
Bounded local run
One fresh Coder Next task finished in four responses, with no corrections and no lead code edits. The record reports 27 frozen checks and 128 combined tests on original and restored source, followed by independent acceptance. That is evidence for one task, not a general reliability score.
Assisted delivery
The input-manifest task stopped at 17 of 18 checks. A separate lead correction addressed the remaining issues; the reviewed utility then passed 23 installed tests and the unchanged 18-test acceptance suite. The adopted tool is recorded as Codex-assisted, while the local attempt remains failed.
What went wrong
Testing
A snapshot command passed its original tests but exited where its imported function was required to return a result. Independent review found the gap. Direct-call checks made that requirement executable, and a later repair addressed it.
Context
Some runs produced passing code but exhausted their budget before an explicit finish. Context handling was also repaired to preserve current source and failure facts while making older unchanged inputs retrievable. Completion had to be designed into the workflow.
Efficiency
A recorded batch ran the same native tests three times. The installed repair reuses only matching successful evidence and invalidates it after an edit. The demonstrated saving was two avoided executions in that case; broader token or cost savings were not measured.
Current boundary
The lab has moved well beyond loading a model. Its current evidence supports bounded, supervised work. Broader autonomy remains something to evaluate.
Working
Defined tasks, bounded local workers, isolated checks, independent review, useful delivered tools, and recorded human adoption are already part of the build.
Still open
General overnight reliability and off-device recovery remain open. Useful successes do not close those gaps. The background heartbeat remains off in the latest build record.
Exploration
The next research thread is a separate distillation lab. Accepted examples and held-out evaluation preparation exist, but the recorded examples remain quarantined. No training or model-quality improvement is claimed.
The connection
The same ideas run through the Tanya Build Cockpit: explicit scope, retained context, review gates, and a person responsible for the result. The Spark and Caraval work puts those ideas under practical pressure with local models, constrained execution, and real delivery attempts.
The strongest result so far is a workflow that can preserve a useful candidate, expose a failed assumption, and give the next attempt a better starting point. The failed runs helped build that workflow as much as the successful ones.
Build something useful. Keep the evidence. Make the next run better.
Local inference, controlled coding workflows, and the engineering needed to turn model output into reviewable software.