Choose an AI agent by running a job you can check. Record what it produced, what you had to fix, and which tools it could use. Nate Herk’s conversation with Mark Kashef makes a useful distinction: the model reasons about the task, while the software around it determines how that reasoning can become work.
01Check what the agent can do
That surrounding software is the harness: tools, context management, and the environment where actions run. Anthropic’s Claude Code documentation describes a loop of gathering context, taking action, and checking results. Tool results give the model evidence for its next step.
In the video’s example at 7:54, Kashef asks a local model to create a landing page and start it on his computer. Writing HTML and starting a server require different capabilities. The example concerns that session’s setup; it does not establish a limit for every local model application.
Before replacing a model after a failure, locate the failure:
| What happened | What to check |
|---|---|
| It described an action but never attempted it | Is the needed tool available in this session? |
| A tool returned access denied | Does the task require access the session lacks? |
| It read the wrong instructions | Which project and personal instruction files loaded? |
| It used the right inputs and tools but produced a wrong result | Repeat the task and compare model performance |
02Compare complete runs
Use the same starting files, request, and completion checks in separate workspaces. Record the app version, model, settings, loaded instructions, and available tools. If two apps use different models, the result compares the complete setups. It cannot tell you which individual part caused the difference.
For the landing-page job, decide in advance what counts as done: the page exists, the local server responds, and the page displays the requested content. Inspect the result yourself. The agent’s final message is only a claim about completion.
Keep a row for each run:
| Agent and model | Checks passed | Time | Cost, if shown | Manual fixes |
|---|---|---|---|---|
| First setup | Record results | Record elapsed time | Record or mark unavailable | Describe corrections |
| Second setup | Record results | Record elapsed time | Record or mark unavailable | Describe corrections |
Repeat before choosing; one successful run gives little evidence about consistency. For a larger set of tasks with expected results, use Score your agent on known good answers. Revisit the comparison after releases, as in Recheck your model choices each release.
03Test skills when you move them
Kashef describes adapting skills across agents at 17:26. The Agent Skills specification gives compatible tools a shared file format: a SKILL.md with a name, description, and instructions, plus optional scripts and reference files. A shared format still leaves execution requirements to check.
Keep the task instructions and reference material together. Record anything that depends on the app or machine:
- Where the agent discovers the skill and when it should use it.
- Which tools the instructions name and whether the new agent exposes them.
- Which script runtimes, packages, and relative file paths must exist.
- Which permissions the task needs and how the result will be checked.
The specification supports a compatibility field for environment requirements. Use a small example with a known result to verify the move. A readable file alone does not prove that the new agent can run the procedure. One skills library for your whole team covers maintaining the shared files once they work.
04Keep changes local first
At 31:14, Kashef describes keeping instructions specific to a project until broader use earns them a place in his personal defaults. That makes a failure easier to investigate: a new instruction in the consulting project has fewer places to affect.
Claude Code supports this distinction through project and user instruction files. Its documentation also distinguishes written guidance from enforced controls. Preserve permission limits when experimenting with shorter instructions.
Kashef reviews different parts of his setup at different intervals. For your own setup, record when an instruction last helped and retest it when the model or workflow changes.
“you want to make sure that your skill is actually adding value”
Mark Kashef, “How to Actually Choose the Right AI Agent” · 24:54Compare a procedure with and without its extra instructions while keeping the same access limits and completion checks. Trim the instructions your model outgrew covers that review. Archive a removed procedure with its test results so you can restore it if a later run needs it.
05How we use Harness Firmware
Our Harness Firmware template puts this approach into repository files. It stores reusable skills, project lessons, and verification rules alongside the work. This is our implementation, separate from the tools discussed in Nate’s video.
Claude Code and Codex get their own instruction files and skill entry points. Shared project knowledge stays in committed reference files, while each agent’s workflow accounts for its available tools and permissions. Moving the files still requires the checks above.
A confirmed lesson stays in its project first. After review, a reusable improvement can move back to the template for future projects. Every new repo starts with your lessons explains the structure; the current repository has setup instructions. Use the same sample task to compare your existing setup with a project using Harness Firmware, and record whether it reduces the corrections you need to make.