Don't make the agent remember how to do its job

An agent is working on a change in your project, pushes it, and opens a pull request.

The pipeline starts, but it does not pass. You assume it is a one-off error and run them again. This time everything passes, so you move on. Two days later, someone else on your team hits the same case. Guess how they will react.

That is how a flaky test can sit in a repository for weeks. It fails too rarely for anyone to file a ticket right away, and too inconsistently to reproduce easily. Each instance looks harmless on its own. Only the history across many runs shows that the test suite is slowly losing credibility.

An agent can analyse one failed run. But who is going to remember every run that follows?

I wanted to build a process that collects CI results on its own, looks for these patterns, and keeps going after I close my laptop. So I decided to try it.

#One prompt is not enough

Fixing failing tests sounds straightforward. Fetch the GitHub Actions results, find the failed tests, and open an issue.

But one run proves nothing. To tell a flaky test from a normal failure, you need to compare results from many commits and many runs. You also need a threshold for when the pattern is serious enough to file an issue.

I used two tools for this.

Pi is a programmable sandbox that we can shape around our needs. It offers commands, extensions, and an SDK for controlling agent sessions. I used it as a workshop for building the solution itself and a mechanism for future integrations.

Kody gives that solution a remote home. It is a personal assistant that connects us to GitHub, stores state in Storage, runs on a schedule, and uses a Cloudflare runtime.

Pi helps us build the behaviour. Kody runs it after the laptop is closed.

kody-package-graduate/  ├── index.ts  ├── core.ts  ├── kody.ts  ├── package.json  └── package-lock.json

#Pi meets Kody

Pi is built on primitives, which lets us adapt the agent in almost any direction. We can extend its core by installing packages for things like subagents, web access, plan mode, or MCP, which we will use to connect to Kody.

MCP usually brings external-service integrations to mind. Kody uses it differently. Once connected to an agent, Kody exposes only two tools:

• search: finds available capabilities
• execute: runs TypeScript code in the Kody runtime

Before using them, though, we need to connect to GitHub. Kody offers a native integration that makes this much easier.

Connecting GitHub took a moment. I did not need to register my own OAuth App or hand a token over to Pi. Kody keeps all credentials and exposes only a secure way to connect to GitHub.

To make sure the connection works properly, we can run the code below through execute in Kody's remote environment.

The response is { login: "olafsulich" }.

#From experiment to package

Connecting to GitHub is only the beginning. We are authenticated, but we still need to pull out information about failing tests.

Through MCP, Pi calls execute to run code and check what works. That is how, one thread at a time, we work our way towards the solution: inspect completed runs, locate artifacts, download them, and parse the test results.

Along the way, we discover things that are easy to miss at first. Workflow status is not the source of truth, a green workflow can hide a failure, and a red one does not tell us which test case failed. JUnit is uploaded independently of the workflow result and records the outcome of every test case.

After enough iteration, we have a working and tested solution. We can turn the script into a Kody Package, a small project with code, tests, and a manifest. The manifest tells Kody what the package exposes and what it is allowed to run.

Instead of creating a skill that makes the agent fetch and check flaky tests every time, we give it a ready, deterministic capability that it can use when needed. That saves tokens and gives us a more reliable solution.

#Giving the package a life of its own

The package is ready. Now we need to run it repeatedly, collect results, and take action based on them. Kody handles all of that.

With a Job, we set the schedule for Kody to trigger an action. For the flaky test detector, Kody runs the script we prepared earlier in Cloudflare every three hours. These runs have no model involved. They only execute a process we have already tested.

We also need to persist results between runs. A test is only considered flaky after several runs, so we use Kody Storage, a private database. It tracks which tests have failed and how many times.

When a test crosses the configured failure threshold, the detector opens a GitHub issue listing the failing tests. For each one, it reports how many times the test ran and its failure rate.

#A command, not a checklist

Before publishing and configuring the whole failed-test detection workflow, we still had a few things to do.

1. Verify the author contract

   Check the README, title, and a suitable description.

2. Run the local tests

   Run the package's local test suite.

3. Create the Kody package repository

   Create a package repository on Kody to hold the source code.

4. Publish with the schedule disabled

   Publish the package with the schedule disabled, then check the output and any approvals waiting for secret access.

5. Invoke and inspect manually

   Invoke the code exactly as the scheduler will later, inspect the result, and confirm that the expected values were saved to Storage.

6. Enable the schedule

   Enable the schedule and verify that the whole flow works.

Every step matters if we want the final workflow to work reliably. It is still a lot to do manually every time we add a package.

We could tell the agent which steps to take, but that would not give us confidence in the final result. It is better to make the process deterministic.

For that, we use a Pi extension that does all the work for us. Under the hood, the command runs deterministic code: it takes a ready package directory and moves it to production the same way every time.

Before it finishes the configuration, kody-package-graduate reviews the package code, manifest, and declared configuration in a completely separate session. This is possible through Pi's createAgentSession.

Pi creates a new agent session with a tailored prompt, restricted tools, and no additional extensions or skills. The reviewer's job is to check the acceptance criteria and look for potential gaps. Because it is a separate session, the agent does not inherit the context of the main thread.

kody-package-graduate makes it much easier to create and release new packages. Let us make a new github-issue-prioritizer package for triaging open GitHub issues.

It fetches open issues from GitHub, sends the title, description, and existing labels to a model, then classifies each issue by type and priority. The result can look like this:

{
  "category": "bug",
  "priority": "high",
  "confidence": 0.91
}

Based on that result, the package adds controlled labels such as ai:priority:high and ai:type:bug. Existing labels added by people remain untouched. It stores the decisions in Kody Storage so later runs retain the history of earlier classifications.

While working with Pi, we used execute to explore the Kody API and test our assumptions. The package code was written locally, then we ran:

/kody-package-graduate ./packages/github-issue-prioritizer --job prioritize-issues

The command validated, reviewed, and released the package. Soon after, it was published on Kody, its job was enabled, and it could run independently of an active Pi session.

#Close the laptop

We connected two great tools. Pi gives us a programmable environment where an agent can create and test its own behaviour. Kody gives that behaviour a remote runtime, integrations, state, and a schedule, so it can keep running after the laptop is closed.