---
title: "Don't make the agent remember how to do its job"
description: "How Pi and Kody turn exploratory agent work into a tested package with remote runtime, durable state, and a schedule that keeps running after the laptop closes."
author: "olaf"
author_name: "Olaf Sulich"
tags: ["agents","pi","kody","automation","testing"]
published_at: "2026-08-27T14:54:24.530Z"
canonical: "https://mega.dev/pi-kody-integration"
---

An agent is working on a change in your project, pushes it, and opens a pull request.

The pipeline starts, but it does not pass. You assume it is a one-off error and run them again. This time everything passes, so you move on. Two days later, someone else on your team hits the same case. Guess how they will react.

That is how a flaky test can sit in a repository for weeks. It fails too rarely for anyone to file a ticket right away, and too inconsistently to reproduce easily. Each instance looks harmless on its own. Only the history across many runs shows that the test suite is slowly losing credibility.

An agent can analyse one failed run. But who is going to remember every run that follows?

I wanted to build a process that collects CI results on its own, looks for these patterns, and keeps going after I close my laptop. So I decided to try it.

---

## One prompt is not enough

Fixing failing tests sounds straightforward. Fetch the GitHub Actions results, find the failed tests, and open an issue.

But one run proves nothing. To tell a flaky test from a normal failure, you need to compare results from many commits and many runs. You also need a threshold for when the pattern is serious enough to file an issue.

<Plate src="ci-flaky-detector-requirements" alt="Five requirements for a flaky-test detector: GitHub access, a collector for CI results, durable cross-run state, a schedule, and a remote runtime that works after the laptop is closed." fig="FIG.01" width="wide" />

I used two tools for this.

Pi is a programmable sandbox that we can shape around our needs. It offers commands, extensions, and an SDK for controlling agent sessions. I used it as a workshop for building the solution itself and a mechanism for future integrations.

Kody gives that solution a remote home. It is a personal assistant that connects us to GitHub, stores state in Storage, runs on a schedule, and uses a Cloudflare runtime.

Pi helps us build the behaviour. Kody runs it after the laptop is closed.

## Pi meets Kody

Pi is built on primitives, which lets us adapt the agent in almost any direction. We can extend its core by installing packages for things like subagents, web access, plan mode, or MCP, which we will use to connect to Kody.

MCP usually brings external-service integrations to mind. Kody uses it differently. Once connected to an agent, Kody exposes only two tools:

- `search`: finds available capabilities
- `execute`: runs TypeScript code in the Kody runtime

Before using them, though, we need to connect to GitHub. Kody offers a [native integration](https://kody.codes/guides/github) that makes this much easier.

Connecting GitHub took a moment. I did not need to register my own OAuth App or hand a token over to Pi. Kody keeps all credentials and exposes only a secure way to connect to GitHub.

To make sure the connection works properly, we can run the code below through `execute` in Kody's remote environment.

<Plate src="github-integration-smoke-test" alt="A TypeScript smoke test that creates an authenticated GitHub fetch, calls the user endpoint, fails on a non-success response, and returns the connected user's login." fig="FIG.02" width="wide" />

The response is `{ login: "olafsulich" }`.

## From experiment to package

Connecting to GitHub is only the beginning. We are authenticated, but we still need to pull out information about failing tests.

Through MCP, Pi calls `execute` to run code and check what works. That is how, one thread at a time, we work our way towards the solution: inspect completed runs, locate artifacts, download them, and parse the test results.

Along the way, we discover things that are easy to miss at first. Workflow status is not the source of truth, a green workflow can hide a failure, and a red one does not tell us which test case failed. JUnit is uploaded independently of the workflow result and records the outcome of every test case.

<Plate src="ci-github-failure-collection" alt="Five steps for finding flaky tests in GitHub Actions: authenticate, list completed workflow runs, locate the uploaded JUnit artifact, download and parse it, then compare outcomes across runs." fig="FIG.03" width="wide" />

After enough iteration, we have a working and tested solution. We can turn the script into a Kody Package, a small project with code, tests, and a manifest. The manifest tells Kody what the package exposes and what it is allowed to run.

Instead of creating a skill that makes the agent fetch and check flaky tests every time, we give it a ready, deterministic capability that it can use when needed. That saves tokens and gives us a more reliable solution.

## Giving the package a life of its own

The package is ready. Now we need to run it repeatedly, collect results, and take action based on them. Kody handles all of that.

With a Job, we set the schedule for Kody to trigger an action. For the flaky test detector, Kody runs the script we prepared earlier in Cloudflare every three hours. These runs have no model involved. They only execute a process we have already tested.

We also need to persist results between runs. A test is only considered flaky after several runs, so we use Kody Storage, a private database. It tracks which tests have failed and how many times.

<Plate src="kody-storage-converge" alt="Three scheduled Job runs write results, read stored history, and check the failure rate before a later run opens a GitHub issue; all observations converge in private Kody Storage." fig="FIG.04" width="wide" />

When a test crosses the configured failure threshold, the detector opens a GitHub issue listing the failing tests. For each one, it reports how many times the test ran and its failure rate.

## A command, not a checklist

Before publishing and configuring the whole failed-test detection workflow, we still had a few things to do.

<Process>
<Step title="Verify the author contract">
Check the README, title, and a suitable description.
</Step>
<Step title="Run the local tests">
Run the package's local test suite.
</Step>
<Step title="Create the Kody package repository">
Create a package repository on Kody to hold the source code.
</Step>
<Step title="Publish with the schedule disabled">
Publish the package with the schedule disabled, then check the output and any approvals waiting for secret access.
</Step>
<Step title="Invoke and inspect manually">
Invoke the code exactly as the scheduler will later, inspect the result, and confirm that the expected values were saved to Storage.
</Step>
<Step title="Enable the schedule">
Enable the schedule and verify that the whole flow works.
</Step>
</Process>

Every step matters if we want the final workflow to work reliably. It is still a lot to do manually every time we add a package.

We could tell the agent which steps to take, but that would not give us confidence in the final result. It is better to make the process deterministic.

For that, we use a Pi extension that does all the work for us. Under the hood, the command runs deterministic code: it takes a ready package directory and moves it to production the same way every time.

<Plate src="kody-graduate-command" alt="A comparison of a prompt checklist and the /kody-package-graduate command: fixed gates, an independent reviewer, a disabled-first proof, and no remote changes when any check fails." fig="FIG.05" width="wide" />

Before it finishes the configuration, `kody-package-graduate` reviews the package code, manifest, and declared configuration in a completely separate session. This is possible through Pi's `createAgentSession`.

Pi creates a new agent session with a tailored prompt, restricted tools, and no additional extensions or skills. The reviewer's job is to check the acceptance criteria and look for potential gaps. Because it is a separate session, the agent does not inherit the context of the main thread.

`kody-package-graduate` makes it much easier to create and release new packages. Let us make a new `github-issue-prioritizer` package for triaging open GitHub issues.

It fetches open issues from GitHub, sends the title, description, and existing labels to a model, then classifies each issue by type and priority. The result can look like this:

<Terminal label="classification result">
```json
{
  "category": "bug",
  "priority": "high",
  "confidence": 0.91
}
```
</Terminal>

Based on that result, the package adds controlled labels such as `ai:priority:high` and `ai:type:bug`. Existing labels added by people remain untouched. It stores the decisions in Kody Storage so later runs retain the history of earlier classifications.

<Plate src="github-issue-prioritizer" alt="Five steps turn open GitHub issues into a repeatable capability: collect issues, classify them, stop on low confidence, add controlled labels, and keep the decisions in history." fig="FIG.06" width="wide" />

While working with Pi, we used `execute` to explore the Kody API and test our assumptions. The package code was written locally, then we ran:

<Terminal label="graduate the package">
```sh
/kody-package-graduate ./packages/github-issue-prioritizer --job prioritize-issues
```
</Terminal>

The command validated, reviewed, and released the package. Soon after, it was published on Kody, its job was enabled, and it could run independently of an active Pi session.

## Close the laptop

We connected two great tools. Pi gives us a programmable environment where an agent can create and test its own behaviour. Kody gives that behaviour a remote runtime, integrations, state, and a schedule, so it can keep running after the laptop is closed.