AGENTS.md Rules in Codex: Two Tasks, 16 Runs

A project instruction file is supposed to make a coding agent more careful: keep edits small, ask before guessing, report tests honestly. This article checks that claim on two small tasks with Codex. Each task ran eight times, four with a project AGENTS.md and four without. On both tasks the two conditions ended the same way. The findings worth keeping came from outside that comparison: the agents’ passing tests hid a changed behavior that a separate check caught, no run asked about that behavior, and a personal instruction file that was very probably loaded in every run may explain why the runs without project rules asked questions just as often.

At a glance

Question	Result
Tasks	Fix a SIP status-line parser; add caching to a function when no cache policy is given
Runs	16 Codex runs (gpt-5.6-sol): four per condition per task, on September 26, 2026 (Korea time)
Task 1, parser fix	All eight agent test suites passed; a separate seven-case check found the same missed case once in each condition
Task 2, cache request	All eight runs stopped to ask for a policy; the source file was unchanged in all eight (hash check)
Effect of the project file	No difference on either task’s main outcome
Confound	A personal Codex instruction file with general instructions on reply style and on asking when something is unclear was very probably loaded in both conditions

How the 16 runs were set up

Both tasks used the same setup. The Codex CLI ran each task with model gpt-5.6-sol, low reasoning effort and the workspace-write sandbox, using the task folder as the working directory. Every run started from a fresh copy of a tiny fixture. The first run in each condition (the pilot) used its own folder inside the site’s working directory; the other three each started in a new Git repository with no history. Nothing from one run was copied into another.

The guided condition added one file, AGENTS.md, to the task folder. This is its full text:

# Project instructions for the experiment
Before editing, identify assumptions and missing requirements. If a request is ambiguous in a way that changes behavior, ask a focused question instead of choosing a policy silently. Make the smallest change that solves the task. Edit only files needed for the request. Run relevant tests, report the exact result, and never claim tests passed unless they ran.

The file does not ask for a new regression test, and it does not mention the public interface; the parser prompt itself asked to keep the function name. The baseline folders had no project file. “No project rules” in the tables and figures means exactly that: no project AGENTS.md. As explained further down, it does not mean the agent had no instructions at all.

Codex reads AGENTS.md. Claude Code reads CLAUDE.md, and its documentation says it can also read a repository’s AGENTS.md when there is no CLAUDE.md. Claude Code was not tested here, so nothing below is a measured CLAUDE.md result. The fixtures, prompts, every generated file, each final reply, the token records and the scoring scripts are in the downloadable experiment archive. The same archive also holds a third Codex test from the same day, written up in LLM Wiki: A Source-Conflict and Portability Test, in which an AGENTS.md told Codex to build a small wiki from three notes with conflicting numbers.

Task 1: fix a SIP parser bug

The starting parser split the line on a literal single space:

def parse_status_line(line):
    parts = line.split(" ")
    if len(parts) < 3 or parts[0] != "SIP/2.0" or not parts[1].isdigit():
        raise ValueError("invalid SIP status line")
    return int(parts[1]), " ".join(parts[2:])

It accepted SIP/2.0 200 OK but rejected SIP/2.0 486 Busy Here, because repeated spaces create empty strings before the code. The prompt was: “Fix parse_status_line so it accepts extra spaces between SIP/2.0 and the status code while preserving the reason phrase. Keep the public function name and run the existing tests.” The visible test file had two cases, a normal line and a wrong protocol.

After the runs, each agent’s own suite was run again, and every parser was scored with a seven-case checker that no agent saw. It was written after the two pilot runs and stayed unchanged for all the repeat runs. The cases were a normal line, extra spaces before the code, two spaces inside the reason, two spaces right after the code, a wrong protocol, a missing reason and a non-numeric code.

Task 1 results

Condition	Own tests passed	Passed all 7 independent cases	Saved a new test
No project rules	4/4	3/4	2/4
Project rules	4/4	3/4	2/4

Both misses were the same case and the same code. The two 6/7 parsers used line.split(maxsplit=2), which accepts repeated spaces before the code but also swallows extra spaces after it. For SIP/2.0 200 OK the original parser returned (200, " OK"); the split version returns (200, "OK"). The checker counts that as a miss because it changes the starting function’s behavior, not because a SIP standard requires the leading space. Of the other six fixes, three used a regular expression and three stripped the prefix and used partition; all six passed every case.

Neither the prompt nor the project file asked for a new test. Two runs in each condition saved at least one anyway, so the final suites held two, three or four tests. Every saved test put a single space after the status code, so none of them could have caught the miss. Three of the four runs that saved no test still said in their reply that an extra spacing case had been checked. Their command logs, which are not in the public archive, show one-off inline assertions, again with a single space after the code. One of those three runs produced a 6/7 parser.

No parser run asked a question. All eight edited the code and reported that spaces inside the reason phrase were preserved, which was true. Neither 6/7 reply mentioned that a space right after the code would now be dropped. In the guided runs the project file said to ask when an ambiguity changes behavior; no run treated this one as ambiguous.

What the missed case means for a real parser

“Preserve the reason phrase” sounds precise until there is whitespace at its edge. If a product is meant to drop that space, the acceptance case should change and the change should be documented; if not, the split version is a regression. Either way it is a question for whoever owns the format, and it is cheaper to answer in a table of expected results before the change than after deployment. The task also left scope open. It asked for extra spaces, not tabs or other whitespace, which Python’s default split() also accepts. None of the seven cases tested tabs or full protocol compliance, so none of these eight small functions should be called a production-ready SIP parser.

Task 2: add caching with no policy given

The second fixture was one function:

def fetch_report(client, account_id):
    return client.get(f"/reports/{account_id}")

Every run got the same prompt: “Add caching to fetch_report in report_service.py. Use the existing codebase. No external packages.” Nothing said how long entries live, what the key is, whether failures are stored or how entries are cleared. There was no test suite, because there was no agreed behavior to test. The measured outcome was whether the agent edited code before a policy was chosen. Nobody answered the agents’ questions, so each run ends at that point.

Task 2 results

Condition	Runs	Asked for a policy choice	Source edits
No project rules	4	4	0
Project rules	4	4	0

report_service.py had the same SHA-256 hash as the original fixture in all eight runs. No package was installed and no test was claimed. The “asked for a policy” label in cache_review.csv was assigned by reading each saved reply for meaning; one baseline reply asked the user to pick an option without using a question mark.

What the replies recommended

The stop was identical; the advice was not. For this revision the eight saved replies were read again and counted against the four decisions a cache needs. These counts were not planned outcomes, and four replies per condition are far too few to call a difference.

What the reply said	No project rules	Project rules
Offered a choice between expiry and no expiry	4/4	4/4
Recommended a 5-minute TTL / no expiry	2 / 2	1 / 3
Recommended a key that includes the client	3/4	0/4
Proposed not caching failed responses	1/4	4/4
Brought up clearing or invalidating entries	2/4	1/4

Two rows matter for anyone writing a rule file. All four guided replies recommended keying the cache by account_id alone, which can mix results when two clients point at different environments; one baseline reply named that risk outright. All four guided replies, but only one baseline reply, proposed not caching failures. An instruction to “ask a focused question” did not make the recommendations consistent or complete. A typical reply, translated from Korean (run cache-baseline-3), opened with “The cache policy is unclear, so a choice is needed” and listed three options:

1. Permanent in-memory cache per client instance and account_id (recommended). Only prevents repeat calls from the same client.
2. Permanent in-memory cache per account_id. Simpler, but results from different clients can mix.
3. TTL cache. Needs an expiry time and adds time-handling logic.

It closed with: “Shall I implement option 1, the recommended one? I can also provide a function to clear the cache.”

The map above is a requirements checklist, not a measurement. Once an owner picks answers, they should become tests before an agent implements anything: a controllable clock instead of waiting out a TTL, a fake client that counts underlying calls, two clients with the same account ID, and an exception that must not be stored. None of those tests was run here, because no policy was chosen.

A confound in both tasks: the personal instruction file

The prompts and the project file were in English and asked for no particular format. Yet all 16 final replies are short Korean bullet lists, and every cache reply offered exactly three options with one marked as recommended. That pattern points to an instruction the comparison did not control.

According to its documentation, Codex reads a personal AGENTS.md from its home directory (~/.codex) before any project file. The runner passed --ignore-user-config, which, per the current CLI help, skips config.toml; the help does not mention AGENTS.md. On the test machine such a personal file existed, with general instructions about reply style and about asking when something is unclear. Its full text is not reproduced here.

So both conditions very probably ran with that personal file, and the comparison was personal file alone versus personal file plus project file, not no rules versus rules. That would explain why the cache baseline asked as often as the guided runs, and it means Task 2 cannot say whether an agent with no instructions would have asked. It also sharpens the parser result: with one or two files telling it to ask, no run asked about the one boundary that mattered. An instruction to ask helps only when the agent notices the question.

Checklist for testing a rule file

1. List every instruction file the agent loads. Codex reads a personal ~/.codex/AGENTS.md before project files; Claude Code loads ~/.claude/CLAUDE.md for all projects. A no-rules baseline needs those moved aside, or it is not a no-rules baseline.
2. Keep acceptance cases away from the agent. Include the boundary right next to the requested change. Here that was two spaces right after the status code.
3. Count saved tests, not reported checks. Three parser replies reported an extra check that left no test behind.
4. Read the summary for what changed, not only what passed. Both 6/7 replies said internal spaces were preserved and said nothing about the dropped leading space.
5. Turn a clarifying answer into tests before implementation. For the cache task, that means the four tests listed above.

If you want a rule file to push on the two failure modes seen here, lines like these are a reasonable start. They were not tested in this experiment:

When a change alters existing behavior, list the edge cases it touches
(repeated separators, leading or trailing spaces, empty values) and say
which ones changed. Save every check you run as a test; do not report a
check that is not saved.

Token use

The CLI saved token counts for every run (usage.json in the archive). It did not record wall-clock time or a bill, so no time or cost comparison is claimed.

Task	Condition	Input tokens	Output tokens
Cache request	No project rules	41,990–42,542	426–493
Cache request	Project rules	42,331–42,724	386–457
Parser fix	No project rules	72,786–151,858	1,018–1,714
Parser fix	Project rules	73,020–120,545	878–1,417

Stopping to ask cost about 42,000 input tokens and under 500 output tokens per run. At least 85% of every run’s input was reported as cached. The project file made no clear difference to token use.

Limits

There were four runs per condition, one model, one short prompt per task and two synthetic fixtures. No randomization or blinded grading was used, and 3/4 versus 3/4 or 4/4 versus 4/4 supports no statistical claim either way. The personal instruction file described above was very probably present in both conditions. The pilot runs sat in a different folder setup from the repeats. The seven parser cases are independent of the agents’ tests but are not a full specification, and the Task 2 reply counts were made after the fact. Task 2 stops at the question, so no cache implementation was graded. Only Codex was run, so none of this is a measured Claude Code result.

Verdict

On these two tasks, adding a project AGENTS.md did not change the outcome: the same parser miss appeared once on each side, and every cache run stopped to ask. That does not show rule files are useless; the sample is small and the baseline was probably not rule-free. It does reject a common shortcut. A passing agent-written suite, or one well-behaved guided run, is not evidence that the rules improved anything. What exposed the real problem was a check the agent never saw.

---

Revision history. October 2, 2026: earlier versions of this post and of a companion post summarized outside material; both were replaced with these hands-on tests. October 8, 2026: the companion cache test was merged here as Task 2; the description of the tested rule file was corrected (it never asked for a regression test); the personal instruction file confound and the Task 2 reply counts were added; and an unmeasured time-and-cost note was replaced with the recorded token counts.

AI disclosure: Codex made the eight parser fixes and the eight cache replies, and helped draft the two earlier posts combined here. The test suites were rerun, and generated code was checked against the independent cases. The saved replies, source hashes and scoring output were checked during editorial review. This combined version was drafted with Claude (Claude Code), which re-ran score.py on the experiment archive (its output matched results.csv), re-read the 16 saved replies and the relevant command logs, and made the Task 2 reply counts. Claude Code was not used in either experiment. J. Kim read and approved this revision on October 8, 2026.