LLM Wiki: A Source-Conflict and Portability Test

What the LLM Wiki pattern is

LLM Wiki is a pattern Andrej Karpathy described in a short design note on GitHub: instead of having a model search raw documents from scratch for every question, you let it keep a folder of interlinked Markdown pages up to date as new material arrives. The note separates raw sources that the model reads but never edits, wiki pages that the model writes and maintains, and a schema file such as AGENTS.md or CLAUDE.md that sets the conventions. Its routine operations are ingest, query and a periodic lint pass for contradictions, stale claims and orphan pages. The pattern is worth trying when you keep adding material on one subject over weeks, such as research notes, a reading log or team decisions, and want earlier conclusions and conflicts to stay visible instead of being rediscovered on every question. For a handful of documents, reading the originals each time may be simpler; the note describes its plain index file as working at moderate scale, around 100 sources and hundreds of pages.

This post tests one property the pattern depends on: whether a generated wiki still gives a reader a working path to its sources after the wiki is copied somewhere else. The short answer is no, unless the sources travel with it. The checklist below turns that result into steps you can apply to your own wiki.

At a glance

Question	Result
Input	Three synthetic notes: proposal 3, staging bug 4, approved limit 2
First Codex output	A source-linked index.md, retry-policy.md and log.md
Copy test	The answer stayed correct, but 18 local link occurrences broke when only wiki/ was copied
Repair	Exact source copies moved inside wiki/sources/; link check returned 0 broken links
Recorded usage	Four Codex runs: 233,404 input tokens (207,616 reported as cached) and 7,639 output tokens

An LLM Wiki is only useful if the next reader can tell a current decision from an older proposal and can still open the cited source. This experiment gave Codex three short notes that mention three different retry counts, then copied the generated wiki to a new folder. The answer survived the first copy. The citations did not. That distinction became the most useful finding.

The notes and all generated files are in the experiment archive. The case is synthetic, not a real customer’s policy or a production incident. Its small size makes every claim and link inspectable. It does not tell us how a wiki will perform after thousands of documents.

Checklist: a wiki that still works after it moves

These seven checks come from what broke and what fixed it in this test. The first six were exercised here. The last one was not, because the notes were synthetic.

1. Keep the originals read-only. Both instruction files told Codex not to change raw/. After the build and repair runs, the three notes in raw/ still matched the fixtures byte for byte.
2. Copy exact sources into the wiki folder. Put them in a subfolder such as wiki/sources/ and compare each copy with its original. Here all three copies matched.
3. Link only inside the wiki folder. A link such as ../raw/2026-09-24-approved.md works only while raw/ sits next to the wiki. In this test such links broke 18 times: 8 in retry-policy.md, 7 in index.md and 3 in log.md.
4. Label every number by status and date. Keep approved policy, drafts and observed behavior apart, and record which document supersedes which. The wiki kept 2 (approved September 24), 3 (September 20 draft, superseded) and 4 (a client bug seen in staging) separate.
5. Run the link check on the copy, not the original. The original layout reported 0 broken links only because raw/ was still beside the wiki.
6. Check the answer and the citations separately. Ask a fresh session a question using only the copied folder, then run a link checker. In the first copy the answer was right and the links were broken.
7. Check who may read each source before copying it. A copied incident note becomes as visible as the wiki itself. This test used synthetic notes and did not cover permissions.

The claim being tested

The pattern claims that a generated wiki can preserve useful, source-backed context across later AI sessions. This experiment split that claim into two checks. First, could a new Codex session answer a question from the wiki without being given the original raw/ directory? Second, would the source links still work after moving the wiki? A response can get the number right from summarized text while leaving its citations unusable. Both properties matter if the wiki is intended to be a durable reference rather than a one-off answer.

The source conflict was constructed to catch a common summary failure: treating the latest observed number as policy. A September 20 design draft proposed three timeout retries. A September 22 staging note observed a buggy client making four. A September 24 approved note set the allowed maximum at two and superseded the draft. An answer of “four retries” would confuse observed behavior with authorization; “three” would promote an unapproved proposal.

Test setup and method

The three raw Markdown files were placed under raw/. Codex was instructed to leave them unchanged and produce three pages under wiki/: an index, a policy page and a chronological log. Every factual statement had to cite a source filename, and the agent had to keep proposal, observation and approval separate. This instruction was in Codex’s AGENTS.md format. A Claude Code project would use its own CLAUDE.md; this run was not a test of Claude Code.

The first session produced the requested pages. Its policy page had distinct headings for the approved policy, earlier proposal and observed behavior. It named the relevant original files. To test persistence, only wiki/ was copied into a second folder. That folder did not contain raw/. A fresh Codex session was asked for the current limit and what the other two numbers meant. It answered two approved, three proposed and four observed, with the correct filenames. At the level of textual recall, the wiki was sufficient for this one question.

The next check examined links instead of stopping at the correct answer. The generated pages used links such as ../raw/2026-09-24-approved.md. In the original workspace, those links pointed to existing files. In the copied folder, ../raw/ did not exist. A small Markdown link checker found 18 broken local link occurrences across the three pages. That number counts occurrences, not 18 distinct source files; there were only three raw notes. The query session could still name filenames because they were written in the wiki prose. A reader could not follow those links in the copied version.

Results: right answer, broken citations

Stage	Correct policy answer	Broken local link occurrences
Generated wiki in its original folder	2 approved; 3 draft; 4 bug	0 in the original layout
Only wiki/ copied to a new folder	Same answer	18
Repaired portable wiki/ copied again	Same answer	0

The first row’s zero depends on the raw directory remaining next to the wiki; it does not mean the wiki was self-contained. The second row exposed that assumption. This is the kind of defect an answer-only evaluation misses: a model can repeat a source name without providing a working trail to the source.

For the repair, Codex copied the three original notes byte-for-byte into wiki/sources/ and changed the links in wiki/*.md to point inside that directory. A file comparison found no difference between copied notes and fixtures. A second copy of the complete wiki/ folder included those source snapshots. The link checker reported zero broken local links, and another fresh Codex session again identified the two-retry approved policy and the older numbers correctly. The repair improved portability without changing the numerical claims.

Why a source snapshot matters

A filename in prose is better than no provenance, but it is not a usable citation if the target is absent. Keeping immutable source snapshots beside the generated pages lets a reviewer compare the summary with the original wording after the wiki is exported or moved. That is particularly useful when a later document supersedes an earlier one. A wiki page may say “current maximum: two,” but the source trail should still show that three was a draft and four was an incident observation.

The portable layout in this small test was:

wiki/
  index.md
  retry-policy.md
  log.md
  sources/
    2026-09-20-design.md
    2026-09-22-incident.md
    2026-09-24-approved.md

In a real organization, copying source documents into a wiki raises access-control and retention questions. A private incident note should not become public just because it is needed for provenance. The experiment had no sensitive data; real deployment would need permissions, redaction rules and a way to remove or update sources lawfully. An internal link checker cannot tell whether the reader is authorized to open a link. Portability is one requirement, not the whole governance model.

How to reproduce the check

The archive contains the three input notes, generated pages, repair prompt, repaired wiki, query replies and wiki_links.py. After extracting it, these commands repeat the mechanical checks:

python3 wiki_links.py runs/wiki-query/wiki
# runs/wiki-query/wiki: 18 broken local links (exit code 1)

python3 wiki_links.py runs/wiki-portable-query/wiki
# runs/wiki-portable-query/wiki: 0 broken local links

cmp fixtures/wiki/raw/2026-09-24-approved.md runs/wiki-portable-query/wiki/sources/2026-09-24-approved.md
# no output means the copy is identical

The same comparison works for the other two notes. These mechanical checks are separate from the model’s answer, which is important: a fluent answer should not certify its own citations.

The follow-up query asked only one factual question with three numbers. A broader evaluation would ask about unrelated topics, partially conflicting updates, revoked policies and inaccessible sources. It would also verify every claim against the raw text, not just whether the page names a plausible file. This demo makes the failure mode concrete without claiming that a three-note wiki is production-ready.

Token usage and cost

Each run saved the usage numbers that Codex CLI reported at the end of the session. All four runs used the gpt-5.6-sol model with the low reasoning setting.

Run	Input tokens	Cached input tokens	Output tokens
Build the wiki	76,047	69,120	1,842
Query the first copy	29,082	24,960	324
Repair the wiki	85,030	74,880	5,078
Query the repaired copy	43,245	38,656	395
Total	233,404	207,616	7,639

The repair used more input and output tokens than the original build. Re-running wiki_links.py or a file comparison yourself uses no model tokens. Wall-clock time and billed cost were not recorded, so this post does not give a price. The usage files for the repair and the second query are in the archive; the first two rows come from the same run folders but were not packaged into it.

Limits of this test

All notes were authored for the experiment. The dates and gateway policy are fictional. Each query session ran once, so a correct answer here does not establish reliability over repeated use. The link check verifies paths, not truth or authorization. The copied source files stayed unchanged in this test, but future updates would need versioning and a clear supersession process. The generated wiki also mixed Korean prose with English source notes, another detail a real team would need to standardize.

No comparison was made with simply reading all raw notes on every query. That approach may be preferable for a three-document set. The wiki pattern becomes interesting when retrieval, continuity or conflict tracking saves enough work to justify maintaining it. This experiment does not measure that break-even point.

The related AGENTS.md rule-file experiment in Codex comes from the same archive and follows the same principle: keep the input, the generated result and an independent check available together.

LLM Wiki verdict

The wiki retained the correct distinction among two approved retries, three proposed retries and four retries observed during a bug. Yet the first copied version had 18 broken source-link occurrences. Moving exact source snapshots inside the wiki reduced that count to zero while preserving the answer. For a persistent LLM Wiki, “the model answered correctly” is necessary but insufficient; the citation trail must survive the way the wiki will actually be used. If you adopt the pattern, treat the source folder as part of the wiki rather than something that sits next to it.

---

Revision history: On October 2, 2026, this page replaced an earlier summary of outside material with the hands-on test above. On October 8, 2026, a later revision added the pattern overview, the checklist and the token-usage table.

AI disclosure: Codex generated and repaired the synthetic wiki, answered the follow-up questions and helped draft the October 2 version of this article. Claude helped restructure this revision and add the pattern overview, checklist and token table. The source copies, saved answers and link-check results were checked during editorial review. The token figures are copied from the saved usage.json files. The example uses no real customer or production records. J. Kim read and approved this revision on October 8, 2026.