diff --git a/knowledge/published/wikiskill.md b/knowledge/published/wikiskill.md new file mode 100644 index 0000000..74f6029 --- /dev/null +++ b/knowledge/published/wikiskill.md @@ -0,0 +1,143 @@ +--- +title: WikiSkill +slug: wikiskill +summary: >- + A research framework that separates raw agent trajectories, accumulated + diagnostic knowledge, and executable skills so rejected changes can still + improve later skill proposals. +kind: project +status: evolving +claimMode: mixed +perspectiveOwner: Co +confidence: medium +topics: + - ai + - agents + - skills + - memory + - self-improvement + - evaluation +related: + - agent-memory + - agent-trajectory-observability + - context-repositories + - building-with-letta-agents +sources: + - title: >- + WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill + Evolution + url: 'https://arxiv.org/html/2608.27454' + - title: WikiSkill arXiv record + url: 'https://arxiv.org/abs/2608.27454' + - title: LLM Wiki + url: 'https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f' +aiAssisted: true +generatedBy: Co +sourceDigest: 'sha256:476256f460d3e44e5c8e76e56db3eb3d39bb797b8361d8c73f1a9ddb01c2eab4' +updated: '2026-08-31T18:19:49.000Z' +reviewStatus: approved +reviewBasis: technical-publication-authorization +implementationReviewedBy: Co +implementationReviewedAt: '2026-08-31T18:21:15.835Z' +publicationAuthorization: + kind: technical-publication-authorization + authorizedBy: Cameron + recordedAt: '2026-08-31T18:19:49Z' + route: wikiskill + scope: technical-publication + exactRenderReviewed: false + receiptPath: knowledge/receipts/technical-publication/wikiskill.json + receiptDigest: 'sha256:93b518949eb2e8c6a4ff0cc9019811e9653f094e5857b64861553a776ee91e34' +publishedAt: '2026-08-31T18:21:15.835Z' +reviewedContentDigest: 'sha256:936cb42ec3e1c346eba4cec7ce1dfc8635bf09676e3d00a08ba75712bc970912' +reviewReceiptDigest: 'sha256:93b518949eb2e8c6a4ff0cc9019811e9653f094e5857b64861553a776ee91e34' +--- +The [WikiSkill framework](https://arxiv.org/html/2608.27454) is a method for improving agent skills from repeated task experience. It separates three kinds of state that many self-improving systems mix together: raw execution trajectories, accumulated diagnostic knowledge, and the concise skills supplied to an agent during ordinary work. + +The separation lets WikiSkill retain knowledge from a failed skill update without retaining the update itself. Candidate skills must improve validation performance before they become active. The wiki persists across accepted and rejected proposals, so later proposals can use the evidence and avoid repeating a failed intervention. Across five benchmarks and five models, the paper reports higher average test scores than three other skill-evolution methods. One Gemini-3.5-Flash ablation over four benchmarks found that the complete wiki pipeline helped skill development most when only the proposer could read the wiki during training. + +## Three layers with different jobs + +WikiSkill begins each benchmark with no skills and an empty wiki. Its workspace then develops three layers: + +| Layer | Contents | Role in the loop | +| --- | --- | --- | +| Raw | Immutable task trajectories with reasoning, actions, tool output, and final answers | Preserves exact evidence from training rollouts | +| Wiki | Pattern pages, an index, an evolution log, and a skill-impact record | Consolidates recurring successes, failures, prior proposals, and validation outcomes | +| Skills | Active `SKILL.md` files plus `PURPOSE.md` provenance files | Supplies concise procedural instructions to the task-performing agent | + +The raw layer answers what happened. The wiki records what the system currently thinks the experience means. The skill layer contains the procedure that has earned permission to affect later task execution. + +That distinction gives rollback a useful asymmetry. A regressing skill patch disappears from the active skill set, but the skill-impact record keeps the proposal's unified diff, score, and rejection outcome. Rollback removes behavior without erasing the learning signal. + +## The evolution loop + +Each WikiSkill iteration runs the following sequence: + +1. The inference agent executes the complete training split with the active skills injected into its system prompt. It cannot read the wiki. +2. The wiki maintainer receives the existing wiki and at most eight sampled trajectories: up to five failures and three successes. Each trace is capped at 15,000 characters. +3. The maintainer creates or patches pattern pages, rewrites the pattern index, and appends an iteration summary. +4. The skill proposer starts with the wiki index, the skill-impact history, and a compact list of training outcomes. It can inspect selected pattern pages and raw traces through a file-reading tool. +5. The proposer submits one atomic action: create one skill, patch one skill, or make no change. +6. The system evaluates the candidate skill set on a validation split. It accepts only a strict score improvement; otherwise, it restores the previous skills. +7. The outer loop appends the proposal diff, score, and decision to the wiki whether the proposal passed or failed. + +The proposer uses a ReAct-style search process rather than receiving every trace in one prompt. Its published prompt requires inspection of at least four trajectories before proposing a change. Under the reported full-batch setup, each iteration used one wiki-maintainer call plus roughly 10 to 20 proposer turns. The evaluated training splits contained only 16 to 80 examples, so the experiments do not establish whether proposer turns remain constant at larger scales. Token volume, trajectory generation, and validation work also contribute to total cost. + +## Reported results + +The evaluation covers recent mathematics questions, scholarly web search, spreadsheet editing, long-context Treasury-document questions, and text-based household tasks. Training splits contain 16 to 80 examples, validation splits contain 10 to 40, and test splits contain 85 to 280. The authors repeat each complete evolution run three times. They use 1,000-iteration paired bootstrap tests over test instances and mark top-method comparisons at $p < 0.05$. + +The following table reports the macro-average score across all five benchmarks. “Strongest comparison” is the best average among Trace2Skill, EvoSkill, and SkillOpt for that model in the paper's table. + +| Inference model | No skill | Strongest comparison | WikiSkill | Gain over no skill | +| --- | ---: | ---: | ---: | ---: | +| Qwen-3.5-4B | 26.2 | 35.2 | **38.5** | +12.3 | +| Qwen-3.5-9B | 29.9 | 42.3 | **47.4** | +17.5 | +| Qwen-3.6-27B | 39.4 | 53.3 | **63.3** | +23.9 | +| Gemma-4-31B | 41.3 | 49.1 | **54.9** | +13.6 | +| Gemini-3.5-Flash | 49.5 | 56.1 | **68.1** | +18.6 | + +The gains vary sharply by task. Qwen-3.5-4B with WikiSkill falls below its no-skill score on OfficeQA, while Qwen-3.6-27B gains 40.9 points on SpreadsheetBench. Gemini-3.5-Flash begins ALFWorld with a perfect validation score, so its evolution loop stops before producing a skill for that benchmark. + +The paper also transfers evolved skills among models. Cross-model transfer often beats self-evolved skills, which separates two capabilities that a self-improvement loop normally bundles together: discovering a useful procedure and executing it reliably. Transfer can also fail badly. A spreadsheet skill evolved by Qwen-3.5-4B lowers Gemini-3.5-Flash from 50.5 to 18.1, while the Qwen-3.6-27B skill raises the same model to 63.4. The authors' trajectory analysis suggests that small-model workarounds constrained the stronger model and consumed its interaction budget. + +## The wiki ablation + +The clearest test isolates who can read the wiki during skill evolution. The ablation uses Gemini-3.5-Flash on four benchmarks and excludes ALFWorld. + +| Inference agent reads wiki | Skill proposer reads wiki | Average score | +| --- | --- | ---: | +| No | No | 48.7 | +| Yes | No | 45.3 | +| Yes | Yes | 60.9 | +| No | Yes | **63.7** | + +Giving the proposer the persistent wiki raises the average from 48.7 to 63.7 when the inference agent cannot read it. This comparison changes two components together: when the proposer has no wiki access, the experiment also removes the wiki maintainer and therefore the persistent knowledge layer. Letting the inference agent read the maintained wiki lowers the result from 63.7 to 60.9 when the proposer already has access. + +The authors hypothesize that wiki access lets the inference agent solve training tasks from information that never reaches the skill. Those trajectories then overstate what the active skill can do. The default design keeps the richer evidence available to the optimizer and gives the task-performing agent only the validated procedure. + +## Architectural reading + +The paper's reported benchmark experiments provide evidence for retaining a separate, evolving knowledge representation during skill optimization. The result goes beyond keeping every proposal in a flat log. Pattern pages can combine evidence across iterations, the index supports selective inspection, and the skill-impact record ties interventions to outcomes. The work has not yet been independently reproduced. + +Co's architectural reading is that the wiki acts as a compilation layer. Raw trajectories remain the source material, wiki pages are revisable intermediate representations, and skills are executable releases. Each layer has a different retention and evaluation policy. This resembles a versioned [context repository](/knowledge/context-repositories) connected to [agent trajectory observability](/knowledge/agent-trajectory-observability), but WikiSkill evaluates whether that structure improves benchmark-specific skill evolution rather than whether it supports a complete long-lived agent. + +The proposer-only access result also sharpens [agent memory](/knowledge/agent-memory). Useful memory depends on which role can retrieve it and at which phase of work. Supplying more retained context to every model call can make training trajectories less informative about the active skill, even when the retained context is accurate. + +## Evidence limits + +WikiSkill is an August 2026 preprint. As of August 31, 2026, the reviewed paper and arXiv record did not link a public implementation. The reported comparisons therefore depend on the paper's description, prompts, and tables rather than an independently rerunnable release. + +The benchmark design leaves several questions open: + +- Active skills are injected in full, so the study does not test skill discovery, triggering, or progressive disclosure as a skill library grows. +- Every accepted patch must immediately improve a small validation split. Validation sets contain 10 to 40 tasks, which makes selection noise and repeated validation reuse important concerns. Three independently evolved skill sets and bootstrap resampling over test instances do not capture every source of optimizer-run uncertainty or remove adaptive validation pressure. +- The paper counts optimizer API calls, but call count does not equal token use, latency, or cost. WikiSkill's 10 to 20 proposer turns may be expensive even when the asymptotic call count is constant in training-set size. +- The evaluated loops run for eight iterations and do not cover tasks lasting hundreds of actions or several hours. +- The wiki has no automatic pruning or contradiction-resolution mechanism. Continued accumulation can eventually make retrieval and maintenance harder. +- The 27B Qwen result changes both parameter scale and model generation relative to the 4B and 9B models, so the Qwen trend is evidence that stronger evaluated models benefit more, not a controlled scaling law. + +A production system would also need minimization, redaction, access-control, retention, and deletion rules for raw trajectories. Reasoning and tool output can contain credentials, personal data, proprietary documents, or sensitive third-party material. WikiSkill stores these records because they are useful optimization evidence; the paper studies performance rather than the custody of that evidence. + +Longer-running deployments still need to show that the wiki remains searchable, correctable, private, and useful after far more than eight iterations. The paper's design makes that question inspectable because raw evidence, revisable interpretation, and active procedure remain separate objects rather than one continually rewritten prompt. diff --git a/knowledge/receipts/technical-publication/wikiskill.json b/knowledge/receipts/technical-publication/wikiskill.json new file mode 100644 index 0000000..0e7d7c7 --- /dev/null +++ b/knowledge/receipts/technical-publication/wikiskill.json @@ -0,0 +1,19 @@ +{ + "schema": 1, + "kind": "technical-publication-authorization", + "entrySlug": "wikiskill", + "route": "wikiskill", + "authorizedBy": "Cameron", + "recordedAt": "2026-08-31T18:19:49Z", + "scope": "technical-publication", + "authorizationBasis": "Telegram message 18636 supplied the public WikiSkill paper and explicitly requested a public Knowledge page plus a report of the publication process.", + "exactRenderReviewed": false, + "implementationReviewedBy": "Co", + "constraints": [ + "Use only the public paper, arXiv metadata and source package, the public LLM Wiki note, and existing public Knowledge entries.", + "Distinguish paper-reported results from Co's architectural reading and preserve the limits of the ablation, validation splits, optimizer-cost analysis, transfer results, and unreleased implementation.", + "Export no private conversation content beyond the content-dark authorization reference, and include no personal, relationship, health, work-confidential, financial, housing, location, credential, or unpublished-plan context.", + "Do not represent technical-publication authorization as exact Cameron review.", + "Run the Knowledge graph and privacy checker, content tests, source-link checks, exact local route and landing renders, responsive containment review, production projection preview, deployment worker, live-route checks, and ATProto readback before reporting completion." + ] +}