LLMs are prized for their thinking. In September, OpenAI said its models had proved that the Navier–Stokes equations can blow up in finite time, for a fluid pushed by a smooth outside force; mathematicians are still checking it. In October it released 722 maths manuscripts, many with machine-checked proofs.
But most of what an agent does all day isn’t discovery. It is the same procedures, over and over: list the projects, count the files, run the tests, write the report. Each time, the model works the procedure out from scratch and pays for that thinking again, in time and in tokens.
Software developers solved this long ago. A procedure you repeat becomes a function, a module or a script: written once, tested, then called without anyone re-deriving it. RamenSOPs brings that to agents. When an agent repeats a procedure, it is stored as a tested SOP. The next time, the agent retrieves it and runs it, with no thinking in between, faster than the model could think it through.
The saving comes on reuse: write and test the procedure once, then call it many times.
We measured whether that pays off. With an SOP, a repeated job in the Rameness agent harness got 44% faster and used 48% fewer tokens. But over eight tasks, learning after every run cost as much as the SOPs saved: 64 minutes in all against 61 without learning, and most of that cost came after the useful SOP already existed.
What an SOP is
An SOP (standard operating procedure) is a tested procedure the agent calls like a tool, so a job it has done before runs as code instead of being worked out again by the model. Each one is a small folder: an interface, a script, and the tests that prove it works. SOPs come from three places: built into Rameness, pulled from the shared RamenSOPs registry, or learned privately on your machine.
Rameness learns them at the end of each successful run:
- Review. The model looks back over its last five runs and names multi-step jobs it repeated.
- Score. The decision model judges how likely each is to come up again.
- Worth it? A value check drops anything that wouldn’t save at least 500 tokens per use, or wouldn’t repay the roughly 4,000 tokens it costs to write and test.
- Already exists? Near-duplicates of existing SOPs are skipped or merged.
- Write, test, repair. The model writes the script and its tests, with up to three rounds of repair.
- Register. It is validated if its tests pass, otherwise kept as a candidate.
That end-of-run step is the overhead this post measures, set against what the SOPs save on later tasks.
The experiment
We gave Rameness the same procedure eight times and measured three setups. Every one of the 24 runs scored 100%, so the comparison is purely about cost.
The task (projreport). Each task is a folder of six small Python projects. The agent writes report.json with each project’s number of .py files, line count, TODO comments, and whether its tests pass, checked against a hidden answer by 25 checks. The procedure repeats six times within a task and again in every task, so it is an ideal case for learning once and reusing. It is synthetic and easy on purpose: it measures reuse, not general skill.
| Setup | Learning | Between tasks |
|---|---|---|
| Control | off | fresh start for every task |
| SOP pipeline | on, after every run | learned SOPs and run history carry over |
| Trusted SOPs | on, after every run | as above, but only validated SOPs are offered, and the agent is told not to re-check their output |
All runs used Qwen3.8-Flash-Next (3-bit AutoRound) on one local GPU, on 2026-10-08. We split each run at the agent’s last turn: everything before is task time, everything after is learning.
Results
Rameness bench run records, projreport1–8, 2026-10-08; one series per setup.
The SOPs themselves work. With trusted SOPs the eight tasks took 34.5 minutes of work against 61.3 without, and 155k generated tokens against 299k: 44% less time and 48% fewer tokens. But learning after every run cost 29.9 minutes and 186k tokens, so over eight tasks the total came to 64.4 minutes, slightly more than never learning at all. The first version of the pipeline, which offered every SOP it learned and let the agent re-check their output, came to 80.4 minutes.
Time block compression
The saving comes from compressing one block of each task. Without the SOP, the model spent about 45 seconds and 5,310 tokens of generation working out the inventory and its report. With it, the same block became a call the model wrote in about 3 seconds, and a script that ran in under half a second.
Nine calls of dev.python_project_inventory_report from projreport4–8 in the trusted SOPs series.
Whole tasks got 44% faster, not 94%, because the procedure is one block among several. Reading the task, checking the result and finishing up are still model work, and the agent kept spending time on them.
Where the overhead goes
The learning that paid off was cheap. The cost came from learning that kept running after there was nothing new to learn.
Initial overhead: 6 minutes. Learning after task 1 took 0.4 minutes and found nothing yet, since one run isn’t a repeat. After tasks 2 and 3 it spent 6.2 minutes and 38k tokens and produced the SOP that did the work for the rest of the series: a validated, 396-line script that inventories Python projects and writes the report.
Per-run overhead: 23 minutes for nothing. Tasks 4 to 8 spent another 23.3 minutes and 146k tokens learning, and learned nothing:
| After task | Learning | What happened |
|---|---|---|
| 4 | 6.1 min | Proposed a variant of the existing SOP; extending it and a new version both failed their tests |
| 5 | 0.4 min | Two small procedures, skipped as not worth an SOP |
| 6 | 7.4 min | One more variant failed its tests; two generations failed on malformed output |
| 7 | 9.1 min | Two variants failed their tests; one generation failed on malformed output |
| 8 | 0.3 min | Nothing proposed |
The review kept rediscovering the procedure it had already turned into an SOP, under new names, and the duplicate check sent each one to be merged into the existing SOP, which then failed. When the review found nothing worth trying (tasks 5 and 8), learning cost under half a minute.
What it learned
The first pipeline learned eleven SOPs for what was really one procedure. Trusted mode learned two.
| Setup | SOPs learned | Validated | What they were |
|---|---|---|---|
| SOP pipeline | 11 | 1 | Near-copies of one job under different names: fs.project_audit, fs.audit_report, fs.python_project_audit, fs.audit_python_projects, projects.verify_audit_report, projects.validate_report and five more |
| Trusted SOPs | 2 | 1 | dev.python_project_inventory_report (validated) and dev.generate_and_verify_report (candidate) |
In the first pipeline, unvalidated candidates were offered to the agent, and it tended to re-check an SOP’s output by hand, paying for the SOP and for the work it was meant to replace. Trusted mode offers only validated SOPs, with a note that their tests pass. Between the two series, task time went from 36.4 to 34.5 minutes and learning from 44.0 to 29.9.
The validated SOP is a real tool, not a stub: 396 lines that find the project folders, count files, lines and TODO comments, run each project’s tests, write the report, and cross-check it.
What it means
SOPs pay off on the tasks themselves: 44% less time and 48% fewer tokens, at the same accuracy. Learning is what doesn’t pay yet, and almost all of its cost is the per-run part, not the one-off part. Had learning stopped proposing variants of a procedure it already had, the trusted series would have taken about 41 minutes for eight tasks instead of 64, against 61 with no learning.
The work, then, is in the learning loop rather than the SOPs: recognise a procedure that is already covered before trying to extend it, and spend less on learning once a run brings nothing new.
Caveats:
- One series per setup. Task times vary a lot from run to run: the control’s ranged from 3.7 to 13.2 minutes.
- Easy, repetitive tasks. This is close to a best case for reuse. Real work repeats less often and less exactly.
- Not everything is explained. Trusted-mode tasks 2 and 3 were fast before any SOP existed, probably helped by the shared run history and by chance.
- How we split time. Everything after the agent’s last turn counts as learning, including a few seconds of bookkeeping.
The tasks are generated by bench/tasks/projreport/make.py in the Rameness repository; learned SOPs that pass review can be shared through RamenSOPs.