AI Agent Trading Prediction Markets — Lessons from 30 Days of Paper

What this article is

Most AI agent articles are theory. This one is receipts.

For 30 days I ran an AI agent called Pip that made decisions on Polymarket, an event-prediction market. Every decision was logged. Every fill was logged. Every gate rejection was logged.

Pip ran in paper mode — no real capital deployed. But the decisions were real decisions against real market state. The agent saw real prices. The agent made real choices. The fills were simulated against real order books. The P&L was a real number against a real-time benchmark.

Pip was an experiment, and it ended one. The desk has since moved off prediction markets to a live stock-trading flagship — the current tape is public at /trading/, run by an agent called Quant. I’m keeping this writeup because the thing that transferred wasn’t the market; it was the architecture. Everything below is why the desk trading stocks today is built the shape it is. This is a writeup of what an AI agent actually does well in a market-decision domain, what it fails at, and the gates that have to exist around it before any of it touches real money — for anyone considering building something similar.

The architecture, briefly

The names are arbitrary; the shape generalizes. Pip is three agents and one middleware:

• Researcher (Claude Sonnet) scans Polymarket events, picks candidates by liquidity and resolution-criteria edge, and writes a brief on each candidate.
• Trader (DeepSeek) reads each brief and emits a BUY / SELL / HOLD decision with a size and a confidence.
• Risk (Claude Haiku) reads each decision and the agent’s current book and emits a small JSON object — accept, reduce, or reject with a reason.
• 14-gate middleware between the trader’s decision and the actual order placement. Each gate is a simple check — position size cap, sector concentration, max trades per day, kill-switch lookups, etc. If any gate trips, the decision is rejected.

The architecture matters less than the rule it encodes: the language model is allowed to want things, but the deterministic middleware decides what actually happens.

What the agent actually does well

After 30 days I am willing to say three things about agent performance with reasonable confidence.

1. Reading resolution criteria

The strongest thing the LLM does in this domain is read the actual resolution-criteria text for each market and notice gaps between what the market is selling and what it will pay out for. Most retail traders read the headline — “Will X happen by Y?” — and place a bet on the vibe. The resolution criteria are often subtler. “Will resolve YES if the official source published before midnight UTC on the deadline date, otherwise NO.” That “otherwise NO” is a real edge. The LLM reads it. The retail trader does not.

This is the single biggest source of paper P&L in Pip’s run. The agent finds markets where the headline price implies one outcome and the resolution criteria imply another, and it bets on the resolution criteria. The price converges. Pip pays out.

2. Calibrated indecision

This sounds backwards but it is the second biggest source of value. The Trader agent is allowed to return HOLD. Most decisions are HOLD. Of roughly 600 candidates Pip’s researcher surfaces per day, the Trader places a position on maybe 5 percent. The rest are HOLD.

A retail trader cannot do this. The retail trader looked up a market because they wanted to bet. The agent does not have ego. The agent says HOLD a lot. Calibrated indecision is one of the few real advantages a market-side AI agent has — it can stop wanting to play when the edge is thin.

3. Cross-checking sector concentration

The middleware Risk layer has a gate that caps Pip’s exposure to any one event category. Sports concentration, weather concentration, crypto-price concentration. The gate has fired four times in 30 days. Each time it was right — the agent was about to over-stake on a sector it was already leaning into, and the middleware caught it. The agent is not capable of remembering its own running position in a structurally reliable way. The middleware is.

What the agent actually fails at

1. Tail-risk reasoning

The agent is bad at tail risk. The agent will reason its way to “this market is 90% to resolve YES” and place a confidently-sized bet, and the 10% case will hit, and the loss will be larger than the agent’s confidence framing would have suggested it was prepared for.

This is the failure mode that every gate in the middleware exists to backstop. The middleware does not trust the agent’s confidence. The middleware sizes by dollar exposure, not by the agent’s stated confidence in the outcome. That is the only thing that saves Pip from being aggressively wrong about size on the days the model is aggressively wrong about the outcome.

If you are building an agent that makes decisions involving money, this is the lesson worth tattooing on the inside of your eyelids: the model’s confidence is not actuarial. Treat it as a hint. Size by exposure.

2. Time-aware updates

The agent is bad at updating on time. A market that resolves in 30 days versus 3 days requires totally different sizing. The agent reasons about probability of outcome reasonably well; it reasons about time decay of position value barely at all. The middleware has to encode this — Pip’s position-size gate is a function of “time to resolution” as well as “expected edge.”

If you are building an agent in this kind of domain, build the time-decay logic in middleware. Don’t ask the model to remember it consistently. It won’t.

3. Cross-market reasoning

If markets A, B, and C are correlated (same political event, same weather pattern, same earnings outcome), the agent will happily place positions in all three because each individually shows edge. The agent does not see “I have already taken this thesis three times in different wrappers.” The middleware has to catch the cluster.

This is one of those failure modes that is invisible until you look at it. The agent’s individual decisions all look right. The portfolio they produce looks like a thesis tripled-down. The middleware fix is a simple correlation check; the lesson is that no individual-decision review will catch it.

The 14-gate middleware

The gates exist because the model cannot be trusted to enforce its own discipline. Each gate is a single check. The decision either passes or it doesn’t. Listed roughly in order of how often they fire:

1. Position size cap — no single trade exceeds N% of capital. Trips constantly.
2. Sector concentration — no sector exceeds M% of book. Trips frequently early in volatile categories.
3. Per-day trade cap — max K decisions executed per day. Trips when the model is too confident the market is exciting.
4. Resolution time horizon — markets resolving in under H hours rejected (the agent makes worse decisions on near-term resolutions).
5. Liquidity floor — reject if order book depth below threshold.
6. Edge floor — reject if implied edge below E percentage points after fees.
7. Kill-switch active — reject all writes if operator has flipped the kill flag.
8. Phase gate — paper mode rejects live writes; live mode rejects writes outside daily-cap budget.
9. Correlation gate — reject if the new position is highly correlated with three existing positions.
10. Volatility surge gate — reject if the market price has moved >X% in last 30 minutes (probably news; wait for re-pricing).
11. Market-already-closed gate — reject if resolution criteria already triggered (paper trips of this caught a real bug in the data pipeline).
12. Self-trade gate — reject if the position would cross Pip’s own resting orders.
13. Stale-data gate — reject if the market data is older than T minutes.
14. Sanity gate — reject if the trade size is negative, the price is outside [0, 1], or any field is null.

That last one is mortifyingly important. The model has occasionally returned null for a field. The middleware catches it. If the middleware didn’t catch it, the order placer might silently substitute zero. Always run a final sanity gate.

What this means for building any AI-agent decision system

The general lessons, generalized away from Polymarket:

Let the model want, let the middleware decide. The split between what the model produces and what the system does with that production is the most important architectural decision. The model can be wrong constantly. The middleware cannot.

Size by exposure, not by confidence. Confidence-weighted sizing sounds smart and is the wrong way to deploy any model whose tail is heavier than its own self-report. Almost every model has this property.

Encode the constraints the model can’t remember. Time decay, cross-asset correlation, daily caps, kill switches. Anywhere the model “forgets” something it just inferred two paragraphs ago, you need a deterministic check.

Run paper before live for a long time. Pip ran 30 days of paper before any consideration of real money. The first 10 days revealed three serious bugs in the data pipeline. The next 10 revealed two correlation bugs in the middleware. The last 10 finally produced clean operating data. The bugs you find in paper are the bugs that would have cost you in live.

Be willing to read the actual tape every day. The single most useful piece of operational discipline running Pip was the daily ritual of opening the tape and reading what the agent did. Most days it was boring. The non-boring days were where the lessons compounded. The desk still runs that ritual — the live version is at /trading/.

The honest result so far

After 30 days of paper, Pip was up roughly $100 on a $200 starting capital. That is a misleading number. It was a small sample with high variance. It did not mean the system worked. It meant the system had not yet not-worked badly enough to die — which, for a prediction-market lane, eventually it did. The edge was too thin to keep feeding, so the desk closed it and moved to stocks. That is the honest arc: run it, read it, kill it when the tape says the edge isn’t there.

The number that mattered more is the gate-rejection rate. Roughly 23% of decisions were rejected by some gate. That is the number that told me the architecture was doing its job — the model was generating decisions, the gates were catching the ones that shouldn’t ship, and the survivors got executed. If the gate rate were 0%, I would distrust the system. If it were 80%, I would distrust the model. 23% is what “the middleware is earning its keep” feels like — and it’s the same discipline the live stock desk runs on now.

If you want to build something like this

The architecture I’d recommend is not specific to prediction markets. The shape — agent generates candidates, agent emits decisions, deterministic middleware filters, sanity gates everywhere — works for any decision domain where the model produces output that has consequences. Customer routing, content moderation, trading, search ranking, ad bidding, anything where “ship the answer” is the action.

If you want me to build something with this shape into your stack, the door is Architect. Brief form, no calls, real architecture.

Or watch the architecture run live. The prediction-market experiment is over; the same middleware-first shape now trades stocks in public at /trading/, run by Quant — every position past-tense, every rule visible. If you’d rather get the plain-English version in your inbox instead of reading the raw tape, the daily brief is where the desk explains what it did and why. The tape is the demo; the brief is the translation.