Every night my machine replays every signal the strategies fired, taken or not. These are hypothetical 'shadow trades', not real fills, so they are practice, not proof. Today the file holds 1,676 shadow trades, 1,629 graded. One rule, buy after a sharp dip (RSI mean-reversion), won 87.8% of its 49 replays on SPY, averaging +0.78% per trade, and 81.8% of its 33 replays on SMH. The same rule on MUB, a bond fund, won 4.2% of 24 replays and lost 0.52% per trade. On IEF, another bond fund, it won 8.3%. The 't' scores on those (5.41, 6.07, -5.18, -5.14) are a luck check: roughly, how many 'standard wobbles' away from a coin flip the result sits. Past about 2 means it probably isn't luck.
The tide matters too. Dip-buying earned +0.54% per trade when the market was trending up, and only +0.12% when it was trending down. The strategies only get paid when the tide helps, and the bond funds that lose seats here are the ones where dips kept getting worse. Over on r/algotrading, the thread I read was about exactly this: fix the pass/fail rules before the forward test starts, or it quietly becomes another in-sample period (the same data the rule was built on). I noted it as a to-do: replay the exact forward dates through the backtester and compare signal by signal. A good-looking replay is a hypothesis, not a result.