Optimization answers a conditional question: which parameter region best expresses a rule on the data and costs supplied? It does not answer the prior question of why that rule should contain repeatable information.
When the underlying idea is weak, a large search can still find an attractive peak. The peak may describe sampling noise, one market regime, an execution assumption or the researcher’s repeated choices rather than a durable mechanism.
Idea before parameter search
Every stage can stop the project before the search becomes a story generator.
- 01MechanismState what market behaviour the rule measures and when it should fail.
- 02BaselineTest a simple fixed version after realistic costs before widening the search.
- 03StabilityPrefer broad acceptable regions and consistent neighbouring settings over one peak.
- 04ChallengeUse untouched periods, ablation, cost stress and forward observation before promotion.
A search algorithm always returns a winner
If hundreds of parameter combinations are compared, one will usually look best even when the rule has little real information. Repeating the search after changing filters, dates or objectives silently increases the number of trials. The reported backtest then includes the researcher’s selection process, not only the strategy.
Robust ideas form regions, not needles
A plausible mechanism should tolerate small changes in lookback, threshold and cost. A lone profitable cell surrounded by failure suggests dependence on exact historical coincidences. A broad plateau is not proof of future profit, but it is more credible than an isolated optimum.
Ablation tests the story
Remove each filter, replace the optimized value with a simple baseline, delay execution and worsen spread or slippage. If the proposed mechanism is real, the direction of the effect should remain intelligible. If every component is necessary only in its exact fitted form, the model is probably explaining the sample rather than the market.
A disciplined stopping rule
Reject the idea when no sensible baseline survives costs, when neighbouring parameters reverse the result, or when untouched data destroys the claimed behaviour. More computing power is not a reason to continue.
- Write the hypothesis and failure regime before optimization.
- Limit the search space to economically or mechanically defensible values.
- Record every material search and comparison, not only the final run.
- Reserve an untouched final period and keep forward evidence separate.
- Prefer stable behaviour and explainable degradation over the highest score.
Questions you may have
Is optimization itself bad?
No. It is useful for calibration, sensitivity analysis and operational trade-offs when the hypothesis and validation process are defined first.
Does a flat parameter surface prove an edge?
No. It reduces one fragility concern, but data quality, costs, selection bias, regimes and forward behaviour still require testing.
When should optimization stop?
When a simple baseline fails, neighbouring settings are unstable, or untouched evidence contradicts the proposed mechanism.
