Quant finance is often portrayed as precise and data-driven — but ironically, some of the biggest mistakes happen because of a false sense of statistical certainty. Nowhere is this more apparent than in backtesting. As we developed Astrai’s internal research infrastructure, we intentionally re-implemented backtests from scratch to avoid black-box dependencies and to expose ourselves to the gritty details. In doing so, we ran headfirst into many well-known but often-overlooked traps: lookahead bias (using information not yet available), survivorship bias (excluding delisted assets), and excessive tuning (optimizing for past noise instead of future signal). But we also encountered subtler issues — like how rebalancing frequency interacts with signal decay, or how clustered volatility can mask poor capital allocation. This blog post explores those findings in depth, with hands-on code snippets and our evolving philosophy: a backtest is a storytelling device, not a truth machine. Good backtests raise questions; great ones survive them. We’re committed to sharing not just what worked, but what failed, and why that matters more.