Why it matters
Search is still the largest source of intent-driven traffic, and Robots.txt is one of the levers that decides whether the right people ever find a page. Neglect it and even excellent work stays invisible; handle it well and the same content keeps earning visits long after launch.
In practice
In practice we fold Robots.txt into the build rather than bolting it on at the end. That means clear structure, honest content and the technical groundwork in place from the first sprint, then steady measurement so we can see what is actually moving in search.