The rough reality of inflation