Avoid state vector normalization if it worsens the round offs (#6556)

This fixes failure of

check/pytest -n0 cirq-core/cirq/circuits/circuit_test.py::test_final_state_vector

which happened because normalization of a state vector at `np.complex64`
precision can subtly increase the overall round-off error.

Follow up to #6522 and #6402

Author: pavoljuhas

cirq-core/cirq/sim/state_vector_simulator.py

@@ -134,7 +134,11 @@ def final_state_vector(self) -> np.ndarray:
                 "skipping renormalization"
             )
             return ret
-        return ret / norm
+        # normalize only if doing so improves the round-off on total probability
+        ret_norm = ret / norm
+        round_off_change = abs(np.vdot(ret_norm, ret_norm) - 1) - abs(np.vdot(ret, ret) - 1)
+        result = ret_norm if round_off_change < 0 else ret
+        return result
 
     def state_vector(self, copy: bool = False) -> np.ndarray:
         """Return the state vector at the end of the computation.