Cross entropy benchmarking functionality does not match documentation

[kevinsung]

Documentation:

Cirq/cirq/experiments/cross_entropy_benchmarking.py

Line 107 in 1d35492

$f_{mn}^{meas} = \langle D * P^{th, mn}_q - 1 \rangle$

Code:

Cirq/cirq/experiments/cross_entropy_benchmarking.py

Lines 335 to 337 in 1d35492

	pp_cross = probs_exp * probs_meas
	pp_exp = probs_exp**2
	f_meas = np.mean(num_states * np.sum(pp_cross, axis=1) - 1.0)

In the code, f_meas seems to be a quite different thing from what's documented: it is obtained by estimating the experimental bitstring probabilities prob_meas through a experimental histogram (note that most of these will be 0 at a modest number of qubits), and then multiplying that number by the theoretical probability.

It appears that the documentation is based on the assumption of exponentially distributed (Porter-Thomas) probabilities, while the code is doing something else. Is that right?

@XiaoMiQC

[dabacon]

@viathor also to comment

[mpharrigan]

@viathor @XiaoMiQC is this an issue?

[XiaoMiQC]

I believe the formula in the documentation is actually equivalent to what is being done in the code. This has been used on actual quantum processors and I have obtained fidelity numbers as expected. One can test the equivalence by generating bits using the Cirq simulator and compute the results using the code. If it is different than what you get out of implementing the formula in the documentation, please let me know.

[dstrain115]

Circ cync: Looks like this is correct actually, @viathor will doublecheck and update status.

[viathor]

@XiaoMiQC @kevinsung @dstrain115

Kevin is right. Docstring and code disagree. The former describes the linear cross-entropy benchmarking (which was our ultimate choice in QS experiment back in 2019). The fidelity estimate in this method is produced by averaging the expression 2^n p_th - 1 over the experimentally observed bitstrings (where p_th stands for the ideal/theoretical probability of each bitstring).

OTOH, the code appears to use Vadim et al's Least Mean Square for XEB wherein the fidelity estimate is computed as the ratio (dH_measured * dH_ideal) / dH_ideal**2. Both are known to work, but produce slightly different estimates with different statistical properties (IIRC, the former has better variance).

I don't think it makes sense to change the code, but we should update the docstring.

---

Also, it is worth noting that a number of XEB-based estimators of fidelity are implemented in fidelity_estimation.py. In particular, see linear_xeb_fidelity_from_probabilities and least_squares_xeb_fidelity_from_probabilities which implement the two estimators referred to above. In addition, the module implements log XEB and HOG score.

Thus, some of the XEB functionality is unfortunately duplicated. What do y'all think about removing cross_entropy_benchmarking.py? Does anyone still use this code? Could they switch to the newer and more versatile alternative in fidelity_estimation.py and related(*) modules?

(*) Note that fidelity_estimation.py has a narrower focus than cross_entropy_benchmarking.py. For example, it does not have code to generate RQCs. That said, AFAIK code to do all other XEB-related tasks exists elsewhere in cirq.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days

[github-actions]

Issue closed due to inactivity.

[mpharrigan]

for your second comment @viathor there's #3775

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days