Skip to content

Improve performance benchmarks coverage after migration to ASV #3840

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
tanujkhattar opened this issue Feb 19, 2021 · 1 comment
Open

Improve performance benchmarks coverage after migration to ASV #3840

tanujkhattar opened this issue Feb 19, 2021 · 1 comment
Labels
area/performance kind/task A task that's part of a larger effort no QC knowledge needed Want to contribute to Cirq, but don't know quantum computing? This issue is for you. status/needs-agreed-design We want to do this, but it needs an agreed upon design before implementation triage/accepted A consensus emerged that this bug report, feature request, or other action should be worked on
Milestone
2025-T2: Performa...

Comments

@tanujkhattar
Copy link
Collaborator

Currently, the performance benchmarks in Cirq are very limited. We should improve the coverage of performance benchmarks by adding more tests.

We should discuss

  • Which functions / code pieces are worth benchmarking?
  • Should we have benchmarks for all functions / modules ?

Decide a strategy and then implement the additional benchmarks using ASV.

Related: #3822 #1124
@tanujkhattar tanujkhattar added kind/task A task that's part of a larger effort triage/discuss Needs decision / discussion, bring these up during Cirq Cynque labels Feb 19, 2021
@tanujkhattar tanujkhattar mentioned this issue Feb 19, 2021
Merged
@95-martin-orion
Copy link
Collaborator

Discussed in Cirq sync:

  • We need to incorporate benchmarks into the PR review workflow (specifically highlighting regressions, not blocking PRs)
  • General concern around hard-to-detect gradual slowdown of code

@tanujkhattar Do we have this kind of support, or designs around it?

@mpharrigan mpharrigan added status/needs-agreed-design We want to do this, but it needs an agreed upon design before implementation triage/accepted A consensus emerged that this bug report, feature request, or other action should be worked on and removed triage/discuss Needs decision / discussion, bring these up during Cirq Cynque labels May 19, 2021
@tanujkhattar tanujkhattar added the no QC knowledge needed Want to contribute to Cirq, but don't know quantum computing? This issue is for you. label Jan 5, 2022
@tanujkhattar tanujkhattar mentioned this issue Aug 29, 2022
Merged
CirqBot pushed a commit that referenced this issue Sep 7, 2022
@tanujkhattar
Fix asv setup and add benchmarks for circuit construction (#5845)
6554899 
Part of #5833 and #3840
Fixes #4695

Running the benchmarks on master currently gives the following output for circuit construction benchmarks:

```bash
[ 75.00%] ··· circuit_construction.NDCircuit.time_circuit_construction                                                                                                                                                                                                                  ok
[ 75.00%] ··· ===================== ============= ============= ============= ============
              --                                           Depth(D)                       
              --------------------- ------------------------------------------------------
               Number of Qubits(N)        1             10           100          1000    
              ===================== ============= ============= ============= ============
                        1             38.3±0.2μs     235±3μs     2.12±0.01ms   21.2±0.1ms 
                        10            109±0.6μs      901±4μs     8.73±0.02ms   89.2±0.2ms 
                       100             770±4μs     7.32±0.02ms    74.4±0.4ms    764±9ms   
                       1000          7.30±0.02ms    72.0±0.2ms     739±4ms     7.57±0.03s 
              ===================== ============= ============= ============= ============

[ 87.50%] ··· circuit_construction.SurfaceCode_Rotated_Memory_Z.time_surface_code_circuit_construction                                                                                                                                                                                  ok
[ 87.50%] ··· ========== =============
               distance               
              ---------- -------------
                  3       1.97±0.01ms 
                  5       7.67±0.07ms 
                  7        22.6±0.1ms 
                  9        59.8±0.4ms 
                  11       122±0.7ms  
                  13        232±1ms   
                  15        407±2ms   
                  17        701±2ms   
                  19       1.09±0.02s 
                  21       1.60±0.02s 
                  23       2.25±0.01s 
                  25       3.24±0.06s 
              ========== =============

[100.00%] ··· circuit_construction.SurfaceCode_Rotated_Memory_Z.track_surface_code_circuit_operation_count                                                                                                                                                                              ok
[100.00%] ··· ========== =========
               distance           
              ---------- ---------
                  3         369   
                  5         3225  
                  7        12985  
                  9        36369  
                  11       82401  
                  13       162409 
                  15       290025 
                  17       481185 
                  19       754129 
                  21      1129401 
                  23      1629849 
                  25      2280625 
              ========== =========

``` 

Note that the circuit construction for d21 surface code takes only 1.6 seconds. This is because we are constructing moment-by-moment instead of appending operations (in which case the earliest strategy takes a significant amount of time). 

Going forward, we should add more benchmarks for different use cases. Once this is merged, https://cirq-infra.uc.r.appspot.com/ would be updated with the new benchmarks and we can see the improvement / regressions we make to circuit construction performance over time / commits. 


cc @dabacon @maffoo @dstrain115
This was referenced Sep 7, 2022
Merged
Merged
Merged
CirqBot pushed a commit that referenced this issue Sep 21, 2022
@tanujkhattar
Add performance benchmarks for single qubit randomized benchmarking (#…
279702f 
…5866)

Adds circuit construction benchmarks for single qubit randomized benchmarking for 100 qubits and upto depth 1000. 

Part of #3840
cc qh-lab/pyle#29309

The output of the benchmarks is:
```
[ 25.00%] ··· randomized_benchmarking.SingleQubitRandomizedBenchmarking.time_rb_circuit_construction                                                                                                                                                                                    ok
[ 25.00%] ··· ======= ==========
              --      num_qubits / num_circuits
              ------- ----------
               depth   100 / 20 
              ======= ==========
                 1     119±0ms  
                 10    236±0ms  
                 50    776±0ms  
                100    1.41±0s  
                250    3.43±0s  
                500    6.64±0s  
                1000   12.9±0s  
              ======= ==========

[ 50.00%] ··· randomized_benchmarking.SingleQubitRandomizedBenchmarking.time_rb_op_grid_generation                                                                                                                                                                                      ok
[ 50.00%] ··· ======= ==========
              --      num_qubits / num_circuits
              ------- ----------
               depth   100 / 20 
              ======= ==========
                 1     96.8±0ms 
                 10    162±0ms  
                 50    427±0ms  
                100    762±0ms  
                250    1.74±0s  
                500    3.29±0s  
                1000   6.28±0s  
              ======= ==========

```
rht pushed a commit to rht/Cirq that referenced this issue May 1, 2023
@tanujkhattar @rht
Fix asv setup and add benchmarks for circuit construction (quantumlib…
9fc388c 
…#5845)

Part of quantumlib#5833 and quantumlib#3840
Fixes quantumlib#4695

Running the benchmarks on master currently gives the following output for circuit construction benchmarks:

```bash
[ 75.00%] ··· circuit_construction.NDCircuit.time_circuit_construction                                                                                                                                                                                                                  ok
[ 75.00%] ··· ===================== ============= ============= ============= ============
              --                                           Depth(D)                       
              --------------------- ------------------------------------------------------
               Number of Qubits(N)        1             10           100          1000    
              ===================== ============= ============= ============= ============
                        1             38.3±0.2μs     235±3μs     2.12±0.01ms   21.2±0.1ms 
                        10            109±0.6μs      901±4μs     8.73±0.02ms   89.2±0.2ms 
                       100             770±4μs     7.32±0.02ms    74.4±0.4ms    764±9ms   
                       1000          7.30±0.02ms    72.0±0.2ms     739±4ms     7.57±0.03s 
              ===================== ============= ============= ============= ============

[ 87.50%] ··· circuit_construction.SurfaceCode_Rotated_Memory_Z.time_surface_code_circuit_construction                                                                                                                                                                                  ok
[ 87.50%] ··· ========== =============
               distance               
              ---------- -------------
                  3       1.97±0.01ms 
                  5       7.67±0.07ms 
                  7        22.6±0.1ms 
                  9        59.8±0.4ms 
                  11       122±0.7ms  
                  13        232±1ms   
                  15        407±2ms   
                  17        701±2ms   
                  19       1.09±0.02s 
                  21       1.60±0.02s 
                  23       2.25±0.01s 
                  25       3.24±0.06s 
              ========== =============

[100.00%] ··· circuit_construction.SurfaceCode_Rotated_Memory_Z.track_surface_code_circuit_operation_count                                                                                                                                                                              ok
[100.00%] ··· ========== =========
               distance           
              ---------- ---------
                  3         369   
                  5         3225  
                  7        12985  
                  9        36369  
                  11       82401  
                  13       162409 
                  15       290025 
                  17       481185 
                  19       754129 
                  21      1129401 
                  23      1629849 
                  25      2280625 
              ========== =========

``` 

Note that the circuit construction for d21 surface code takes only 1.6 seconds. This is because we are constructing moment-by-moment instead of appending operations (in which case the earliest strategy takes a significant amount of time). 

Going forward, we should add more benchmarks for different use cases. Once this is merged, https://cirq-infra.uc.r.appspot.com/ would be updated with the new benchmarks and we can see the improvement / regressions we make to circuit construction performance over time / commits. 


cc @dabacon @maffoo @dstrain115
rht pushed a commit to rht/Cirq that referenced this issue May 1, 2023
@tanujkhattar @rht
Add performance benchmarks for single qubit randomized benchmarking (q…
9f545b0 
…uantumlib#5866)

Adds circuit construction benchmarks for single qubit randomized benchmarking for 100 qubits and upto depth 1000. 

Part of quantumlib#3840
cc https://github.com/qh-lab/pyle/issues/29309

The output of the benchmarks is:
```
[ 25.00%] ··· randomized_benchmarking.SingleQubitRandomizedBenchmarking.time_rb_circuit_construction                                                                                                                                                                                    ok
[ 25.00%] ··· ======= ==========
              --      num_qubits / num_circuits
              ------- ----------
               depth   100 / 20 
              ======= ==========
                 1     119±0ms  
                 10    236±0ms  
                 50    776±0ms  
                100    1.41±0s  
                250    3.43±0s  
                500    6.64±0s  
                1000   12.9±0s  
              ======= ==========

[ 50.00%] ··· randomized_benchmarking.SingleQubitRandomizedBenchmarking.time_rb_op_grid_generation                                                                                                                                                                                      ok
[ 50.00%] ··· ======= ==========
              --      num_qubits / num_circuits
              ------- ----------
               depth   100 / 20 
              ======= ==========
                 1     96.8±0ms 
                 10    162±0ms  
                 50    427±0ms  
                100    762±0ms  
                250    1.74±0s  
                500    3.29±0s  
                1000   6.28±0s  
              ======= ==========

```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/performance kind/task A task that's part of a larger effort no QC knowledge needed Want to contribute to Cirq, but don't know quantum computing? This issue is for you. status/needs-agreed-design We want to do this, but it needs an agreed upon design before implementation triage/accepted A consensus emerged that this bug report, feature request, or other action should be worked on
Projects
Cirq 2025 roadmap
Status: No status
Milestone
2025-T2: Performance monitoring
Development

No branches or pull requests
6 participants
@mhucka @MichaelBroughton @mpharrigan @tanujkhattar @viathor @95-martin-orion