A benchmark is a condition against which we can judge that a quantum device is “good enough” in some sense – usually many different benchmarking metrics are required. Benchmarking is important for assessing the quality of NISQ (noisy intermediate scale quantum) devices as they scale up to full-fledged fault tolerant computers. As of today, there is a zoo of quantum benchmarking protocols, each with their own sets of assumptions, type of information gained and scalability, as illustrated in the figure below.
- Some benchmarks are powerful tools, as they come with complexity theoretic guarantees of quantum-over-classical advantage. Others come with guarantees of security.
- There are, in general, various tradeoffs that appear in benchmarking protocols. For example, certain benchmarks give you more information by running your quantum circuit multiple times, which increases implementation cost.
- Some benchmarks are designed to assess generic circuit or gate performance, others to assess the correct application of a certain gate or subroutine in a quantum algorithm.
- There are also other more physics-related benchmarks which are designed to assess the performance of certain components of a certain quantum hardware, e.g. Quantum dot efficiency, coherence time of a superconducting qubit, etc.
Frequently asked questions
- What properties are benchmarked? Gate fidelity or average gate/circuit-fidelity, which quantify how close your gate/circuit is to the ideal. Qubit connectivity, average error per gate, energy relaxation and coherence times (T1, T2), characterizing environmental noise effects – which significantly affects gate performance. Whether the Hamiltonian is correct – learning theory. Some broader metrics are quantum volume and cross-entropy benchmarking. At the experiment level, general diagnostic tests.
- What are examples of benchmarking protocols? Tools that come under the umbrella-term randomized benchmarking, such as cycle and cross-entropy benchmarking are used frequently to benchmark quantum devices. Other common methods are direct fidelity estimation and quantum state/process tomography. There are also more holistic benchmarks like quantum volume which are designed to assess generic circuit performance, as well as some application specific ones like verifying a correct implementation of a quantum Fourier transform, or solving MAXCUT with some accuracy on a quantum device.
- What is the best benchmark to use? This is a very interesting and multifaceted open question. It appears to be very hard to find the ”one” benchmark which is suited and fair for all quantum hardware, and which assesses various performance aspects. Usually, researchers will apply a suite of benchmarking protocols in order to report the most accurate overall evaluation of their device.
