Reimplement Parser

[idavis]

An intro to the PR with some useful links for the relevant sections:

This PR has two major components. A custom QASM3 parser, which replaces the IBM parser. And a refactored version of the compiler.

The custom QASM3 parser has three components:

1. The raw lexer.
2. The cooked lexer.
3. Parser.

The refactored compiler has two components:
4. Lowerer.
5. Compiler.

Below is more detailed overview of the purpose of each component:

The OpenQASM 3.0 grammar will be useful while reviewing sections 1-2
https://openqasm.com/grammar/index.html

1. The raw lexer:
   Takes the source code as input and returns a stream of "raw tokens". You can think of these mostly as characters like '(' and type literals.

2. The cooked lexer:
   Takes the raw token stream and returns a stream of cooked tokens. Cooked tokens have more knowledge about QASM3 syntax, and they make parsing easier.

3. Parser:
   Takes the stream of cooked tokens as input and returns an AST. We call it the syntax AST, to differentiate it from the semantic AST, which will be introduced later.

The grammar and specification of QASM3 don't fully agree. We had to take some decissions while implementing the parser. They are documented in the rust code. If you want to double check anything, the language spec will come in handy:
https://openqasm.com/language/index.html

4. Lowerer:
   Takes the AST returned by the parser and performs semantic analysis on it. This is where we report any QASM3 related errors, like "functions can only be defined on the global scope." The lowerer returns a semantic AST. You will want to use the language spec to verify that the errors that we are checking for are actually on the spec, or that we are not missing any checks.
   https://openqasm.com/language/index.html

5. Compiler:
   Takes the semantic AST and compiles it to a Q# AST. This is where we report any Q# related errors, like unsupported QASM3 features because they don't make sense in Q#. The compiler is a very straighforward mapping from the semantic AST to the Q# AST, since most of the heavy lifting is done during lowering.

Recomended Sections for reviewers:

qsc_qasm3/src/lib.rs (Entry point and general structure)
Mine, Stefan

qsc_qasm3/src/parser/completion.rs (Completion)
qsc_qasm3/src/io.rs (IO)
Mine

qsc_qasm3/src/lex.rs (Raw and Cooked lexer)
Scott

qsc_qasm3/src/parser.rs (Parser)
qsc_qasm3/src/parser/ast.rs (QASM3 AST, double check against grammar & Spec)
Scott

qsc_qasm3/src/oqasm_helpers.rs
qsc_qasm3/src/semantic/types.rs (QASM3 types)
qsc_qasm3/src/semantic.rs (Lowerer entry point)
qsc_qasm3/src/lowerer.rs
This requires QASM3 specific knowledge. We can use as many eyes as possible.
Dmitry, Scott

qsc_qasm3/src/stdlib.rs
qsc_qasm3/src/stdlib/angle.rs
qsc_qasm3/src/stdlib/QasmStdrs
Dmitry

qsc_qasm3/src/types.rs (Types)
qsc_qasm3/src/ast_builder.rs
qsc_qasm3/src/compiler.rs (Compiler)
qsc_qasm3/src/runtime.rs (Runtime Features)
Stefan

fuzz/ (Testing)
Mine, Stefan

pip/src/ (Interop)
qsc/src/
qsc_codegen/src/
Mine, Stefan

.github/fuzz (GitHub pipeline)
Ian, Stefan

Tracking items

Lexing

• Basic raw tokens
• Basic cooked tokens
• Cook pragma and annotation

Parsing

• Version Add initial AST and parser infrastructure #2148
• Include Add initial AST and parser infrastructure #2148
• pragma Add initial AST and parser infrastructure #2148
• annotation Add initial AST and parser infrastructure #2148
• Statements -
  • alias Add ability to parse aliases, control flow statements, and expression statements to QASM3 parser #2191
  • array concatenation Add ability to parse aliases, control flow statements, and expression statements to QASM3 parser #2191
  • assignment Add ability to parse aliases, control flow statements, and expression statements to QASM3 parser #2191
  • barrier Add missing statements to QASM3 parser #2200
  • box Add missing statements to QASM3 parser #2200
  • break Add ability to parse aliases, control flow statements, and expression statements to QASM3 parser #2191
  • cal Add missing statements to QASM3 parser #2200
  • calibration grammar Add missing statements to QASM3 parser #2200
  • classical decl
    • Bit, int, uint, float, angle, bool, duration, stretch Adding classical & io decls to the parser, stubbing out array decls #2165
    • Arrays Add array and measurement declaration statements #2180
    • Measurement result Add array and measurement declaration statements #2180
  • const decls Adding classical & io decls to the parser, stubbing out array decls #2165
  • continue Add ability to parse aliases, control flow statements, and expression statements to QASM3 parser #2191
  • def Add subroutine, return, and gate defs, externs, and old style decls #2176
  • defcal Add missing statements to QASM3 parser #2200
  • delay Add missing statements to QASM3 parser #2200
  • end Add ability to parse aliases, control flow statements, and expression statements to QASM3 parser #2191
  • expression statement Add ability to parse aliases, control flow statements, and expression statements to QASM3 parser #2191
  • extern Add subroutine, return, and gate defs, externs, and old style decls #2176
  • for loops Add ability to parse aliases, control flow statements, and expression statements to QASM3 parser #2191
  • gate call Add missing statements to QASM3 parser #2200
  • gate statement Add subroutine, return, and gate defs, externs, and old style decls #2176
  • gphase Add missing statements to QASM3 parser #2200
  • if statement Add ability to parse aliases, control flow statements, and expression statements to QASM3 parser #2191
  • IO decls Adding classical & io decls to the parser, stubbing out array decls #2165
  • measure arrow Add missing statements to QASM3 parser #2200
  • old style decl Add subroutine, return, and gate defs, externs, and old style decls #2176
  • quantum decl Add ability to parse quantum decls and lit exprs #2160
  • reset Add missing statements to QASM3 parser #2200
  • return Add subroutine, return, and gate defs, externs, and old style decls #2176
  • switch statements QASM3 - Parse switch statements #2178
  • while loops Add ability to parse aliases, control flow statements, and expression statements to QASM3 parser #2191
  • gate def Add subroutine, return, and gate defs, externs, and old style decls #2176
• Precedence Implement Pratt Parsing for Qasm3 parser #2166
• Expressions -
  • ident Add initial AST and parser infrastructure #2148
  • unary Implement Pratt Parsing for Qasm3 parser #2166
  • binary Implement Pratt Parsing for Qasm3 parser #2166
  • literal Add ability to parse quantum decls and lit exprs #2160
  • timing literal Add missing statements to QASM3 parser #2200
  • function call Implement Pratt Parsing for Qasm3 parser #2166
  • cast Implement Pratt Parsing for Qasm3 parser #2166
  • assignment Make assignment a StmtKind #2216
  • assignment op Make assignment a StmtKind #2216
• Check for cyclic includes Cyclic and multiple include errors #2278
• Document parsing functions with grammar defs and spec notes Add grammar docstrings to QASM3 parser #2222
• Remove measurement exprs from const decl parsing. Update notes on the grammar docs as this is a spec bug. Unignore unit tests; lower and compile missing items #2272

Lowering

• Constant Evaluation for array sizes and type widths Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
• Statements -
  • alias Implement base declarations and binary expression lowering #2221
  • assignment
    • simple Implement base declarations and binary expression lowering #2221
    • indexed Implement base declarations and binary expression lowering #2221
  • barrier Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
  • box (unimplemented)
  • break Unignore unit tests; lower and compile missing items #2272
  • cal (unimplemented)
  • calibration grammar (unimplemented)
  • classical decls with default and literal initializers
    • Bit, int, uint, float, angle, bool, duration, stretch Implement base declarations and binary expression lowering #2221
    • creg Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
    • qreg Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
    • Bitarrays Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
    • Arrays (unimplemented)
    • Measurement result Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
  • classical decls with all expressions and casting Add initial semantic analysis/transform #2214
    • Bit, int, uint, float, angle, bool, duration, stretch Implement base declarations and binary expression lowering #2221
    • creg Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
      • tests see bit[] tests for cases Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
    • qreg Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
      • tests - see qubit[] tests for cases Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
    • Bitarrays Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
    • Arrays (unimplemented)
    • Measurement result Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
  • continue Unignore unit tests; lower and compile missing items #2272
  • def Compile gphase, gatedef, and control flow statements #2254
  • defcal (unimplemented)
  • delay (unimplemented)
  • end Compile annotations into attributes #2271
  • expression statement Implement base declarations and binary expression lowering #2221
  • extern Lower extern declaration #2274
  • for loops Lower control flow stmts in QASM3 parser #2232
  • gate call Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
    • broadcast (unimplemented)
  • gphase Compile gphase, gatedef, and control flow statements #2254
  • if statement Lower control flow stmts in QASM3 parser #2232
  • IO decls
  • measure arrow Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
  • old style decl Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
  • quantum decl Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
  • reset Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
  • return Compile gphase, gatedef, and control flow statements #2254
    • add cast of return value to containing function return ty Unignore unit tests; lower and compile missing items #2272
  • switch statements Lower control flow stmts in QASM3 parser #2232
  • while loops Lower control flow stmts in QASM3 parser #2232
  • gate def Compile gphase, gatedef, and control flow statements #2254
  • Return semantic error on anonymous blocks and switch statements if version is set to 3.0 as they were introduced in 3.1 Lower control flow stmts in QASM3 parser #2232
• Expressions
  • binary Adding new compiler, lowerer is now infallible #2239
  • unary Adding new compiler, lowerer is now infallible #2239
  • function call Compile gphase, gatedef, and control flow statements #2254
  • literals
    • bit, bool, float, int, uint Implement base declarations and binary expression lowering #2221
    • angle Add angle support to the new compiler #2267
    • complex Lower and compile complex binary ops #2268
    • duration Unignore unit tests; lower and compile missing items #2272
    • cast (unimplemented)
• Review ignored tests from old parser Unignore unit tests; lower and compile missing items #2272
• SimulatableIntrinsic support Compile annotations into attributes #2271
  • Error when not applied to a gate or def Compile annotations into attributes #2271

Compiling

• Only set EntryPoint() attr on operation when compiling in file mode
• Statements -
  • alias (unsupported) Adding new compiler, lowerer is now infallible #2239
    • array concatenation
  • assignment Adding new compiler, lowerer is now infallible #2239
  • barrier Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
  • box (unimplemented)
    • pragma impls (unimplemented)
  • break (unsupported) Unignore unit tests; lower and compile missing items #2272
  • cal (unimplemented)
  • calibration grammar (unimplemented)
  • classical decls Adding new compiler, lowerer is now infallible #2239
    • Bit, int, float, angle, bool, duration, stretch Adding new compiler, lowerer is now infallible #2239
    • creg Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
    • qreg Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
    • Arrays (unimplemented)
    • Measurement result Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
  • continue (unsupported) Unignore unit tests; lower and compile missing items #2272
  • def Compile gphase, gatedef, and control flow statements #2254
    • simulatable intrinsic Compile annotations into attributes #2271
  • end Compile annotations into attributes #2271
  • expression statement Adding new compiler, lowerer is now infallible #2239
  • extern (unimplemented)
  • for loops Compile gphase, gatedef, and control flow statements #2254
  • gate call Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
  • gphase Compile gphase, gatedef, and control flow statements #2254
  • if statement Compile gphase, gatedef, and control flow statements #2254
  • IO decls Adding new compiler, lowerer is now infallible #2239
  • measure arrow Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
  • old style decl Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
  • quantum decl Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
  • reset Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
  • return Compile gphase, gatedef, and control flow statements #2254
  • switch statements Compile gphase, gatedef, and control flow statements #2254
  • while loops Adding new compiler, lowerer is now infallible #2239
  • gate def Compile gphase, gatedef, and control flow statements #2254
    • simulatable intrinsic Unignore unit tests; lower and compile missing items #2272
    • gate scopes capture const variables in the outside scopes. We compiled gates to lambda operations to get this behavior. But this breaks simulatable intrinsics. Now that we have const evaluation, we can overcome this. functions and operations with const eval #2270
  • delay (unimplemented)
  • barrier Add gate calls and the AST items it depends on to the new QASM3 compiler #2246
• Expressions
  • binary Adding new compiler, lowerer is now infallible #2239
  • unary Adding new compiler, lowerer is now infallible #2239
  • function call Compile gphase, gatedef, and control flow statements #2254
  • literals
    • bit, bool, float, int, uint Implement base declarations and binary expression lowering #2221
    • angle Add angle support to the new compiler #2267
    • complex Lower and compile complex binary ops #2268
    • duration (unsupported) Adding new compiler, lowerer is now infallible #2239
    • cast (unimplemented)
• Move runtime functions in new dependency package so that they are parsed as a unit and imported by user code. This enables spans and a file name for runtime calls that get generated.

Fit and Finish

• Move Q# related semantic errors from Lowerer to Compiler
• Add better error messages to const evaluator. We can pass &mut Lowerer as an argument so that it can push semantic errors. Better error messages for const evaluator #2277
• Clean up TODO items Cleanup TODO items #2285
• Review compiler error messages Review errors in lexer, parser, lowerer, and compiler #2286
• Review parser error messages Review errors in lexer, parser, lowerer, and compiler #2286
• Review lexer error messages Review errors in lexer, parser, lowerer, and compiler #2286
• Delete old compiler internals that are no longer used and remove stale refs Remove external parser. Fix Qiskit's failure to emit gates. #2284
• Benchmark parser Add benchmark to new parser #2288

Postponed

• Update ast_builder calls with module names instead of old ns names
• bit x = 0; bit y = ~x; is valid QASM3, but it gets compiled to let x = Zero; let y = ~~~x; which is invalid Q# code, since the Result type in Q# doesn't support unary bitwise negation. This is the same for all other operations that bit should support, Q# Result only supports equality.
• We are currently not enforcing that qasm3 uint types must be positive, since they are compiled to Q# Int which are signed. We might need a UInt type similar to the Angle type introduced in Add angle support to the new compiler #2267 to be able to enforce this constraint.
• uint 63 bitness with 64 bigint. Separate ops for uint?
• Lower Ident as Rc<Symbol> and not as SymbolId to avoid an unnecessary SymbolTable lookup? In that way we don't even need to pass the symbol table to the compiler.
• Use the formal parameters' SymbolIds when creating the function type, so that we can give the user a better error message when they pass an argument that fails implicit casting to a function. There is a catch: we need to insert the function symbol into the symbol table before pushing the scope where the formal parameters live, but we need the SymbolIds of the formal parameters to construct the function symbol.
• Create qubit cleanup calls which can be conditionally added to end of program during compile, not done for fragments Qubit release calls #2281
• Every time we allow an arithmetic lint in compiler/qsc_qasm3/src/semantic/ast/const_eval.rs we need to issue the same lint in Q#.
• Double check cast from int to uint and cast from uint to int in const evaluator. Both types are represented as i64 so there is nothing to do, which is confusing.
• Profile the usage of Box<[Box<T>]> with iai-callgrind in a large OpenQASM3 sample to verify that is actually faster than using Vec. Even though Box uses less stack space, it reduces cache locality, because now you need to be jumping around in memory to read contiguous elements of a list. I suspect that what we really want to use if Box<[T]>.
• Input decls are pushed to the symbol table, but should not be in the stmts list. This may be an issue for tooling as there isn't a way to have a forward declared variable in Q#. compiler/qsc_qasm3/src/compiler.rs::QasmCompiler::compile_output_decl_stmt.
• See the comment on compiler/qsc_qasm3/src/compiler.rs::QasmCompiler::create_entry_item saying "This can create a collision on multiple compiles when interactive. We also have issues with the new entry point inference logic."
• Minimize number of errors reported in the compiler. There are many situations where we report duplicated or unnecessary extra errors that could make harder for the user to fix the actual problem. For example, see the test named fuzzer_issue_2294. The one valuable error there is the one saying "undefined symbol _`.