[SYCL][E2E][Docs] Update test-mode documentation (#16875)

Changes documentation of `build-only` mode to reflect the recent
improvements in this mode.

Author: ayylol

sycl/test-e2e/README.md

@@ -12,6 +12,10 @@
   * [Marking tests as expected to fail](#marking-tests-as-expected-to-fail)
   * [Marking tests as unsupported](#marking-tests-as-unsupported)
 * [SYCL core header file](#sycl-core-header-file)
+* [Compiling and executing tests on separate systems](#compiling-and-executing-tests-on-separate-systems)
+  * [Run only mode](#run-only-mode)
+  * [Build only mode](#build-only-mode)
+  * [Common Issues with separate build and run](#common-issues-with-separate-build-and-run)
 
 ## Overview
 
@@ -393,109 +397,117 @@ machines. By default the `test-mode` parameter is set to `full`, indicating that
 both stages will run. This parameter can be set to `build-only`, or `run-only`,
 to only run the compilation stage, or the execution stage respectively.
 
-**NOTE:** This feature is a work-in-progress and current limitations are
-expected to be addressed in the near future.
-
-The modes work as follow:
-
-* `--param test-mode=full`
-
-  This is the default mode tests run in. Tests are marked as unsupported if no
-  device on the machine can fulfill the `REQUIRES`/`UNSUPPORTED` statements. In
-  this mode all `RUN:` lines are executed normally, and two extra features are
-  added: the `build-and-run-mode` and `run-mode`.
-
-  To make a test only run in `full` mode add a `REQUIRES: build-and-run-mode`
-  line.
-
-* `--param test-mode=build-only`
-
-  This mode can be used to compile all test binaries. To do this all
-  `UNSUPPORTED` and `REQUIRES` statements are ignored unless they contain
-  `UNSUPPORTED: true` or `REQUIRES: build-and-run-mode`. All `RUN:` lines within
-  a test are ran in this mode unless they contain the following expansions:
-  `%{run}`, `%{run-unfiltered-devices}`, or `%if run-mode`.
-
-  Currently, the only triple supported for `build-only` mode is `spir64`.
-
-  * `build-only` future work.
-
-    Note, the fact that `build-only` ignores general `UNSUPPORTED`/`REQUIRES`
-    statements is a current limitation. The logic for taking into account the
-    features that affect compilation, and ignoring those that are only relevant
-    to the execution of the program is currently being worked on.
-
-* `--param test-mode=run-only`
+#### Run only mode
+  
+  Pass: `--param test-mode=run-only`
 
   In this mode, tests will not be compiled, they will only run. To do this only
-  the `RUN:` lines that contain `%{run}`, `%{run-unfiltered-devices}` or `%if
-  run-mode` are executed. Tests are marked as unsupported in the same manner as
-  `full` mode. Since tests are not compiled in this mode, for any test to pass
-  the test binaries should already be in the `test_exec_root` directory, either
-  by having ran `full` or `build-only` modes previously on the system, or having
-  transferred the test binaries into that directory. The `run-mode` feature is
-  added when in this mode.
-
-#### Resolving common Issues with separate compilation and execution
+  the `RUN:` lines that contain a "run" expansion will be executed (`%{run}`,
+  `%{run-unfiltered-devices}`, or `%{run-aux}`). Since tests are not compiled in
+  this mode, for any test to pass the test binaries should already be in the
+  `test_exec_root` directory, either by having ran `full` or `build-only` modes
+  previously on the system, or having transferred the test binaries into that
+  directory. To mark a test as expected to fail at run-time the `XFAIL`
+  expression should use runtime features, such as `run-mode` or device-specific
+  features.
+
+  `%{run-aux}` is an empty expansion and executes a line as is, without
+  expanding for each selected device and without using the `run_launcher`.
+
+#### Build only mode
+
+  Pass: `--param test-mode=build-only`
+
+  This mode can be used to compile all test binaries that can be built on the
+  system. To do this `REQUIRES`, and `UNSUPPORTED` statements are handled
+  differently to accommodate for the fact that in `build-only` mode we do not
+  have any devices, and as a result no device-specific features. Instead of
+  considering these features as missing, we assign a third "unknown" value to 
+  them. When evaluating an expression it will result in an unknown value if its 
+  result could be changed by setting the unknown features to either true or 
+  false. i.e., `false || unknown = unknown` but `true || unknown = true`. If an 
+  expression's final value is unknown we consider it to have met the
+  requirements. The list of device-agnostic features that are not considered
+  unknown in `build-only` is found in the `E2EExpr.py`.
+
+  The triples to compile in this mode are set via the `sycl_build_targets` lit
+  parameter. Valid build targets are: `spir`,`nvidia`, `amd`, `native_cpu`.
+  These correspond to `spir64`, `nvptx64-nvidia-cuda`, `amdgcn-amd-amdhsa`, and
+  `native_cpu` triples respectively. Each build target should be separated with
+  a semicolon. This parameter is set to `all` by default, which enables
+  autodetection for the available build targets. A test can be marked as
+  requiring, or not supporting a particular triple via the `target-*` features.
+  Build targets are selected if they are able to pass the test's requirements
+  independent of the availability of other build targets. This is done to avoid
+  having to deal with a boolean satisfiability problem. For example,
+  `REQUIRES: target-spir && target-nvidia` will always be marked as unsupported
+  since it requires multiple targets simultaneously. Instead we can use
+  `any-target-is-*` features in this case, to check if a target is available in
+  the current lit configuration.
+
+  When executing the test in `build-only`, all `RUN:` lines that do not have a
+  run expansion will execute.
+
+  The `build-mode` feature is added when in this mode.
+
+  Some examples of `REQUIRES`/`UNSUPPORTED` in build-only:
+  If `linux` and `zstd` are available, and `sycl_build_targets` is set to
+  `spir;amd`
+  * `REQUIRES: linux && zstd`: This would be supported, this is treated normally
+  since both features are device-agnostic.
+  * `REQUIRES: linux && sg-32`: Despite the `sg-32` feature not being available,
+  this would be supported. Since the `sg-32` is a device-specific feature it is
+  evaluated as unknown in this expression.
+  * `REQUIRES: windows && sg-32`: This would be unsupported. `sg-32` would be
+  evaluated as unknown, and `windows` would evaluate as false. The fact that we
+  have an unknown value does not affect the end result, since the result of an
+  `&&` expression where one sub-expression is false is always false.
+  * `REQUIRES: windows || sg-32`: this would be supported. Here because the
+  result of the `||` expression would change if we considered `sg-32` to be
+  either true or false the overall expression evaluates to unknown.
+  * `UNSUPPORTED: !sg-32`: this would be supported. `sg-32` is evaluated as
+  unknown, and the negation of unknown is also unknown.
+  * `REQUIRES: target-spir`: This will be supported, and only the `spir64`
+  triple will be selected.
+  * `REQUIRES: target-spir && target-amd`: This will not be supported. When
+  checking if `target-spir` should be selected, `target-amd` will not be an
+  available feature, and vice versa when checking `target-amd`.
+  * `REQUIRES: target-spir && any-target-is-amd`: This will be supported, but
+  only the `spir64` triple will be selected.
+
+#### Common Issues with separate build and run
 
 A number of extra considerations need to be taken to write tests that are able
 to be compiled and executed on separate machines.
 
 * Tests that build and execute multiple binaries need to be written such that
 the output of each compilation has a different name. This way no files are
 overwritten, and all the necessary binaries can be transferred to the running
-system.
+system. For example, instead of setting the output of all compilation steps to
+a file named `%t.out`, we can number them `%t1.out`, `%t2.out`, and so on.
 
 * Two scenarios need to be considered for tests that expectedly fail:
   * Tests that are expected to fail on compilation, and thus also during
-  execution, need to be marked as `XFAIL` with a feature that is device
-  agnostic, or with `XFAIL: *`. Device agnostic features are those which are
-  added added through a method other than processing the output of sycl-ls, for
-  example the OS, or the presence of a library. This needs to be done because
-  sycl-ls is not ran in `build-only` mode.
+  execution, need to be marked as `XFAIL` for a device-agnostic feature, or
+  with `XFAIL: *`. This is due to the fact that there are no devices in
+  `build-only` mode. For example if a test cannot compile for a triple, then it
+  should be marked as `XFAIL` for the corresponding build target feature, rather
+  than a backend feature.
   * If the expected failure occurs during run-time we will need to mark the test
-  with `XFAIL` on a device specific feature (A feature that we add through
-  processing sycl-ls output), or if its expected to always fail on run-time we
-  can use `XFAIL: run-mode`. This is because otherwise the test would compile
-  and pass on `build-only` mode and be reported as an `XPASS`.
-
-* To separate compilation and execution of tests, we classify `RUN:` directives
-as being either build or run lines. If a line contains `%{run}`,
-`%{run-unfiltered-devices}` or `%if run-mode` it is classified as a run line,
-otherwise it is classified as a build line.
-  * All `RUN:` lines that execute test binaries should be marked with either
-  `%{run}` or `%{run-unfiltered-devices}`. Otherwise they will be incorrectly
-  marked as a build line, likely causing a failure at the `build-only` stage as
-  we try to execute the program without having the appropriate devices.
-  * The vast majority of `RUN:` lines that do not execute the test binaries are
-  needed to either set up files prior to compilation, or to compile the binary,
-  as such `RUN:` lines are by default considered as build lines. In the case
-  that we need to run a line on the `run-only` system, and it does not make
-  sense to mark them with `%{run}` or `%{run-unfiltered-devices}`, we can mark a
-  line with `%if run-mode` to specifically make the line a run line. This
-  situation usually appears when we need to run a command in response to the
-  execution of the test binary.
-
-* Currently the `build-only` mode does not support logic to properly assess the
-features in `REQUIRES`/`UNSUPPORTED` to know if a test can be built in the
-system environment, or for `spir64`. Only tests that are marked with `REQUIRES:
-build-and-run-mode` or `UNSUPPORTED: true` are skipped. Thus if a test will fail
-building for the build environment we have on CI or for `spir64` we will need to
-mark this as `REQUIRES: build-and-run-mode`. This is only temporary solution,
-until further work is done to properly mark tests as unsupported on `build-only`
-based on features.
-
-* CPU and FPGA AOT tests are currently expected to fail when compiling and
-executing on separate machines. These failures occur on the `run-only` side,
-because during compilation the host machine's CPU architecture is targeted,
-which may be different than that of the running machine. These tests are marked
-as `REQUIRES: build-and-run-mode` as a result, until they can be refactored to
-compile for the architectures that will be used on the run side.
-
-#### Falling back to `full` testing mode on `run-only`
-
-To not lose coverage of tests marked as `REQUIRES: build-and-run-mode` when
-using `run-only` mode, lit can be called using
-`--param fallback-to-build-if-requires-build-and-run=True`. When this option is
-enabled in `run-only` mode, tests marked as requiring `build-and-run-mode` will
-fallback to running on `full` mode, instead of being reported as unsupported.
+  with `XFAIL` with an expression dependent on runtime features. If it is
+  expected to fail for any device at run-time we can use `XFAIL: run-mode`,
+  This must be done because otherwise the test would compile and pass on
+  `build-only` mode and be reported as an `XPASS`.
+
+* To separate compilation and execution of tests, `RUN:` lines are filtered in
+`build-only` and `run-only` mode based on the presence of "run" expansions.
+  * Any line that is meant to execute the test binary should be marked with
+  `%{run}` or `%{run-unfiltered-devices}` so that it is not ran in `build-only`,
+  and the `run_launcher` substitution is properly employed.
+  * The `%{run-aux}` expansion can be used if a `RUN:` line that does not
+  execute a test binary needs to be ran in `run-only`.
+
+* CPU AOT compilation will target the ISA of the host CPU, thus compiling
+these tests on a different system will lead to failures if the build system and
+run system support different ISAs. To accommodate this, these compilations
+should be delayed to the "run" stage by using the `%{run-aux}` markup.