Skip to content

Commit d301539

Browse files
authored
[libc++][docs] Document the libc++ Lit testing format naming scheme (#73136)
As a new contributor, I found it hard to find the documentation for the meaning of the names of different tests and how those names translate to Lit. This patch moves the documentation to the RST documentation we publish on the website instead of leaving it in the source code only.
1 parent 6c92770 commit d301539

File tree

2 files changed

+93
-64
lines changed

2 files changed

+93
-64
lines changed

libcxx/docs/TestingLibcxx.rst

Lines changed: 88 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -325,18 +325,98 @@ This macro is in a different header as ``assert_macros.h`` since it pulls in
325325
additional headers.
326326

327327
.. note: This macro can only be used in test using C++20 or newer. The macro
328-
was added at a time where most of lib++'s C++17 support was complete.
328+
was added at a time where most of libc++'s C++17 support was complete.
329329
Since it is not expected to add this to existing tests no effort was
330330
taken to make it work in earlier language versions.
331331
332332
333-
Additional reading
334-
------------------
335-
336-
The function ``CxxStandardLibraryTest`` in the file
337-
``libcxx/utils/libcxx/test/format.py`` has documentation about writing test. It
338-
explains the difference between the test named ``foo.pass.cpp`` and named
339-
``foo.verify.cpp`` are.
333+
Test names
334+
----------
335+
336+
The names of test files have meaning for the libc++-specific configuration of
337+
Lit. Based on the pattern that matches the name of a test file, Lit will test
338+
the code contained therein in different ways. Refer to the `Lit Meaning of libc++
339+
Test Filenames`_ when determining the names for new test files.
340+
341+
.. _Lit Meaning of libc++ Test Filenames:
342+
.. list-table:: Lit Meaning of libc++ Test Filenames
343+
:widths: 25 75
344+
:header-rows: 1
345+
346+
* - Name Pattern
347+
- Meaning
348+
* - ``FOO.pass.cpp``
349+
- Checks whether the C++ code in the file compiles, links and runs successfully.
350+
* - ``FOO.pass.mm``
351+
- Same as ``FOO.pass.cpp``, but for Objective-C++.
352+
353+
* - ``FOO.compile.pass.cpp``
354+
- Checks whether the C++ code in the file compiles successfully. In general, prefer ``compile`` tests over ``verify`` tests,
355+
subject to the specific recommendations, below, for when to write ``verify`` tests.
356+
* - ``FOO.compile.pass.mm``
357+
- Same as ``FOO.compile.pass.cpp``, but for Objective-C++.
358+
* - ``FOO.compile.fail.cpp``
359+
- Checks that the code in the file does *not* compile successfully.
360+
361+
* - ``FOO.verify.cpp``
362+
- Compiles with clang-verify. This type of test is automatically marked as UNSUPPORTED if the compiler does not support clang-verify.
363+
For additional information about how to write ``verify`` tests, see the `Internals Manual <https://clang.llvm.org/docs/InternalsManual.html#verifying-diagnostics>`_.
364+
Prefer `verify` tests over ``compile`` tests to test that compilation fails for a particular reason. For example, use a ``verify`` test
365+
to ensure that
366+
367+
* an expected ``static_assert`` is triggered;
368+
* the use of deprecated functions generates the proper warning;
369+
* removed functions are no longer usable; or
370+
* return values from functions marked ``[[nodiscard]]`` are stored.
371+
372+
* - ``FOO.link.pass.cpp``
373+
- Checks that the C++ code in the file compiles and links successfully -- no run attempted.
374+
* - ``FOO.link.pass.mm``
375+
- Same as ``FOO.link.pass.cpp``, but for Objective-C++.
376+
* - ``FOO.link.fail.cpp``
377+
- Checks whether the C++ code in the file fails to link after successful compilation.
378+
* - ``FOO.link.fail.mm``
379+
- Same as ``FOO.link.fail.cpp``, but for Objective-C++.
380+
381+
* - ``FOO.sh.<anything>``
382+
- A *builtin Lit Shell* test.
383+
* - ``FOO.gen.<anything>``
384+
- A variant of a *Lit Shell* test that generates one or more Lit tests on the fly. Executing this test must generate one or more files as expected
385+
by LLVM split-file. Each generated file will drive an invocation of a separate Lit test. The format of the generated file will determine the type
386+
of Lit test to be executed. This can be used to generate multiple Lit tests from a single source file, which is useful for testing repetitive properties
387+
in the library. Be careful not to abuse this since this is not a replacement for usual code reuse techniques.
388+
389+
390+
libc++-Specific Lit Features
391+
----------------------------
392+
393+
Custom Directives
394+
~~~~~~~~~~~~~~~~~
395+
396+
Lit has many directives built in (e.g., ``DEFINE``, ``UNSUPPORTED``). In addition to those directives, libc++ adds two additional libc++-specific directives that makes
397+
writing tests easier. See `libc++-specific Lit Directives`_ for more information about the ``FILE_DEPENDENCIES`` and ``ADDITIONAL_COMPILE_FLAGS`` libc++-specific directives.
398+
399+
.. _libc++-specific Lit Directives:
400+
.. list-table:: libc++-specific Lit Directives
401+
:widths: 20 35 45
402+
:header-rows: 1
403+
404+
* - Directive
405+
- Parameters
406+
- Usage
407+
* - ``FILE_DEPENDENCIES``
408+
- ``// FILE_DEPENDENCIES: file, directory, /path/to/file, ...``
409+
- The paths given to the ``FILE_DEPENDENCIES`` directive can specify directories or specific files upon which a given test depend. For example, a test that requires some test
410+
input stored in a data file would use this libc++-specific Lit directive. When a test file contains the ``FILE_DEPENDENCIES`` directive, Lit will collect the named files and copy
411+
them to the directory represented by the ``%T`` substitution before the test executes. The copy is performed from the directory represented by the ``%S`` substitution
412+
(i.e. the source directory of the test being executed) which makes it possible to use relative paths to specify the location of dependency files. After Lit copies
413+
all the dependent files to the directory specified by the ``%T`` substitution, that directory should contain *all* the necessary inputs to run. In other words,
414+
it should be possible to copy the contents of the directory specified by the ``%T`` substitution to a remote host where the execution of the test will actually occur.
415+
* - ``ADDITIONAL_COMPILE_FLAGS``
416+
- ``// ADDITIONAL_COMPILE_FLAGS: flag1 flag2 ...``
417+
- The additional compiler flags specified by a space-separated list to the ``ADDITIONAL_COMPILE_FLAGS`` libc++-specific Lit directive will be added to the end of the ``%{compile_flags}``
418+
substitution for the test that contains it. This libc++-specific Lit directive makes it possible to add special compilation flags without having to resort to writing a ``.sh.cpp`` test (see
419+
`Lit Meaning of libc++ Test Filenames`_), more powerful but perhaps overkill.
340420

341421
Benchmarks
342422
==========

libcxx/utils/libcxx/test/format.py

Lines changed: 5 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -151,38 +151,11 @@ class CxxStandardLibraryTest(lit.formats.FileBasedTest):
151151
"""
152152
Lit test format for the C++ Standard Library conformance test suite.
153153
154-
This test format is based on top of the ShTest format -- it basically
155-
creates a shell script performing the right operations (compile/link/run)
156-
based on the extension of the test file it encounters. It supports files
157-
with the following extensions:
158-
159-
FOO.pass.cpp - Compiles, links and runs successfully
160-
FOO.pass.mm - Same as .pass.cpp, but for Objective-C++
161-
162-
FOO.compile.pass.cpp - Compiles successfully, link and run not attempted
163-
FOO.compile.pass.mm - Same as .compile.pass.cpp, but for Objective-C++
164-
FOO.compile.fail.cpp - Does not compile successfully
165-
166-
FOO.link.pass.cpp - Compiles and links successfully, run not attempted
167-
FOO.link.pass.mm - Same as .link.pass.cpp, but for Objective-C++
168-
FOO.link.fail.cpp - Compiles successfully, but fails to link
169-
170-
FOO.sh.<anything> - A builtin Lit Shell test
171-
172-
FOO.gen.<anything> - A .sh test that generates one or more Lit tests on the
173-
fly. Executing this test must generate one or more files
174-
as expected by LLVM split-file, and each generated file
175-
leads to a separate Lit test that runs that file as
176-
defined by the test format. This can be used to generate
177-
multiple Lit tests from a single source file, which is
178-
useful for testing repetitive properties in the library.
179-
Be careful not to abuse this since this is not a replacement
180-
for usual code reuse techniques.
181-
182-
FOO.verify.cpp - Compiles with clang-verify. This type of test is
183-
automatically marked as UNSUPPORTED if the compiler
184-
does not support Clang-verify.
185-
154+
Lit tests are contained in files that follow a certain pattern, which determines the semantics of the test.
155+
Under the hood, we basically generate a builtin Lit shell test that follows the ShTest format, and perform
156+
the appropriate operations (compile/link/run). See
157+
https://libcxx.llvm.org/TestingLibcxx.html#test-names
158+
for a complete description of those semantics.
186159
187160
Substitution requirements
188161
===============================
@@ -200,30 +173,6 @@ class CxxStandardLibraryTest(lit.formats.FileBasedTest):
200173
in the same command line. In other words, the test format doesn't perform
201174
separate compilation and linking steps in this case.
202175
203-
204-
Additional supported directives
205-
===============================
206-
In addition to everything that's supported in Lit ShTests, this test format
207-
also understands the following directives inside test files:
208-
209-
// FILE_DEPENDENCIES: file, directory, /path/to/file
210-
211-
This directive expresses that the test requires the provided files
212-
or directories in order to run. An example is a test that requires
213-
some test input stored in a data file. When a test file contains
214-
such a directive, this test format will collect them and copy them
215-
to the directory represented by %T. The intent is that %T contains
216-
all the inputs necessary to run the test, such that e.g. execution
217-
on a remote host can be done by simply copying %T to the host.
218-
219-
// ADDITIONAL_COMPILE_FLAGS: flag1 flag2 flag3
220-
221-
This directive will cause the provided flags to be added to the
222-
%{compile_flags} substitution for the test that contains it. This
223-
allows adding special compilation flags without having to use a
224-
.sh.cpp test, which would be more powerful but perhaps overkill.
225-
226-
227176
Additional provided substitutions and features
228177
==============================================
229178
The test format will define the following substitutions for use inside tests:

0 commit comments

Comments
 (0)