Skip to content

Commit a645b6b

Browse files
dcharkesCommit Queue
dcharkes
authored and
Commit Queue
committed
[doc/ffi] Contributing to dart:ffi doc
Change-Id: I6d75a63b02bec44e8850d2380bc37b332b23d0e4 Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/380261 Reviewed-by: Hossein Yousefi <[email protected]> Reviewed-by: Liam Appelbe <[email protected]> Commit-Queue: Daco Harkes <[email protected]>
1 parent 7e2aecb commit a645b6b

File tree

1 file changed

+308
-0
lines changed

1 file changed

+308
-0
lines changed

‎runtime/docs/contributing_to_dart_ffi.md

Lines changed: 308 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,308 @@
1+
# Contributing to Dart FFI
2+
3+
First, go through the [general contributions guide](https://github.com/dart-lang/sdk/blob/main/CONTRIBUTING.md).
4+
5+
6+
# Dart FFI architecture
7+
8+
Dart FFI is implemented in four main locations:
9+
10+
1. the `dart:ffi` library files,
11+
2. the analyzer,
12+
3. the CFE (common front end), and
13+
4. the VM (virtual machine).
14+
15+
The `dart:ffi` library files contain the public API for `dart:ffi`.
16+
The analyzer checks Dart code using `dart:ffi` does not contain any errors when opening an IDE or using dart analyze.
17+
The CFE performs similar checks, but does so when running Dart code with `dart run` or compiling Dart code with `dart compile`.
18+
If there are no errors, the CFE “transforms” or “desugars” most `dart:ffi` code into lower level concepts.
19+
These are encoded in the kernel file.
20+
The VM (or precompiler) takes the kernel file and compiles it into machine code.
21+
22+
```
23+
╭─────────────╮
24+
 │╭─────────────╮       ╔══════════╗
25+
 ││╭─────────────╮┣━━━▶ ║ Analyzer ║ ┣━━━▶  Error messages
26+
 ┆││ Dart Source │      ╚══════════╝
27+
 ┆┆│             │
28+
  ┆┆             ┆
29+
   ┆             ┆
30+
31+
 ╭─────────────╮
32+
 │╭─────────────╮       ╔═════╗
33+
 ││╭─────────────╮┣━━━▶ ║ CFE ║ ┣━━━▶  Error messages
34+
 ┆││ Dart Source │      ╚═════╝
35+
 ┆┆│             │
36+
  ┆┆             ┆
37+
   ┆             ┆
38+
39+
 ╭─────────────╮                       ╭────────────╮
40+
 │╭─────────────╮       ╔═════╗        │╭────────────╮        ╔════╗
41+
 ││╭─────────────╮┣━━━▶ ║ CFE ║ ┣━━━▶  ││╭────────────╮ ┣━━━▶ ║ VM ║
42+
 ┆││ Dart Source │      ╚═════╝        │││ Kernel AST │       ╚════╝
43+
 ┆┆│             │                     ╰││ (binary)   │
44+
  ┆┆             ┆                      ╰│            │
45+
   ┆             ┆                       ╰────────────╯
46+
```
47+
48+
The analyzer and CFE are implemented in Dart code.
49+
The VM is implemented in C++. The analyzer and CFE are relatively easy to work on.
50+
Working on the VM is more complicated.
51+
So let’s focus on the analyzer and CFE first.
52+
53+
Most bugs/features require contributing to various parts of the implementation in the FFI.
54+
But some bugs might be localized to for example only the analyzer.
55+
56+
57+
# Contributing to the Dart FFI library files
58+
59+
The public API and some of the implementation is ordinary Dart code.
60+
61+
62+
## Implementation files
63+
64+
- `sdk/lib/ffi/(.*).dart` contains the public API, it hides implementation details.
65+
- `sdk/lib/_internal/vm/lib/ffi(.*)_patch.dart` contains the implementation details hidden from the public API.
66+
67+
However, many of the functions are either marked `external` or their implementation throws.
68+
The real implementation of these functions is in the VM.
69+
70+
71+
# Contributing to the Dart FFI analyzer implementation
72+
73+
## Documentation
74+
75+
Familiarize yourself with the architecture of the analyzer by reading the documentation on the [AST (abstract syntax tree)](https://github.com/dart-lang/sdk/blob/main/pkg/analyzer/doc/tutorial/ast.md) and [element model](https://github.com/dart-lang/sdk/blob/main/pkg/analyzer/doc/tutorial/element.md).
76+
77+
78+
## Implementation files
79+
80+
- `pkg/analyzer/lib/src/generated/ffi_verifier.dart`. The whole implementation of the FFI in the analyzer is in this file.
81+
82+
83+
## Test files
84+
85+
Unit tests
86+
87+
- `pkg/analyzer/test/src/diagnostics/ffi_(.*)_test.dart`. Run with `$ dart pkg/analyzer/test/src/diagnostics/ffi_native_test.dart`. These unit tests don’t require rebuilding the SDK.
88+
- These unit tests are run against a mock SDK: `pkg/analyzer/lib/src/test_utilities/mock_sdk.dart`.
89+
90+
Integration tests
91+
92+
- `tests/ffi/static_checks/(.*)_test.dart`. Prefer creating new tests with the “new style” `// [analyzer] ...`. The tests can be run with `$ tools/build.py -mrelease runtime create_platform_sdk && tools/test.py -cfasta -mrelease tests/ffi/static_checks/vmspecific_static_checks_array_test.dart`. These tests require rebuilding the SDK. These tests also can contain `// [cfe] ...` error expectations. Which makes them good for checking that the behavior is similar in the CFE and analyzer implementation. And because these test files are standalone Dart, you can easily pass them as an argument to the analyzer when running from source in the debugger.
93+
94+
95+
## Running in the debugger
96+
97+
If you’re using vscode, you can use the following configuration to run the analyzer from source and be able to set breakpoints.
98+
99+
```
100+
{
101+
"name": "dart analyzer.dart",
102+
"type": "dart",
103+
"request": "launch",
104+
"program": "pkg/analyzer_cli/bin/analyzer.dart",
105+
"args": [
106+
"tests/ffi/static_checks/vmspecific_static_checks_array_test.dart",
107+
],
108+
"toolArgs": [],
109+
"enableAsserts": true,
110+
"cwd": "${workspaceFolder}",
111+
},
112+
```
113+
114+
Replace the file under test in args as needed.
115+
116+
117+
## Gotcha’s
118+
119+
- Methods must be alphabetically sorted. Run `$ dart pkg/analysis_server/test/verify_sorted_test.dart`.
120+
121+
122+
## Learning from already merged PRs
123+
124+
You can learn a lot from looking at previous PRs:
125+
126+
- [Git history of the ffi_verifier.dart](https://github.com/dart-lang/sdk/commits/main/pkg/analyzer/lib/src/generated/ffi_verifier.dart).
127+
128+
129+
# Contributing to the Dart FFI CFE implementation
130+
131+
## Implementation files
132+
133+
- `pkg/vm/lib/modular/transformations/ffi/(.*).dart` contains the CFE transformations for `dart:ffi`.
134+
135+
136+
## Test files
137+
138+
- `pkg/vm/testcases/transformations/ffi/(.*).dart` are the input files and the `.expect` files are a human readable version of kernel files. The tests can be run and their expect files updated with `$ tools/build.py -ax64 -mrelease create_platform_sdk runtime && dart -DupdateExpectations=true pkg/vm/test/transformations/ffi_test.dart`.
139+
140+
141+
## Running in the debugger
142+
143+
Running the transformation tests from source in the debugger can be done with the following configuration.
144+
145+
```
146+
{
147+
"name": "dart pkg/vm/test/transformations/ffi_test.dart",
148+
"type": "dart",
149+
"request": "launch",
150+
"program": "pkg/vm/test/transformations/ffi_test.dart",
151+
"args": [
152+
"compound_copies",
153+
],
154+
"toolArgs": [
155+
"-DupdateExpectations=true",
156+
],
157+
"enableAsserts": true,
158+
"cwd": "${workspaceFolder}",
159+
},
160+
```
161+
162+
Running the CFE on a Dart file can be done with the following configuration.
163+
164+
```
165+
{
166+
"name": "dart gen_kernel.dart",
167+
"type": "dart",
168+
"request": "launch",
169+
"program": "pkg/vm/bin/gen_kernel.dart",
170+
"args": [
171+
"--platform=${workspaceFolder}/xcodebuild/ReleaseARM64/vm_platform_strong.dill",
172+
"third_party/pkg/native/pkgs/ffi/test/allocation_test.dart",
173+
],
174+
"toolArgs": [],
175+
"enableAsserts": true,
176+
"cwd": "${workspaceFolder}",
177+
},
178+
```
179+
180+
You can read more about kernel format in the [VM readme](https://github.com/dart-lang/sdk/blob/main/runtime/docs/README.md).
181+
182+
183+
# Contributing to the the Dart FFI implementation in the VM
184+
185+
In order to be able to work on the VM implementation of Dart FFI, you first need to familiarize yourself with the Dart VM:
186+
187+
- [The VM readme](https://github.com/dart-lang/sdk/blob/main/runtime/docs/README.md)
188+
- [The VM glossary](https://github.com/dart-lang/sdk/blob/main/runtime/docs/glossary.md)
189+
190+
191+
## Implementation files
192+
193+
- `runtime/vm/compiler/ffi/`: relatively self-contained files for the FFI. For example the logic for which registers of the CPU arguments must be passed in calls to C.
194+
- `runtime/vm/compiler/frontend/kernel_to_il.cc`: Contains most of the IL generation.
195+
- `runtime/vm/compiler/backend/il(.*).cc`: Contains most of the machine code generation.
196+
197+
198+
## Test files
199+
200+
Unit tests
201+
202+
- `runtime/bin/ffi_unit_test/run_ffi_unit_tests.cc` and `runtime/vm/compiler/ffi/(.*)_test.cc` contains unit tests for some parts of the FFI. Run (and update) with `$ tools/build.py -mrelease run_ffi_unit_tests && tools/test.py -mrelease --vm-options=--update ffi_unit`.
203+
204+
Integration tests
205+
206+
- `tests/ffi/(.*)_test.dart` Integration tests. Run for AOT on host machine with `$ tools/build.py -mdebug create_platform_sdk runtime ffi_test_functions dart_precompiled_runtime && tools/test.py -mdebug -cdartkp ffi`. Run for JIT on host machine with `$ tools/build.py -mdebug create_platform_sdk runtime ffi_test_functions && tools/test.py -mdebug ffi`.
207+
208+
Test generators
209+
210+
- `tests/ffi/generator/(.*).dart` contains test generators. Writing tests for Dart FFI that need to cover a wide variety of slightly different cases is tedious and error prone. Instead, we prefer generating tests in such cases.
211+
212+
213+
## Running in the debugger
214+
215+
If you’re using vscode on an Arm64 Mac you can use the following configuration. The first command is for running JIT. The second command is for precompiling AOT and the third command is for running AOT. You can find the specific commands to run by passing `-v` to the test.py invocations from above.
216+
217+
For Linux, use out/DebugX64 as the out directory and gdb as MIMode.
218+
219+
```
220+
{
221+
"launch": {
222+
"version": "0.2.0",
223+
"configurations": [
224+
{
225+
"name": "ccpdbg dart",
226+
"type": "cppdbg",
227+
"request": "launch",
228+
"program": "${workspaceFolder}/xcodebuild/DebugARM64/dart",
229+
"args": [
230+
// "--print-flow-graph",
231+
// "--print-flow-graph-filter=Ffi",
232+
// "--disassemble",
233+
"tests/ffi/address_of_typeddata_generated_test.dart",
234+
],
235+
"stopAtEntry": false,
236+
"cwd": "${workspaceFolder}",
237+
"environment": [],
238+
"externalConsole": false,
239+
"MIMode": "lldb",
240+
"sourceFileMap": {
241+
"runtime/": "${workspaceFolder}/runtime/",
242+
},
243+
},
244+
{
245+
"name": "ccpdbg gen_snapshot",
246+
"type": "cppdbg",
247+
"request": "launch",
248+
"program": "${workspaceFolder}/xcodebuild/DebugARM64/gen_snapshot",
249+
"args": [
250+
"--snapshot-kind=app-aot-assembly",
251+
"--assembly=/Users/dacoharkes/dart-sdk/sdk/xcodebuild/DebugARM64/generated_compilations/custom-configuration-4/tests_ffi_isolate_independent_il_based_global_var_test/out.S",
252+
"--iic-impl-il",
253+
"-Dtest_runner.configuration=custom-configuration-4",
254+
"--ignore-unrecognized-flags",
255+
"--packages=/Users/dacoharkes/dart-sdk/sdk/.packages",
256+
"/Users/dacoharkes/dart-sdk/sdk/xcodebuild/DebugARM64/generated_compilations/custom-configuration-4/tests_ffi_isolate_independent_il_based_function_call_2_test/out.dill",
257+
],
258+
"stopAtEntry": false,
259+
"cwd": "${workspaceFolder}/xcodebuild/DebugARM64/",
260+
"environment": [],
261+
"externalConsole": false,
262+
"MIMode": "lldb",
263+
"sourceFileMap": {
264+
"../../": "${workspaceFolder}/",
265+
},
266+
},
267+
{
268+
"name": "ccpdbg dart_precompiled_runtime",
269+
"type": "cppdbg",
270+
"request": "launch",
271+
"program": "${workspaceFolder}/xcodebuild/DebugARM64/dart_precompiled_runtime",
272+
"args": [
273+
"/Users/dacoharkes/dart-sdk/sdk/xcodebuild/DebugARM64/generated_compilations/custom-configuration-4/runtime_tests_vm_dart_memoizable_idempotent_test/out.aotsnapshot",
274+
],
275+
"stopAtEntry": false,
276+
"cwd": "${workspaceFolder}/xcodebuild/DebugARM64/",
277+
"environment": [],
278+
"externalConsole": false,
279+
"MIMode": "lldb",
280+
"sourceFileMap": {
281+
"../../": "${workspaceFolder}/",
282+
},
283+
},
284+
```
285+
286+
## Explore the history.
287+
288+
You can learn a lot by exploring PRs that implement various features.
289+
290+
- [PRs that modify the IL generation](https://dart-review.googlesource.com/q/owner:[email protected]+size:%3E300+kernel_to_il.cc+status:merged) for the FFI.
291+
- [PRs that modify the machine code generation](https://dart-review.googlesource.com/q/owner:[email protected]+size:%3E300+il_x64.cc+status:merged) for the FFI.
292+
293+
294+
# What to work on.
295+
296+
You can look at [issues labeled ‘library:ffi’](https://github.com/dart-lang/sdk/issues?q=is%3Aopen+is%3Aissue+label%3Alibrary-ffi).
297+
More specifically you can also apply the label [’contributions-welcome’](https://github.com/dart-lang/sdk/issues?q=is%3Aopen+is%3Aissue+label%3Acontributions-welcome).
298+
299+
300+
# Preparing  PR
301+
302+
The first line of the commit message is usually `[vm/ffi] ...`.
303+
304+
If your PR modifies the running of Dart files (so not only error messages in analyzer and/or CFE), we should ensure it’s tested on all platforms.
305+
To add all the CI bots that cover running the FFI in different configurations, you can add a footer to the commit message with the result of `$ tools/find_builders.dart ffi/function_structs_by_value_generated_args_test`.
306+
(This only works for already existing tests, so if you’re adding a new test, you can get the list of bots from a pre-existing test, for example the one above.)
307+
308+
Prefer using Gerrit directly for uploading a PR rather than using pull requests via GitHub. For more info see [the Contributing guide](https://github.com/dart-lang/sdk/blob/main/CONTRIBUTING.md#uploading-the-patch-for-review).

0 commit comments

Comments
 (0)