Make CSE tail recursive #1

mshinwell · 2020-12-10T17:46:33Z

As per #9636 upstream.
(Code review by David Allsopp)

(Code review by David Allsopp)

mshinwell · 2021-04-21T10:17:51Z

#9636 hasn't yet been accepted upstream, but this version of the code has been reviewed.

lpw25

The change seems correct. I guess we might worry about the performance cost. Although I don't remember ever seeing CSE showing up as a hotspot in a profile, so it's probably fine.

mshinwell · 2021-04-21T12:59:21Z

Yeah, I agree.

* Fix ghost locations in modular extension AST nodes * Add missing ghostification Thank you, Carl! * Comment update about ghostiness (+ word-wrapping) * Add ghostify function (#1) * Add `Location.ghostify` * Update the parser's `make_ghost` to save an allocation in some cases * Promote parser.ml * Mark the inner mutable arrays for iarrays as ghost * Add comment about ghostification for comprehensions * Explain that ppxlib is where the ghostiness requirement is enforced * Use `Ast_helper.default_loc` to default the generated locations * Restore propagating the location, now via `Ast_helper.default_loc` * Drop obsolete comment * Rewrite advisory comment about locations

* Parse `#0`, `-#1`, `#2.7`, and `-#3.1`, treating them as boxed * Fix parsing of unsuffixed unboxed int literals * Rewrite CR comment * Update comment * promote-menhir * Unboxed literal tests * Adjust error for unsuffixed unboxed integers

* new solver * better printing * remove solver_mono from mode_intf * document solver_polarized * address some comments * address more comments * rename "constrain" to "zap" * rename constrain_legacy * make depend * make fmt * remove _intf.ml * use comonadic in env * Add Allow_Disallow, restricting types of Obj.magic * Fix type errors discovered in previous commit * Use Allow_disallow in more places * show soundness of allow_disallow * Prevent chance of forgetting the "sound" version * minor changes * Remove the Solver_polarized.morph type (#1) * adjust comments * remove address_of * more comments * make [log] non-optional * better comments * More fixes * bind temp results * add C.eq_morph * set_append_changes * compare Value with Alloc * more comments * warn about infinite lattices * Small simplification to set_obj * example of variable cycle * say "uniqueness_op" because monotonicity * rename set to lift * exhaustive match * inline unusual helper functions * fix rebase issues * Small simplification * some renaming and comments * more comment * remove un-needed assertions. * fix chamelon * make fmt * add file header comments * exhausive match * use ref for morph counting * Reorder in maybe_compose * switch from lift to map * avoid caml_curry * dedup in join and meet * better error message * fix tests * comments * inlining newvar_above/blow, and more uncurrying * remove ?logging * make sure arity is correct and avoid closure allocation * remove morphism counting * force inline to avoid caml_applyX * address comments * rename and phy eq in eq_morphvar * Small improvement to [eq_morphvar] * better comments about lattices * don't hint to use exclave when useless * fix type_argument * comments about inner_alloc_mode * magic for eq_obj and eq_morph * allow/disallow for Value.List * Add comment about actual vs expected * comment mode_argument * eq_morph doesn't take dst * Magic_equal * better comment about adjunction * abstraction in ctype * comment close_over and partial_apply * fix eta expansion bug * Alloc.Const.t is record, not tuple * Simplify interface to [close_over] * Generalize the type of close_over * fixed definition of partial adjoint * better comment about partial adjoint * even better comments * make fmt * bootstrap --------- Co-authored-by: Richard Eisenberg <[email protected]> Co-authored-by: Richard Eisenberg <[email protected]>

…sms (#1760) * new solver * better printing * remove solver_mono from mode_intf * document solver_polarized * address some comments * address more comments * rename "constrain" to "zap" * rename constrain_legacy * make depend * make fmt * remove _intf.ml * use comonadic in env * Add Allow_Disallow, restricting types of Obj.magic * Fix type errors discovered in previous commit * Use Allow_disallow in more places * show soundness of allow_disallow * Prevent chance of forgetting the "sound" version * minor changes * Remove the Solver_polarized.morph type (#1) * adjust comments * remove address_of * more comments * make [log] non-optional * better comments * More fixes * bind temp results * add C.eq_morph * set_append_changes * compare Value with Alloc * more comments * warn about infinite lattices * Small simplification to set_obj * example of variable cycle * say "uniqueness_op" because monotonicity * rename set to lift * exhaustive match * inline unusual helper functions * fix rebase issues * Small simplification * some renaming and comments * more comment * remove un-needed assertions. * fix chamelon * make fmt * add file header comments * exhausive match * use ref for morph counting * Reorder in maybe_compose * switch from lift to map * avoid caml_curry * dedup in join and meet * better error message * fix tests * comments * inlining newvar_above/blow, and more uncurrying * remove ?logging * make sure arity is correct and avoid closure allocation * remove morphism counting * force inline to avoid caml_applyX * address comments * rename and phy eq in eq_morphvar * Small improvement to [eq_morphvar] * better comments about lattices * don't hint to use exclave when useless * fix type_argument * comments about inner_alloc_mode * magic for eq_obj and eq_morph * allow/disallow for Value.List * Add comment about actual vs expected * comment mode_argument * eq_morph doesn't take dst * Magic_equal * better comment about adjunction * abstraction in ctype * comment close_over and partial_apply * fix eta expansion bug * Alloc.Const.t is record, not tuple * Simplify interface to [close_over] * Generalize the type of close_over * fixed definition of partial adjoint * better comment about partial adjoint * even better comments * make fmt * bootstrap --------- Co-authored-by: Richard Eisenberg <[email protected]> Co-authored-by: Richard Eisenberg <[email protected]>

Found by -fsanitize=memory -fsanitize-memory-track-origins: ``` > ==102752==WARNING: MemorySanitizer: use-of-uninitialized-value > #0 0x7f2ba7fb4ea4 in caml_runtime_events_read_poll /var/home/edwin/git/ocaml/otherlibs/runtime_events/runtime_events_consumer.c:496:18 > oxcaml#1 0x7f2ba7fbc016 in caml_ml_runtime_events_read_poll /var/home/edwin/git/ocaml/otherlibs/runtime_events/runtime_events_consumer.c:1207:9 > ocaml#2 0x59ba5c in caml_interprete /var/home/edwin/git/ocaml/runtime/interp.c:1058:14 > ocaml#3 0x5a9220 in caml_main /var/home/edwin/git/ocaml/runtime/startup_byt.c:575:9 > ocaml#4 0x540d6b in main /var/home/edwin/git/ocaml/runtime/main.c:37:3 > ocaml#5 0x7f2ba8120087 in __libc_start_call_main (/lib64/libc.so.6+0x2a087) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) > ocaml#6 0x7f2ba812014a in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2a14a) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) > ocaml#7 0x441804 in _start (/var/home/edwin/git/ocaml/runtime/ocamlrun+0x441804) (BuildId: 617637580ee48eff08a2bce790e1667ad09f3b69) > > Uninitialized value was stored to memory at > #0 0x7f2ba7fb4e9d in caml_runtime_events_read_poll /var/home/edwin/git/ocaml/otherlibs/runtime_events/runtime_events_consumer.c:497:69 > oxcaml#1 0x7f2ba7fbc016 in caml_ml_runtime_events_read_poll /var/home/edwin/git/ocaml/otherlibs/runtime_events/runtime_events_consumer.c:1207:9 > ocaml#2 0x59ba5c in caml_interprete /var/home/edwin/git/ocaml/runtime/interp.c:1058:14 > ocaml#3 0x5a9220 in caml_main /var/home/edwin/git/ocaml/runtime/startup_byt.c:575:9 > ocaml#4 0x540d6b in main /var/home/edwin/git/ocaml/runtime/main.c:37:3 > ocaml#5 0x7f2ba8120087 in __libc_start_call_main (/lib64/libc.so.6+0x2a087) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) > ocaml#6 0x7f2ba812014a in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2a14a) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) > ocaml#7 0x441804 in _start (/var/home/edwin/git/ocaml/runtime/ocamlrun+0x441804) (BuildId: 617637580ee48eff08a2bce790e1667ad09f3b69) > > Uninitialized value was created by an allocation of 'buf' in the stack frame > #0 0x7f2ba7fb3dbc in caml_runtime_events_read_poll /var/home/edwin/git/ocaml/otherlibs/runtime_events/runtime_events_consumer.c:402:7 > ``` This is in fact an EV_LIFECYCLE with EV_RING_STOP, which has 0 additional data, and thus msg_length 2: ``` runtime/runtime_events.c: EV_RUNTIME, (ev_message_type){.runtime=EV_LIFECYCLE}, EV_RING_STOP, 0, ``` Attempting to read from `buf[2]` would read uninitialized data. Signed-off-by: Edwin Török <[email protected]>

The toplevel printer detects cycles by keeping a hashtable of values that it has already traversed. However, some OCaml runtime types (at least bigarrays) may be partially uninitialized, and hashing them at arbitrary program points may read uninitialized memory. In particular, the OCaml testsuite fails when running with a memory-sanitizer enabled, as bigarray printing results in reads to uninitialized memory: ``` ==133712==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x4e6d11 in caml_ba_hash /var/home/edwin/git/ocaml/runtime/bigarray.c:486:45 oxcaml#1 0x52474a in caml_hash /var/home/edwin/git/ocaml/runtime/hash.c:251:35 oxcaml#2 0x599ebf in caml_interprete /var/home/edwin/git/ocaml/runtime/interp.c:1065:14 oxcaml#3 0x5a909a in caml_main /var/home/edwin/git/ocaml/runtime/startup_byt.c:575:9 oxcaml#4 0x540ccb in main /var/home/edwin/git/ocaml/runtime/main.c:37:3 oxcaml#5 0x7f0910abb087 in __libc_start_call_main (/lib64/libc.so.6+0x2a087) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) oxcaml#6 0x7f0910abb14a in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2a14a) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) oxcaml#7 0x441804 in _start (/var/home/edwin/git/ocaml/runtime/ocamlrun+0x441804) (BuildId: 7a60eef57e1c2baf770bc38d10d6c227e60ead37) Uninitialized value was created by a heap allocation #0 0x47d306 in malloc (/var/home/edwin/git/ocaml/runtime/ocamlrun+0x47d306) (BuildId: 7a60eef57e1c2baf770bc38d10d6c227e60ead37) oxcaml#1 0x4e7960 in caml_ba_alloc /var/home/edwin/git/ocaml/runtime/bigarray.c:246:12 oxcaml#2 0x4e801f in caml_ba_create /var/home/edwin/git/ocaml/runtime/bigarray.c:673:10 oxcaml#3 0x59b8fc in caml_interprete /var/home/edwin/git/ocaml/runtime/interp.c:1058:14 oxcaml#4 0x5a909a in caml_main /var/home/edwin/git/ocaml/runtime/startup_byt.c:575:9 oxcaml#5 0x540ccb in main /var/home/edwin/git/ocaml/runtime/main.c:37:3 oxcaml#6 0x7f0910abb087 in __libc_start_call_main (/lib64/libc.so.6+0x2a087) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) oxcaml#7 0x7f0910abb14a in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2a14a) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) oxcaml#8 0x441804 in _start (/var/home/edwin/git/ocaml/runtime/ocamlrun+0x441804) (BuildId: 7a60eef57e1c2baf770bc38d10d6c227e60ead37) SUMMARY: MemorySanitizer: use-of-uninitialized-value /var/home/edwin/git/ocaml/runtime/bigarray.c:486:45 in caml_ba_hash ``` The only use of hashing in genprintval is to avoid cycles, that is, it is only useful for OCaml values that contain other OCaml values (including possibly themselves). Bigarrays cannot introduce cycles, and they are always printed as "<abstr>" anyway. The present commit proposes to be more conservative in which values are hashed by the cycle detector to avoid this issue: we skip hashing any value with tag above No_scan_tag -- which may not contain any OCaml values. Suggested-by: Gabriel Scherer <[email protected]> Signed-off-by: Edwin Török <[email protected]> Co-authored-by: Edwin Török <[email protected]>

mshinwell force-pushed the 4.11 branch 4 times, most recently from d9886b0 to 3f44c2c Compare December 23, 2020 15:32

mshinwell force-pushed the 4.11 branch 6 times, most recently from 2eaced5 to 4564823 Compare February 17, 2021 16:43

mshinwell force-pushed the 4.11 branch 2 times, most recently from 7e986a7 to c00ca0e Compare February 23, 2021 17:49

mshinwell force-pushed the 4.11 branch from 2552e31 to 9813994 Compare March 1, 2021 16:37

Make CSE tail recursive

f75889d

(Code review by David Allsopp)

mshinwell force-pushed the cse-tailrec branch from d820a63 to f75889d Compare April 21, 2021 10:15

lpw25 approved these changes Apr 21, 2021

View reviewed changes

mshinwell merged commit 0ae5245 into oxcaml:4.11 Apr 21, 2021

mshinwell referenced this pull request in mshinwell/oxcaml May 17, 2021

Make CSE tail recursive (#1)

b7c123d

poechsel pushed a commit to poechsel/flambda-backend that referenced this pull request May 26, 2021

Make CSE tail recursive (oxcaml#1)

66ee51d

poechsel pushed a commit to poechsel/flambda-backend that referenced this pull request May 27, 2021

Make CSE tail recursive (oxcaml#1)

e8ee4f6

poechsel pushed a commit that referenced this pull request Jun 28, 2021

Make CSE tail recursive (#1)

9ebd804

poechsel pushed a commit that referenced this pull request Jun 29, 2021

Make CSE tail recursive (#1)

a9e3c66

goldfirere mentioned this pull request Sep 24, 2023

Better error when using a float64 library without extension #1859

Merged

goldfirere referenced this pull request in goldfirere/flambda-backend Jan 1, 2024

Remove the Solver_polarized.morph type (#1)

a3f786e

goldfirere referenced this pull request in goldfirere/flambda-backend Jan 31, 2024

Remove the Solver_polarized.morph type (#1)

f4c9021

Ekdohibs mentioned this pull request Feb 8, 2024

Improved meet for naked immediates (is_int and get_tag) #2259

Open

ncik-roberts mentioned this pull request Apr 8, 2024

Implement mixed blocks in runtime 4 #2422

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Make CSE tail recursive #1

Make CSE tail recursive #1

Uh oh!

mshinwell commented Dec 10, 2020

Uh oh!

mshinwell commented Apr 21, 2021

Uh oh!

lpw25 left a comment

Uh oh!

mshinwell commented Apr 21, 2021

Uh oh!

Uh oh!

Make CSE tail recursive #1

Make CSE tail recursive #1

Uh oh!

Conversation

mshinwell commented Dec 10, 2020

Uh oh!

mshinwell commented Apr 21, 2021

Uh oh!

lpw25 left a comment

Choose a reason for hiding this comment

Uh oh!

mshinwell commented Apr 21, 2021

Uh oh!

Uh oh!