Speed up linking by changing cmxa format

[stedolan]

This patch improves linking time, by changing the cmxa format to be smaller and avoid redundant checking of module digests.

On large executables (~600 MB), this shortens linking time by about a third, makes the ocamlopt process take ~10x less memory, and reduces the size of cmxa files by anywhere from 5x to 70x.

(The patch grew larger than I'd like. If it's now unreviewably big, let me know and I'll split it up next week)

The main change is that the format of cmxa files goes from this:

type unit_infos =
  { mutable ui_name: modname;             (* Name of unit implemented *)
    mutable ui_symbol: string;            (* Prefix for symbols *)
    mutable ui_defines: string list;      (* Unit and sub-units implemented *)
    mutable ui_imports_cmi: crcs;         (* Interfaces imported *)
    mutable ui_imports_cmx: crcs;         (* Infos imported *)
    mutable ui_curry_fun: Clambda.arity list; (* Currying functions needed *)
    mutable ui_apply_fun: apply_fn list;  (* Apply functions needed *)
    mutable ui_send_fun: apply_fn list;   (* Send functions needed *)
    mutable ui_export_info: export_info;
    mutable ui_force_link: bool }         (* Always linked *)

type library_infos =
  { lib_units: (unit_infos * Digest.t) list;  (* List of unit infos w/ MD5s *)
    (* In the following fields the lists are reversed with respect to
       how they end up being used on the command line. *)
    lib_ccobjs: string list;            (* C object files needed *)
    lib_ccopts: string list }           (* Extra opts to C compiler *)

to this:

type lib_unit_info =
  { li_name: modname;
    li_symbol: string;
    li_crc: Digest.t;
    li_defines: string list;
    li_force_link: bool;
    li_imports_cmi : Bitmap.t;  (* subset of lib_imports_cmi *)
    li_imports_cmx : Bitmap.t } (* subset of lib_imports_cmx *)

type library_infos =
  { lib_imports_cmi: (modname * Digest.t option) array;
    lib_imports_cmx: (modname * Digest.t option) array;
    lib_units: lib_unit_info list;
    lib_generic_fns: generic_fns;
    (* In the following fields the lists are reversed with respect to
       how they end up being used on the command line. *)
    lib_ccobjs: string list;            (* C object files needed *)
    lib_ccopts: string list }           (* Extra opts to C compiler *)

Instead of keeping a set of imported cmi/cmx files and hashes for each module in a library, the cmxa file now contains one unified list of all of the cmi/cmx files and hashes, and each module contains a bitmap indicating which of these it depends on. Since the dependencies of individual modules in a library tend to be almost identical, this saves a lot of space, as well as saving time during linking (since we only need to check them for consistency once). Similarly, the set of generic functions (caml_curry and friends) is now maintained once per library rather than separately for each module.

The other change to linking is to read cmxas, check consistency and construct the list of objects to link in a single pass, rather than in three separate passes. This removes the need to hold all of the cmxa files in memory at once, greatly reducing the maximum memory usage (and speeding things up somewhat)

[mshinwell]

@xclerc and I are going to review this.

[mshinwell]

This looks good. Two comments:

• It looked to me that the ml_objfiles_rev variables are maybe actually just ml_objfiles, since the code uses List.fold_right. It would be worth double-checking that the order matches the original code here.
• I'm unsure why we need to change ocaml/asmcomp/ etc. and it's maybe best if we don't, to ease merging with upstream. I read the backend/ version of the diff to those files in detail but not the ones in asmcomp/.

Is it worth trying to upstream this patch right away?

@xclerc could you proceed with your review on this one please?

[xclerc]

Sorry, I keep forgetting the policy: should we update
the magic numbers?

[mshinwell]

Sorry, I keep forgetting the policy: should we update the magic numbers?

Good point, yes I think we should do that for the formats involved.

[stedolan]

@mshinwell:

This looks good. Two comments:

• It looked to me that the ml_objfiles_rev variables are maybe actually just ml_objfiles, since the code uses List.fold_right. It would be worth double-checking that the order matches the original code here.

There was indeed an issue there! (The reason for the rev is that the C linker expects the opposite order of objects compared to ocamlopt, and there was some confusion over which function should do this reversal). Fixed now, and the linker command line is now unchanged.

• I'm unsure why we need to change ocaml/asmcomp/ etc. and it's maybe best if we don't, to ease merging with upstream. I read the backend/ version of the diff to those files in detail but not the ones in asmcomp/.

This was because of an issue with the build system, now resolved since #585 has been merged. The ocaml/asmcomp directory is no longer affected by this PR.

Sorry, I keep forgetting the policy: should we update the magic numbers?

Good point, yes I think we should do that for the formats involved.

I bumped the relevant numbers.