Add gleam support to Mix #14262

Papipo · 2025-02-13T09:55:10Z

This PR adds support for the gleam language.

Support for gleam.toml in path deps
Support gleam in deps loader.
Support for gleam in deps.compile
Require specific gleam binary version (hardcoded to 1.9.0 for now, see below)
Handle absent gleam binary
Support git deps (support for them in gleam is in main already but not yet released)
Setup gleam on CI (Otherwise gleam tests will be skipped).

Notes:

Packages having a gleam binary version requirements are handled automatically when executing gleam compile-package.
The gleam binary version requirement for mix integration is hardcoded because I could just try to run gleam export package-info but I can't know the reason for a bad exit status (maybe the dep path was wrong and the command was run in a dir without a gleam.toml).
The required command to export information about gleam packages and their deps is not yet released but has already been merged. Should land on v1.10.0.
When there are [erlang] options set in the gleam.toml a .app file is generated in the ebin folder of the dependency. Right now this generated file is very basic, having only the vsn, applications (if extra_applications was specified) and mod (if application_start_module was specified). Not sure if there is room for improvement there (like using a .app.src file instead? I don't know the internals of this).

lib/mix/lib/mix/gleam.ex

lib/mix/lib/mix/tasks/deps.compile.ex

Papipo · 2025-02-17T09:02:55Z

@josevalim when working on this I was always thinking of erlang target packages, but it would be nice to be able to use javascript ones as well (ie. to use them in Phoenix frontend). Not sure how we can handle that, any hints?

josevalim · 2025-02-17T09:55:52Z

It should be possible because we use heroicons, a regular JS package, as a dependency on Phoenix applications. So we could probably support a :gleam_targets or similar option that is used by by deps.compile? You will probably need to set app: false true, similar to how heroicons are added.

lpil · 2025-02-17T13:24:48Z

Hello! You shouldn't need to do anything special target-wise as Gleam performs dead code elimination for code that is for other targets.

@josevalim when working on this I was always thinking of erlang target packages, but it would be nice to be able to use javascript ones as well (ie. to use them in Phoenix frontend). Not sure how we can handle that, any hints?

I would not expect Mix to become a front-end build tool by adding Gleam support. I think the workflow should be the same as it is today etc, with the desired build tool (esbuild, webpack, gleam etc) being run as a Phoenix watcher.

Adding frontend dependencies to the BEAM application dependency tree would either be wasted work compiling and managing them, or the programmer would need to go through every dependency in the lock file and add configuration to their mix.exs for each one they don't want to use on the backend.

We are going to #14262 (comment), so I also need the target config, if any. I think that just exporting everything is the best bet.

There is no such config. A Gleam package doesn't have an explicit target, each function in the public API is available on Erlang, JavaScript, or both, and they get either compiled or eliminated as needed. Whether a package is for Erlang or JavaScript is typically a matter of how the dependant is using them, not a matter of the configuration of the dependency.

There is a target flag in gleam.toml, but that is "if I run code from this page (almost always the tests) what target should be used by default, unless otherwise specified".

lib/mix/lib/mix/tasks/deps.compile.ex

josevalim · 2025-02-20T12:18:08Z

lib/mix/test/test_helper.exs

@@ -269,6 +269,15 @@ Enum.each(fixtures, fn fixture ->
  File.cp_r!(source, dest)
 end)

+## Set up Gleam fixtures
+


And we should probably skip gleam tests if gleam is not installed, right?

Done. I guess I need to add gleam to CI (although I need to wait until my gleam PR is merged and released).

- Add Mix.Gleam module - Add specific gleam binary version requirement - Rely on `gleam export package-info`

- shell_cmd! wasn't handling tuples - Fix documentation

This is an optional value within [erlang] in the gleam.toml file. It will be used for the `mod` value when generating a .app file

Papipo · 2025-03-26T08:54:36Z

I am kind of copypasting this from the Gleam discord in case someone wants to try this out before merging into main:

What is this?

These changes add first-class support for Gleam in Elixir's Mix, thus allowing gleam path dependencies in mix.exs.
Those dependencies are expected to be 100% valid gleam projects and as such need to have a gleam.toml file.

What's missing?

Probably nothing. Work on the gleam compiler is already merged and work on the Mix side is completed, but there might me rough edges, that's what we need to test.

Instructions

Mix integration will expect you to have gleam >= 1.10.0 installed and that hasn't been released yet. So:

Install Gleam using the nightly release.
- Notice that some package managers might allow you to install it this way.
Grab this elixir branch.
Edit lib/mix/lib/mix/gleam.ex and replace @required_gleam_version with 1.9.0.
Run make in the root folder of the Elixir project and add its bin folder to your $PATH

If you were using mix_gleam, undo its setup.

Usage

After running mix deps.get the Mix workflow doesn't change. deps.compile should compile Gleam path deps.
If you use gleam code from your elixir app, it should just work (as long as you have a gleam.toml file - you might want to move gleam code to a sibling folder or into gleam/src) I think that otherwise the mix.exs file will take precedence over the gleam one.

What needs to be tested?

Anything related to Erlang/OTP shenanigans (.app files, extra_applications and ensuring those start, etc.)
If you have an existing project using mix_gleam, that's a great opportunity to replace it with this.
Gleam Git dependencies.
Just try to break it please.

Thanks

Just report back with your findings and: thanks :person_bowing:

Note: I hope these instructions are enough and correct. Let me know if that's not the case.

lib/mix/lib/mix/tasks/deps.compile.ex

josevalim · 2025-03-26T09:02:29Z

@Papipo thanks for all the work so far!

One question: I thought that gleam shipped as a Hex package, so wouldn't it be possible for the user to depend on the gleam package or similar instead of us depending on a system wide gleam installation?

Papipo · 2025-03-26T09:08:58Z

@Papipo thanks for all the work so far!

One question: I thought that gleam shipped as a Hex package, so wouldn't it be possible for the user to depend on the gleam package or similar instead of us depending on a system wide gleam installation?

No, gleam is a single binary written in Rust. I don't know if we can provide a hex archive maybe? I don't know how these work.

How is rebar installation handled? I know that mix is able to install it locally, but I am not sure what that means exactly. It's still a binary as well, right?

josevalim · 2025-03-26T09:14:48Z

For Rebar, we have some binaries which we upload to hex.pm, and we version control them every year or so. Although I don't think this will work for Gleam, because people want to upgrade it more frequently, and Rebar is only a build tool, not really a compiler. So I guess we need to be clear we are using the system one and it is up to them to enforce the team uses the same version across the board (which is how we deal with make and the elixir executable itself).

Papipo · 2025-03-26T09:17:26Z

For Rebar, we have some binaries which we upload to hex.pm, and we version control them every year or so. Although I don't think this will work for Gleam, because people want to upgrade it more frequently, and Rebar is only a build tool, not really a compiler. So I guess we need to be clear we are using the system one and it is up to them to enforce the team uses the same version across the board (which is how we deal with make and the elixir executable itself).

In fact the binary is also the LS, so you definitely want to have full control of the version you are using, etc.

josevalim · 2025-04-02T13:20:20Z

Quoting @lpil on the forums:

The Gleam compiler only generates Erlang from Gleam modules, but it doesn’t know what any Elixir modules in the package expand to, so it cannot create .app files unless it owns the whole project and so will be performing Elixir and Erlang compilation. This means that any build tool that uses the Gleam compiler will need to generate the required BEAM .beam and .app files.

Apologies but I am a bit confused. We are using gleam compile-package which emits the .beam files already, right? Here is what we invoke: https://github.com/elixir-lang/elixir/pull/14262/files#diff-da9fa810e81e1fe6ac8d4defde11dc457f0066fc9e442e85a96ea2e182414eacR336 - in this case, doesn't Gleam already own the whole package? We don't invoke anything else so if additional steps are necessary, they are not being executed here.

The .app file has things like app name, version, description, registered processes, app environment, and module names. We would need to lift all of this information from the Gleam .toml and make assumptions on how Gleam wants those be used. Worst case scenario, couldn't Gleam support an --app file on compile-package that generates it, based on the outputs of its --out directory?

Papipo · 2025-04-02T14:44:06Z

Yes, compile-package does emit beam files.

From @lpil comment over elixirforum I assume the problem are colocated Elixir files within a Gleam project. The Gleam compiler knows nothing about them so any modules they define can't be written in the .app file, I guess.

What I have done here is use the Mix task to generate the .app file (and inject the two options gleam.toml supports for the erlang target: mod and extra_applications). As you can see we now have a full-fledged .app file.

Let me know if this looks good or if I should revert it. Thanks.

josevalim · 2025-04-02T16:00:32Z

Yes, for collocated projects that’s the responsibility of Mix, but for packages I would say that’s Gleam responsibility. Otherwise there is even more we need to understand from Gleam TOML’s and we may need to keep track of it as it evolves. So is there a chance for it to emit an .app file either by default or via a flag? On the plus side, if Rebar wants to integrate Gleam as well, they can reuse it to emit apps too. :)

Papipo · 2025-04-07T07:28:27Z

Apparently this seems to be working fine except that people sometimes need to compile twice . Could this be code path related maybe? It's probably something stupid I missed.

josevalim · 2025-04-07T08:05:04Z

Can they upload a repo that reproduces the error? Once we move the .app building to gleam compile-package, I think this is good to go :)

lpil · 2025-04-07T12:00:08Z

Hi gang!

I'm a bit stuck as to how gleam compile-package would create a .app file as the Gleam compiler doesn't know the names of all the BEAM modules produced by compiling the package.

We could produce a .app.src file for Mix to parse, edit, and write. Would that work?

josevalim · 2025-04-07T12:20:51Z

I think there is a misunderstanding somewhere, probably on my side, so I will try to break down what I understand is happening here and please correct me if I am wrong.

This is a pull request to compile Gleam packages, either from Hex or Git
The package may have dependencies, which we download and make them available before we compile the package
When it is time to compile the package, we call gleam compile-package with these args: ["--target", "erlang", "--package", package, "--out", out, "--lib", lib]
We don't compile anything else

So because the only thing we call is gleam compile-package then I am assuming it compiles gleam files into Erlang ones and then the Erlang ones into .beam. Otherwise, this task is not enough, as it would otherwise simply emit Erlang files that I would not be able to invoke/use.

However, when you say:

I'm a bit stuck as to how gleam compile-package would create a .app file as the Gleam compiler doesn't know the names of all the BEAM modules produced by compiling the package.

It seems gleam compile-package is expecting something else to run or to compile the package. Then the question is what is this something after? My expectation is that Gleam would fully know how to compile a "Gleam package".

Thank you,

lpil · 2025-04-07T12:47:35Z

Thanks for the info, I understand better now.

So because the only thing we call is gleam compile-package then I am assuming it compiles gleam files into Erlang ones and then the Erlang ones into .beam. Otherwise, this task is not enough, as it would otherwise simply emit Erlang files that I would not be able to invoke/use.

Ah! This is the misunderstanding. gleam compile-package is an API for the Gleam compiler so it compiles Gleam modules to Erlang. It does not compile Erlang and Elixir to BEAM, that is the responsibility of a BEAM tool that will run the Gleam compiler, the Erlang compiler, and the Elixir compiler.

Because there's no mapping between Elixir file names and BEAM files, and because Gleam packages commonly contain and use Elixir code, there's no way to for the Gleam compiler to know the names of all the BEAM modules that would need to be listed in the .app file.

Papipo · 2025-04-07T13:16:18Z

I think rebar compile is able to use an .app.src file, compile the erlang files in /src and then fill in the modules entry in the resulting .app. If we add Elixir to the Mix 🍸 how would this work?

lpil · 2025-04-07T13:21:14Z

The output of gleam compile-package isn't a rebar project, so I wouldn't expect it to be compiled by rebar. I would have expected Mix's Elixir+Erlang compilation capability would be used to compile the Elixir and Erlang modules.

I suspect José will have some guidance on the best approach to creating the .app file. The two options I can immediately see are gleam creating a .app.src file for Mix to modify, or for Mix to create it (seeing as it has all the information already from gleam export package-information's stable interface + the output of it compiling the Elixir+Erlang). There could be some other options also.

josevalim · 2025-04-07T13:22:11Z

Thanks!

Ah! This is the misunderstanding. gleam compile-package is an API for the Gleam compiler so it compiles Gleam modules to Erlang. It does not compile Erlang and Elixir to BEAM, that is the responsibility of a BEAM tool that will run the Gleam compiler, the Erlang compiler, and the Elixir compiler.

So when consuming a package like gleam_stdlib, how do I know which additional tool should I run? Because it could be Rebar, it could be Mix, or whatever, right?

lpil · 2025-04-07T13:40:39Z

There's some confusion here due to the gleam binary containing both a build tool and a compiler. Here Mix is using the Gleam compiler, not the Gleam build tool (which is unsuited to this task as it's not flexible enough).

Build tools that compile Gleam packages (e.g. the Gleam build tool, Make, Nix, and hopefully Mix soon!) need to compile the Erlang produced by the Gleam compiler + any Elixir files into .BEAM modules and through some means create a .app with the final list of modules.

The logic for picking a tool is like so:

.erl -> erlc
.ex -> elixirc
.gleam -> gleam compile-package + erlc

josevalim · 2025-04-07T14:00:35Z

Well put. That highlights precisely the issue here: for compiling dependencies, Mix expects build tools, not compilers. If you look at the file here, the other options are Mix, Rebar and Make, which are all build tools. In a nutshell, Mix uses a build tool to manage dependencies, but it uses compilers to compile your own project. So gleam compile-package would be perfect if someone wants to have Gleam and Elixir together in the same project but not for dependencies.

So this puts us at an impasse because if we use Gleam only as a compiler for dependencies, then we will have to reimplement Gleam's own build tool, as seen in this pull request which already has to parse the .toml file to deal with application_start_module and extra_applications keys. Any time you add new keys under the [erlang] namespace, we will have to play catch up and add them there, which means Mix needs to know more about Gleam than the other build tools. For the other tools, we only care about deps.

So I think it would be really important to be able to use Gleam as a build tool. For what is worth, Rebar3 had to add features for it to be compilable by Mix and Mix had to add features for it to be compilable by Rebar3 (most of those features were basically configuring the path things are written to).

lpil · 2025-04-07T14:31:10Z

Unless OTP adds new functionality there will be no new additions to the Erlang configuration for Gleam, it purely is a description of the .app file.

It's unlikely we'll ever have capacity to make a second Rust based Gleam build tool specifically for Mix to use, but even if we did it would produce quite an inferior experience as it would be a BEAM instance starting a Gleam instance starting a BEAM instance. This is a higher overhead than the Gleam build tool's original design where a single Gleam instance created a BEAM instance per-package, which significantly impacted compile times.

Mix already has the mechanisms required to compile a directory of Elixir and Erlang, so the only missing piece is how to generate the .app file.

Alternatively we could write a Gleam build tool in a single module of Elixir or Erlang, possibly by adapting the Gleam build tool's Erlang and Elixir compile daemon. This would be straightforward and easy, and then it could possibly be vendored in the Mix codebase?

Or perhaps it could be supplied via Hex in some way. I am not familiar with the Mix plugin system so I don't know what is possible there.

How do either of these options sound to you?

josevalim · 2025-04-07T15:42:59Z

but even if we did it would produce quite an inferior experience as it would be a beam instance starting a Gleam instance starting a BEAM instance.

Oh, this is a very good point. So it seems the current direction is the way to go indeed. Thank you.

@Papipo, doesn't this PR then needs to be changed to compile the Erlang source code generated by the Gleam package? Implementation wise, I wonder if perhaps we should drop a mix.exs in disk and then just call mix_dep? This way we trigger the compilation as any other Elixir package.

Or alternatively, gleam compile-package could accept a --mix-exs file, and drop one in disk. Any preferences @lpil? Doing it in gleam compile-package should give you more long term control.

lpil · 2025-04-08T09:41:23Z

Oh! That's a good idea, much easier.

I think I would prefer to not have Gleam generate the mix.exs as it's intended to be agnostic to any other tooling or language.

Papipo · 2025-04-09T11:43:07Z

Implementation wise, I wonder if perhaps we should drop a mix.exs in disk and then just call mix_dep? This way we trigger the compilation as any other Elixir package.

From what I've seen, we could dump the [erlang] options from gleam.toml directly into the mix.exs application function. The alternative would be to support only specific options but this would require more maintenance.

If gleam options there adhere to the expected format (which I hope is the same as Erlang's) we should be good with the former, no?

josevalim · 2025-04-09T13:51:45Z

Yes, exactly.

Papipo · 2025-04-10T15:34:43Z

If the root Mix project has a path dependency on a gleam dependency, where should I put this mix.exs? Since the code isn't copied over to /deps I would be tampering with the local project.

josevalim · 2025-04-10T16:30:59Z

oh, that would be a problem with this approach. So maybe, instead of writing the file to disk, you should just push the project data, like we do on Mix.install? https://github.com/elixir-lang/elixir/blob/main/lib/mix/lib/mix.ex#L930-L934

It does have one issue though, in that we would have no way to pass application properties (because there is no def application) but perhaps we can fix that by changing compile.app to read also look for an :application key in the project config.

josevalim reviewed Feb 13, 2025

View reviewed changes

lib/mix/lib/mix/gleam.ex Outdated Show resolved Hide resolved

josevalim reviewed Feb 13, 2025

View reviewed changes

lib/mix/lib/mix/tasks/deps.compile.ex Outdated Show resolved Hide resolved

Papipo force-pushed the add-gleam-compiler branch from 421dc7c to 0afa895 Compare February 13, 2025 11:55

Papipo mentioned this pull request Feb 13, 2025

Add command to export deps/config gleam-lang/gleam#4240

Closed

Papipo force-pushed the add-gleam-compiler branch 3 times, most recently from b91ae71 to c181ecd Compare February 20, 2025 11:34

josevalim reviewed Feb 20, 2025

View reviewed changes

lib/mix/lib/mix/tasks/deps.compile.ex Outdated Show resolved Hide resolved

josevalim reviewed Feb 20, 2025

View reviewed changes

Papipo force-pushed the add-gleam-compiler branch from a18bf89 to 26c8aff Compare March 17, 2025 12:31

Papipo force-pushed the add-gleam-compiler branch from 26c8aff to 8a7d9fa Compare March 25, 2025 08:59

Papipo added 7 commits March 25, 2025 09:59

Add fixture for gleam support

6de56db

Add Gleam integration with Mix

cb236e7

- Add Mix.Gleam module - Add specific gleam binary version requirement - Rely on `gleam export package-info`

Add support for git dependencies in gleam packages

a08ef98

Exclude gleam tests if gleam is missing

86266fb

Fix deps.compile for gleam

1ab8e9b

- shell_cmd! wasn't handling tuples - Fix documentation

Add support for application_start_module

b09c9dc

This is an optional value within [erlang] in the gleam.toml file. It will be used for the `mod` value when generating a .app file

Handle gleam extra_applications

360db23

Papipo force-pushed the add-gleam-compiler branch from 8a7d9fa to 360db23 Compare March 25, 2025 09:14

Papipo changed the title ~~[DRAFT] Add gleam support to Mix~~ Add gleam support to Mix Mar 26, 2025

Papipo marked this pull request as ready for review March 26, 2025 08:46

josevalim reviewed Mar 26, 2025

View reviewed changes

lib/mix/lib/mix/tasks/deps.compile.ex Outdated Show resolved Hide resolved

Remove redundant quotes

f0c0a29

Do not force app: false in gleam deps

7086167

Papipo force-pushed the add-gleam-compiler branch 2 times, most recently from b0eda69 to b70f8fd Compare April 3, 2025 09:34

Generate app file for gleam deps on compilation

b64f23a

Papipo force-pushed the add-gleam-compiler branch from b70f8fd to b64f23a Compare April 3, 2025 10:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add gleam support to Mix #14262

Add gleam support to Mix #14262

Papipo commented Feb 13, 2025 •

edited

Loading

Papipo commented Feb 17, 2025

josevalim commented Feb 17, 2025

lpil commented Feb 17, 2025

josevalim Feb 20, 2025

Papipo Feb 20, 2025 •

edited

Loading

Papipo commented Mar 26, 2025

josevalim commented Mar 26, 2025

Papipo commented Mar 26, 2025

josevalim commented Mar 26, 2025 •

edited

Loading

Papipo commented Mar 26, 2025

josevalim commented Apr 2, 2025 •

edited

Loading

Papipo commented Apr 2, 2025

josevalim commented Apr 2, 2025

Papipo commented Apr 7, 2025

josevalim commented Apr 7, 2025

lpil commented Apr 7, 2025

josevalim commented Apr 7, 2025

lpil commented Apr 7, 2025 •

edited

Loading

Papipo commented Apr 7, 2025

lpil commented Apr 7, 2025 •

edited

Loading

josevalim commented Apr 7, 2025

lpil commented Apr 7, 2025 •

edited

Loading

josevalim commented Apr 7, 2025 •

edited

Loading

lpil commented Apr 7, 2025 •

edited

Loading

josevalim commented Apr 7, 2025

lpil commented Apr 8, 2025 •

edited

Loading

Papipo commented Apr 9, 2025

josevalim commented Apr 9, 2025

Papipo commented Apr 10, 2025

josevalim commented Apr 10, 2025

Add gleam support to Mix #14262

Are you sure you want to change the base?

Add gleam support to Mix #14262

Conversation

Papipo commented Feb 13, 2025 • edited Loading

Papipo commented Feb 17, 2025

josevalim commented Feb 17, 2025

lpil commented Feb 17, 2025

josevalim Feb 20, 2025

Choose a reason for hiding this comment

Papipo Feb 20, 2025 • edited Loading

Choose a reason for hiding this comment

Papipo commented Mar 26, 2025

What is this?

What's missing?

Instructions

Usage

What needs to be tested?

Thanks

josevalim commented Mar 26, 2025

Papipo commented Mar 26, 2025

josevalim commented Mar 26, 2025 • edited Loading

Papipo commented Mar 26, 2025

josevalim commented Apr 2, 2025 • edited Loading

Papipo commented Apr 2, 2025

josevalim commented Apr 2, 2025

Papipo commented Apr 7, 2025

josevalim commented Apr 7, 2025

lpil commented Apr 7, 2025

josevalim commented Apr 7, 2025

lpil commented Apr 7, 2025 • edited Loading

Papipo commented Apr 7, 2025

lpil commented Apr 7, 2025 • edited Loading

josevalim commented Apr 7, 2025

lpil commented Apr 7, 2025 • edited Loading

josevalim commented Apr 7, 2025 • edited Loading

lpil commented Apr 7, 2025 • edited Loading

josevalim commented Apr 7, 2025

lpil commented Apr 8, 2025 • edited Loading

Papipo commented Apr 9, 2025

josevalim commented Apr 9, 2025

Papipo commented Apr 10, 2025

josevalim commented Apr 10, 2025

Papipo commented Feb 13, 2025 •

edited

Loading

Papipo Feb 20, 2025 •

edited

Loading

josevalim commented Mar 26, 2025 •

edited

Loading

josevalim commented Apr 2, 2025 •

edited

Loading

lpil commented Apr 7, 2025 •

edited

Loading

lpil commented Apr 7, 2025 •

edited

Loading

lpil commented Apr 7, 2025 •

edited

Loading

josevalim commented Apr 7, 2025 •

edited

Loading

lpil commented Apr 7, 2025 •

edited

Loading

lpil commented Apr 8, 2025 •

edited

Loading