|
| 1 | +--- |
| 2 | +id: lux-package-conflicts |
| 3 | +title: How Lux prevents package conflicts |
| 4 | +--- |
| 5 | + |
| 6 | +Have you wondered how Lux prevents many packages with the same name but with the same version from conflicting with each other? |
| 7 | +Or how it prevents packages from being updated when they shouldn't be? This post explains how we do it. |
| 8 | + |
| 9 | +## The Problem |
| 10 | + |
| 11 | +First, let's quickly break down what problems we're trying to solve as a package manager for Lua. Our job is to take |
| 12 | +a rock (the name for a package in Lua land), unpack it, put the Lua files in the correct place and ensure all dependencies are installed. As will be evident soon, |
| 13 | +managing these dependencies is not an easy feat. |
| 14 | + |
| 15 | +In Lux, packages are installed in something called a rock tree. This tree contains a lockfile, which describes which packages |
| 16 | +are installed and how they all relate with one another. The tree also contains a `lux/` directory - it's here that all Lua files are unpacked so that they can be used in |
| 17 | +your scripts. There are three types of rock trees - those that are created for each Lua project (which hold local packages like |
| 18 | +dependencies for a given project), those that are installed for the current user (for things like local binaries or helper packages) |
| 19 | +and those that are system-wide. |
| 20 | + |
| 21 | +A rock tree is structured as follows: |
| 22 | +- `/lux/<lua-version>` - contains lux for a given Lua version |
| 23 | +- `/lux/<lua-version>/<rock>/etc` - documentation and supplementary files for the rock |
| 24 | +- `/lux/<lua-version>/<rock>/lib` - shared libraries (.so files) |
| 25 | +- `/lux/<lua-version>/<rock>/src` - actual code of the rock |
| 26 | +- `/bin` - binary files produced by various packages |
| 27 | + |
| 28 | +`<lua-version>` can be any of `5.1`, `5.2`, `5.3`, `5.4` - simple! Here's a question that might throw you for a loop - what |
| 29 | +should `<rock>` be? Should it just be the name and version of the rock: `/lux/5.1/[email protected]/...`? As we're |
| 30 | +about to find out, it's much, *much* more complicated. |
| 31 | + |
| 32 | +Let's consider the following case: `rock1` is already installed in the user-wide rock tree. `rock1` relies on a hypothetical rock called |
| 33 | +`dependency`, whose latest version is `1.2.0`. |
| 34 | +The rockspec for `rock1` states that it permits `dependency < 1.0.0`, meaning any version of `dependency` prior |
| 35 | +to `1.0.0`. The most recent version of `dependency` prior to `1.0.0` is `0.9.0`, so that's what's installed in the |
| 36 | +rock tree at this point. Got that? |
| 37 | + |
| 38 | +Now, let's say we run `lx install rock2`. `rock2` *also* relies on `dependency`, but it specifies that |
| 39 | +it expects `dependency >= 1.0.0`, which is a completely different constraint to what `rock1` wanted. |
| 40 | +With the aforementioned system, this is no problem: we'll have two directories, `[email protected]/` |
| 41 | +and `[email protected]/` installed separately, easy! |
| 42 | + |
| 43 | +Now, consider a more complicated case: imagine that we install `rock3`, which relies on `dependency <= 1.0.0`, |
| 44 | +and that `dependency` is shared between `rock1` and `rock3`. If you're lost already, let me reiterate. `rock1` |
| 45 | +requires `dependency < 1.0.0`, whereas `rock3` requires `dependency <= 1.0.0`. Imagine that both of these packages |
| 46 | +share the same installation of `[email protected]/` - this is fine because the version `0.9.0` satisfies both |
| 47 | +`< 1.0.0` as well as `<= 1.0.0`. |
| 48 | + |
| 49 | +Why am I blabbing on about this? Imagine an unsuspecting person (you) now runs `lx update`, which updates all |
| 50 | +packages to their latest possible releases. Can you see the problem? `rock3` will want to update `[email protected]` |
| 51 | +to `1.0.0`, because that's the latest available version that satisfies `<= 1.0.0`, |
| 52 | +but `rock1` expects `< 1.0.0`, which **does not include** `1.0.0`. We have a conflict! Two packages want two different |
| 53 | +versions of a *shared dependency*. Yikes. We'll need a better system. |
| 54 | + |
| 55 | +But wait, you thought I was done frying your brain?? Reconsider. Lux also permits doing something special - it allows |
| 56 | +you to **pin** packages. A pinned package's version should never, ever change. Consider *this*: |
| 57 | +- `rock1` requires `< 1.0.0` |
| 58 | +- `rock3` requires `<= 1.0.0` |
| 59 | +- The user runs `lx pin [email protected]`, freezing the dependency. |
| 60 | + |
| 61 | +Upon running `lx update`, we have a three-sided clash: `rock3` wants to upgrade `[email protected]` to `1.0.0`, `rock1` will |
| 62 | +want to keep it the same, *and* because the package is pinned it will also be unable to move. |
| 63 | + |
| 64 | +I'll leave it up to your imagination to figure out even more complicated version conflicts like this. |
| 65 | + |
| 66 | +--- |
| 67 | + |
| 68 | +You might be thinking, "Just don't update the package then, simple." |
| 69 | +That is a solution, and it's one that `luarocks` has used. There's something that isn't talked about enough, though: users |
| 70 | +do incredibly, incredibly silly things. Sometimes they will find an unholy combination of invocations that completely |
| 71 | +messes up their rock tree state. In this "dependencies can be shared" implementation, it's not a matter of *if* something will |
| 72 | +mess up, but *when*. Maybe they somehow *do* update the dependency on accident, rendering `rock1` completely broken. |
| 73 | +Maybe `rock3` **doesn't work** with versions prior to `1.0.0`, and the developer mistakenly said they support everything |
| 74 | +prior to `1.0.0`. There are too many things that can go wrong! |
| 75 | + |
| 76 | +## Hashes, my Beloved |
| 77 | + |
| 78 | +After a few drafts of how we could solve these problems in Lux, |
| 79 | +we somewhat ended up reinventing/reimplementing a similar system to the Nix store. Here's what we did: |
| 80 | +- Instead of storing a rock in a `name@version/` directory, we've instead resorted to a `<hash>-<name>@<version>/` directory. |
| 81 | + The "short hash", as I've called it, is built up of the name, version, pin state and constraint (e.g. `< 1.0.0`). Any difference in any of these variables |
| 82 | + will result in a completely different hash. |
| 83 | +- The package is uniquely identifiable in the lockfile by this hash - this leaves no ambiguity. A pinned package is different |
| 84 | + to an unpinned package; a constrained package is different to an unconstrained one, and so on. |
| 85 | + |
| 86 | +Let's imagine the super complicated situation that I presented at the end of the last section: |
| 87 | +- `rock1` wants `dependency < 1.0.0` |
| 88 | +- `rock3` wants `dependency <= 1.0.0` |
| 89 | +- `dependency` is pinned |
| 90 | + |
| 91 | +With this system, each iteration of `dependency` will have a different hash (the first two have different constraints, and the third has a different pin state), and as such we will have **3 different installations** |
| 92 | +of `dependency` in our rock tree, completely independent of one another. Any of them can be individually updated and no clashes |
| 93 | +can possibly occur between them. |
| 94 | + |
| 95 | +What's unique about this system is that dependencies still can be shared! Two different packages *can* share the same version |
| 96 | +of `dependency`, provided that the hash is exactly the same. In those cases, we can guarantee that absolutely nothing can go wrong, |
| 97 | +no matter how much you try to muck about. |
| 98 | + |
| 99 | +This is almost the exact same solution that Nix uses - all packages and dependencies are prefixed with a hash, except that in Nix's |
| 100 | +case it creates a mega hash from all information about the package, including the hash of its sources. We don't do this, because |
| 101 | +we'd have no way of actually *finding* the rock in the filesystem. If a user went "please delete `rock1`", how would we know |
| 102 | +the name of the directory we should delete? We don't have access to all the sources ahead of time, or at the very least that'd |
| 103 | +be stupidly inefficient. |
| 104 | + |
| 105 | +## Conclusion |
| 106 | + |
| 107 | +Now you know! If there's anything you should learn from today, it's this short rhyme: hashes are a great way of preventing clashes. |
| 108 | +Dependencies are a complicated thing, but with a little cheekiness, we can eliminate almost all their issues entirely. |
0 commit comments