Load configuration at runtime instead of build time

[zacps]

This will make it possible to publish self contained docker images which can then be used in simple docker/kubernetes deployments.

This is a breaking change, in particular private environment variables must now have the VITE_ prefix, and users will need to make sure environment variables are available at runtime.

Fixes #425

[anly2]

Instead of renaming the variables with VITE_ wouldn't it be better to improve the usages to follow https://kit.svelte.dev/docs/modules#$env-dynamic-private

There will be import adjustments, but it will be more intuitive.
(And no variable renames expected)

If secrets are made dynamic, then the docker images will stay "env agnostic" and can be built once.
(I know feature flags are also different across environments, but those are not really a problem like secrets)

[zacps]

By no means an expert on Vite/Svelte, but here is my understanding:

The problem is that Vite doesn't load environment variables unless they're prefixed with VITE_. This means they're not exposed (at build time), and even for $env/dynamic svelte requires them to be defined.

It does seem like it should be possible to manually load variables at build time to get around this, but I couldn't find a place to load them early enough that they got injected into $env/dynamic. It should be possible to do outside the process, like in the npm build script, but this potentially breaks editor linting.

[anly2]

I am no expert at all either, but both the documentation and this tutorial don't have anything particularly special: https://learn.svelte.dev/tutorial/env-dynamic-private

So wouldn't expect renames, but changing the usage places and imports.

[zacps]

You are correct, I must have had something else miss-configured when I thought the prefix was required.

Fixed!

[anly2]

Admittedly, I couldn't quite get it to work myself either, so I feel your pain.

(I tried updating just the mongodb vars, and with the dynamic vars it builds and runs locally just fine.
But when I build a docker image and provide the .env.local outside, I fail. I suspect it is something to do with how I mount the env file, but haven't gotten to the bottom of it)

[anly2]

TL;DR: You only need to fix the MONGODB_URL

---

As a follow up to my previous comment, I actually made it work with relatively few changes.

The only problem really was the bloody MONGODB_URL check that throws!
I defaulted that in the .env file and everything else was fine...

Only annoying thing was (which could just be me doing things wrong) that I expected the $env/static/private and the $env/dynamic/private to overlap but they don't seem to?!
I expect if I provide it in the .env.local file and use /*dynamic*/ env.MONGODB_URL that I would get the value. But I didn't.
So I worked around this with these ugly fallbacks: const mongoUrl = env.MONGODB_URL || MONGODB_URL; (which, again, I expected to already be the case)

[zacps]

Just to clarify, did you need to change that in this PR? Or does it work for you as is?

[anly2]

I had to do the following changes:

In .env give MONGO_URL an actual default value:

MONGODB_URL=mongodb://localhost:27017 # your mongodb URL here

And then I updated the database.ts file because I was actually providing the url as an actual env var (not as a line in a dotenv file):

import { MONGODB_URL, MONGODB_DB_NAME, MONGODB_DIRECT_CONNECTION } from "$env/static/private";
import { env } from "$env/dynamic/private";

// ...

const mongoUrl = env.MONGODB_URL || MONGODB_URL; // I expected vars to work like this already!
if (!mongoUrl)  // throw ...

const client = new MongoClient(mongoUrl, {
	directConnection: (env.MONGODB_DIRECT_CONNECTION ?? MONGODB_DIRECT_CONNECTION) === "true",
});

// ...

const db = client.db(
	(env.MONGODB_DB_NAME ?? MONGODB_DB_NAME) + (import.meta.env.MODE === "test" ? "-test" : "")
);

// ...

First part made it build ok. (I really don't understand why the code is actually executed at build time)
Second part allowed me to override to the correct value at runtime.

[zacps]

import { MONGODB_URL, MONGODB_DB_NAME, MONGODB_DIRECT_CONNECTION } from "$env/static/private";

This line isn't in this PR.

const mongoUrl = env.MONGODB_URL || MONGODB_URL

This is unnecessary with the $env/dynamic imports.

[anly2]

Coming back to this with better understanding:

The code gets executed at build time as per the Code Analysis discussed here:
sveltejs/kit#7716

The database.ts file has side effects that connect to the database, or throw, regardless of mode.
This can be improved by wrapping the actual connect() code with if (building) as per the linked discussion.

Not throwing or attempting to connect at build time makes it build fine.

But whether the correct environment values are used is a different matter.

The $env module is not explained in great detail, but to summarise:

• $env/static/* are values which were loaded at build time
  • Example: MONGODB_URL=mongodb://localhost:27017 in .env
• $env/dynamic/* are ONLY the values supplied as env vars to the runtime
  • Example: MONGODB_URL=mongodb://localhost:27017 node /app/build/index.js
• There is no automatic "fallback" from "$env/dynamic" to "$env/static" values (not intuitive at all imo)
  • In other words, we would need to write code like this:
    const mongoUrl = /* "$env/dynamic/private" */ env.MONGODB_URL ?? /* "$env/static/private" */ MONGODB_URL
• Any dotenv files at runtime are NOT loaded automatically.
  If you only have the file present at runtime, like .env.local
  Or if you add a call to load an extra file at runtime, say .env.models: dotenv.config({ path: "./.env.models" });
  Then this will NOT update any of the $env modules.
  Neither "$env/dynamic" nor "$env/static" will have the values from the file(s).
  The call to dotenv.config ONLY added the values in process.env

All of this makes it hard to provide env vars at runtime.

---

To resolve all of this, I replaced all of the env var usages with $env/dynamic/* - like you have.
But I have also added a bit of code in the very beginning that populates $env/dynamic with values, if empty.

try {
  const { env: dynamicEnv } = await import("$env/dynamic/private");
  const staticEnv = await import("$env/static/private");
  // Iterate these objects as PUBLIC_ vars are split out already
  Object.keys(staticEnv ?? {}).forEach(k => dynamicEnv[k] ??= process.env[k] ?? staticEnv[k])
  // ^^^ SAME for public vars
} catch () {}

---

Separately, I didn't want to update the usages EVERYWHERE in code, so I used destructuring right after the import statement:

import { env } from "$env/dynamic/private";
const {
	COOKIE_NAME,
	EXPOSE_API,
	MESSAGES_BEFORE_LOGIN
} = env;

[zacps]

Moved the database check to runtime only.

The one outstanding issue I've encountered with this PR is that it doesn't currently make it possible to use a dynamic APP_BASE; I think this should be possible but probably requires a sveltekit bump (at least) to get access to the reroute hook.

[zacps]

I've abandoned my attempt to make APP_BASE relative, it is exceptionally difficult to make work.

Happy to provide my notes if anyone else wants to attempt but for now this PR just leaves APP_BASE static.

[nsarrazin]

Hey! Will close this PR as we handled it with #1092. Sorry for the double work, we needed to figure it out for our infra on HuggingChat. 😅 Appreciate the work on this