Skip to content

Commit 0030636

Browse files
authored
Merge branch 'develop' into etienne/sec-197-use-nonewpriviliges-for-postgres
2 parents e0708b6 + ef851d1 commit 0030636

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

72 files changed

+358
-389
lines changed

.github/CODEOWNERS

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
* @supabase/backend @supabase/postgres
2-
migrations/ @supabase/cli @supabase/backend
2+
migrations/ @supabase/dev-workflows @supabase/postgres @supabase/backend
33
docker/orioledb @supabase/postgres @supabase/backend
44
common.vars.pkr.hcl @supabase/postgres @supabase/backend

.github/workflows/qemu-image-build.yml

+2-2
Original file line numberDiff line numberDiff line change
@@ -23,10 +23,10 @@ jobs:
2323

2424
- uses: DeterminateSystems/nix-installer-action@main
2525

26-
- name: Set PostgreSQL versions - only builds pg15 atm
26+
- name: Set PostgreSQL versions - only builds pg17 atm
2727
id: set-versions
2828
run: |
29-
VERSIONS=$(nix run nixpkgs#yq -- '.postgres_major[0]' ansible/vars.yml | nix run nixpkgs#jq -- -R -s -c 'split("\n")[:-1]')
29+
VERSIONS=$(nix run nixpkgs#yq -- '.postgres_major[1]' ansible/vars.yml | nix run nixpkgs#jq -- -R -s -c 'split("\n")[:-1]')
3030
echo "postgres_versions=$VERSIONS" >> $GITHUB_OUTPUT
3131
3232
build:

README.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ Unmodified Postgres with some useful plugins. Our goal with this repo is not to
8181

8282
| Goodie | Version | Description |
8383
| ------------- | :-------------: | ------------- |
84-
| [PgBouncer](https://www.pgbouncer.org/) | [1.16.1](http://www.pgbouncer.org/changelog.html#pgbouncer-116x) | Set up Connection Pooling. |
84+
| [PgBouncer](https://www.pgbouncer.org/) | [1.19.0](http://www.pgbouncer.org/changelog.html#pgbouncer-119x) | Set up Connection Pooling. |
8585
| [PostgREST](https://postgrest.org/en/stable/) | [v12.2.3](https://github.com/PostgREST/postgrest/releases/tag/v12.2.3) | Instantly transform your database into an RESTful API. |
8686
| [WAL-G](https://github.com/wal-g/wal-g#wal-g) | [v2.0.1](https://github.com/wal-g/wal-g/releases/tag/v2.0.1) | Tool for physical database backup and recovery. | -->
8787

@@ -126,4 +126,4 @@ TODO: find way to automate this
126126

127127
We are building the features of Firebase using enterprise-grade, open source products. We support existing communities wherever possible, and if the products don’t exist we build them and open source them ourselves.
128128

129-
[![New Sponsor](https://user-images.githubusercontent.com/10214025/90518111-e74bbb00-e198-11ea-8f88-c9e3c1aa4b5b.png)](https://github.com/sponsors/supabase)
129+
[![New Sponsor](https://user-images.githubusercontent.com/10214025/90518111-e74bbb00-e198-11ea-8f88-c9e3c1aa4b5b.png)](https://github.com/sponsors/supabase)

amazon-arm64-nix.pkr.hcl

-5
Original file line numberDiff line numberDiff line change
@@ -228,11 +228,6 @@ build {
228228
destination = "/tmp"
229229
}
230230

231-
provisioner "file" {
232-
source = "ebssurrogate/files/unit-tests"
233-
destination = "/tmp"
234-
}
235-
236231
# Copy ansible playbook
237232
provisioner "shell" {
238233
inline = ["mkdir /tmp/ansible-playbook"]

ansible/files/gotrue-optimizations.service.j2

+1
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ Description=GoTrue (Auth) optimizations
55
Type=oneshot
66
# we don't want failures from this command to cause PG startup to fail
77
ExecStart=/bin/bash -c "/opt/supabase-admin-api optimize auth --destination-config-file-path /etc/gotrue/gotrue.generated.env ; exit 0"
8+
ExecStartPost=/bin/bash -c "cp -a /etc/gotrue/gotrue.generated.env /etc/auth.d/20_generated.env ; exit 0"
89
User=postgrest
910

1011
[Install]

ansible/files/gotrue.service.j2

+1-1
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ Description=Gotrue
44
[Service]
55
Type=simple
66
WorkingDirectory=/opt/gotrue
7-
ExecStart=/opt/gotrue/gotrue
7+
ExecStart=/opt/gotrue/gotrue --config-dir /etc/auth.d
88
User=gotrue
99
Restart=always
1010
RestartSec=3

ansible/tasks/setup-gotrue.yml

+7
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,13 @@
3030
owner: gotrue
3131
mode: 0775
3232

33+
- name: gotrue - create /etc/auth.d
34+
file:
35+
path: /etc/auth.d
36+
state: directory
37+
owner: gotrue
38+
mode: 0755
39+
3340
- name: gotrue - unpack archive in /opt/gotrue
3441
unarchive:
3542
remote_src: yes

ansible/tasks/setup-wal-g.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@
4444

4545
- name: Create symlink to make wal-g-v2 the default wal-g
4646
ansible.builtin.file:
47-
src: /usr/local/bin/wal-g-v2
47+
src: /home/wal-g/.nix-profile/bin/wal-g-2
4848
dest: /usr/local/bin/wal-g
4949
state: link
5050
force: yes

ansible/tasks/test-image.yml

+3-64
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,3 @@
1-
- name: install pg_prove
2-
apt:
3-
pkg:
4-
- libtap-parser-sourcehandler-pgtap-perl
5-
when: debpkg_mode
6-
71
# - name: Temporarily disable PG Sodium references in config
82
# become: yes
93
# become_user: postgres
@@ -16,9 +10,9 @@
1610
become_user: postgres
1711
shell:
1812
cmd: >
19-
sed -i.bak
20-
-e 's/\(shared_preload_libraries = '\''.*\)pgsodium,\(.*'\''\)/\1\2/'
21-
-e 's/\(shared_preload_libraries = '\''.*\)supabase_vault,\(.*'\''\)/\1\2/'
13+
sed -i.bak
14+
-e 's/\(shared_preload_libraries = '\''.*\)pgsodium,\(.*'\''\)/\1\2/'
15+
-e 's/\(shared_preload_libraries = '\''.*\)supabase_vault,\(.*'\''\)/\1\2/'
2216
-e 's/\(shared_preload_libraries = '\''.*\), *supabase_vault'\''/\1'\''/'
2317
-e 's/pgsodium.getkey_script=/#pgsodium.getkey_script=/'
2418
/etc/postgresql/postgresql.conf
@@ -74,53 +68,6 @@
7468
LOCALE_ARCHIVE: /usr/lib/locale/locale-archive
7569
when: stage2_nix
7670

77-
78-
- name: Check psql_version and modify migrations if oriole-xx
79-
block:
80-
- name: Check if psql_version is psql_orioledb-xx
81-
set_fact:
82-
is_psql_oriole: "{{ psql_version in ['psql_orioledb-16', 'psql_orioledb-17'] }}"
83-
84-
- name: Remove specified extensions from SQL file
85-
ansible.builtin.command:
86-
cmd: >
87-
sed -i '/\\ir.*\(timescaledb\|postgis\|pgrouting\|plv8\).*\.sql/d' /tmp/migrations/tests/extensions/test.sql
88-
when: is_psql_oriole
89-
become: yes
90-
91-
- name: Remove specified extension files from extensions directory
92-
ansible.builtin.find:
93-
paths: /tmp/migrations/tests/extensions
94-
patterns:
95-
- '*timescaledb*.sql'
96-
- '*plv8*.sql'
97-
- '*postgis*.sql'
98-
- '*pgrouting*.sql'
99-
register: files_to_remove
100-
when: is_psql_oriole
101-
102-
- name: Delete matched extension files
103-
ansible.builtin.file:
104-
path: "{{ item.path }}"
105-
state: absent
106-
loop: "{{ files_to_remove.files }}"
107-
when: is_psql_oriole
108-
become: yes
109-
110-
- name: Run Unit tests (with filename unit-test-*) on Postgres Database
111-
shell: /usr/bin/pg_prove -U postgres -h localhost -d postgres -v /tmp/unit-tests/unit-test-*.sql
112-
register: retval
113-
failed_when: retval.rc != 0
114-
when: debpkg_mode or stage2_nix
115-
116-
- name: Run migrations tests
117-
shell: /usr/bin/pg_prove -U supabase_admin -h localhost -d postgres -v tests/test.sql
118-
register: retval
119-
failed_when: retval.rc != 0
120-
when: debpkg_mode or stage2_nix
121-
args:
122-
chdir: /tmp/migrations
123-
12471
- name: Re-enable PG Sodium references in config
12572
become: yes
12673
become_user: postgres
@@ -132,14 +79,6 @@
13279
shell: /usr/lib/postgresql/bin/psql --no-password --no-psqlrc -d postgres -h localhost -U supabase_admin -c 'SELECT pg_stat_statements_reset(); SELECT pg_stat_reset();'
13380
when: debpkg_mode or stage2_nix
13481

135-
- name: remove pg_prove
136-
apt:
137-
pkg:
138-
- libtap-parser-sourcehandler-pgtap-perl
139-
state: absent
140-
autoremove: yes
141-
when: debpkg_mode
142-
14382
- name: Stop Postgres Database
14483
become: yes
14584
become_user: postgres

ansible/vars.yml

+3-3
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,8 @@ postgres_major:
88

99
# Full version strings for each major version
1010
postgres_release:
11-
postgresorioledb-17: "17.0.1.052-orioledb.etcro"
12-
postgres15: "15.8.1.059-rc.etcro"
11+
postgresorioledb-17: "17.0.1.054-orioledb"
12+
postgres15: "15.8.1.061"
1313

1414
# Non Postgres Extensions
1515
pgbouncer_release: "1.19.0"
@@ -51,7 +51,7 @@ postgres_exporter_release_checksum:
5151
arm64: sha256:29ba62d538b92d39952afe12ee2e1f4401250d678ff4b354ff2752f4321c87a0
5252
amd64: sha256:cb89fc5bf4485fb554e0d640d9684fae143a4b2d5fa443009bd29c59f9129e84
5353

54-
adminapi_release: 0.75.0
54+
adminapi_release: 0.76.0
5555
adminmgr_release: 0.24.1
5656

5757
vector_x86_deb: "https://packages.timber.io/vector/0.22.3/vector_0.22.3-1_amd64.deb"

ebssurrogate/files/unit-tests/unit-test-01.sql

-58
This file was deleted.

ebssurrogate/scripts/surrogate-bootstrap-nix.sh

-3
Original file line numberDiff line numberDiff line change
@@ -180,9 +180,6 @@ function setup_chroot_environment {
180180
# Copy migrations
181181
cp -r /tmp/migrations /mnt/tmp/
182182

183-
# Copy unit tests
184-
cp -r /tmp/unit-tests /mnt/tmp/
185-
186183
# Copy the bootstrap script into place and execute inside chroot
187184
cp /tmp/chroot-bootstrap-nix.sh /mnt/tmp/chroot-bootstrap-nix.sh
188185
chroot /mnt /tmp/chroot-bootstrap-nix.sh

flake.nix

+5-1
Original file line numberDiff line numberDiff line change
@@ -822,12 +822,15 @@
822822
exit 1
823823
fi
824824
825+
echo "Running migrations tests"
826+
pg_prove -p 5435 -U supabase_admin -h localhost -d postgres -v ${./migrations/tests}/test.sql
827+
825828
# Copy logs to output
826829
for logfile in $(find /tmp -name postgresql.log -type f); do
827830
cp "$logfile" $out/postgresql.log
828831
done
829832
exit 0
830-
'';
833+
'';
831834
in
832835
rec {
833836
# The list of all packages that can be built with 'nix build'. The list
@@ -865,6 +868,7 @@
865868
local-infra-bootstrap = mkApp "local-infra-bootstrap" "local-infra-bootstrap";
866869
dbmate-tool = mkApp "dbmate-tool" "dbmate-tool";
867870
update-readme = mkApp "update-readme" "update-readme";
871+
show-commands = mkApp "show-commands" "show-commands";
868872
};
869873

870874
# 'devShells.default' lists the set of packages that are included in the

migrations/README.md

+2-5
Original file line numberDiff line numberDiff line change
@@ -42,8 +42,6 @@ nix run github:supabase/postgres/mybranch#dbmate-tool -- --version 15
4242

4343
aiming to provide a single source of truth for migrations on the platform that can be depended upon by those components. For more information on goals see [the RFC](https://www.notion.so/supabase/Centralize-SQL-Migrations-cd3847ae027d4f2bba9defb2cc82f69a)
4444

45-
46-
4745
## How it was Created
4846

4947
Migrations were pulled (in order) from:
@@ -53,9 +51,8 @@ Migrations were pulled (in order) from:
5351

5452
For compatibility with hosted projects, we include [migrate.sh](migrate.sh) that executes migrations in the same order as ami build:
5553

56-
1. Run all `db/init-scripts` with `postgres` superuser role.
57-
2. Run all `db/migrations` with `supabase_admin` superuser role.
58-
3. Finalize role passwords with `/etc/postgresql.schema.sql` if present.
54+
1. Run all `db/migrations` with `supabase_admin` superuser role.
55+
2. Finalize role passwords with `/etc/postgresql.schema.sql` if present.
5956

6057
Additionally, [supabase/postgres](https://github.com/supabase/postgres/blob/develop/ansible/playbook-docker.yml#L9) image contains several migration scripts to configure default extensions. These are run first by docker entrypoint and included in ami by ansible.
6158

migrations/db/init-scripts/00000000000000-initial-schema.sql

-57
This file was deleted.

migrations/db/migrate.sh

-7
Original file line numberDiff line numberDiff line change
@@ -38,11 +38,6 @@ begin
3838
end if;
3939
end \$\$
4040
EOSQL
41-
# run init scripts as postgres user
42-
for sql in "$db"/init-scripts/*.sql; do
43-
echo "$0: running $sql"
44-
psql -v ON_ERROR_STOP=1 --no-password --no-psqlrc -U postgres -f "$sql"
45-
done
4641
psql -v ON_ERROR_STOP=1 --no-password --no-psqlrc -U postgres -c "ALTER USER supabase_admin WITH PASSWORD '$PGPASSWORD'"
4742
# run migrations as super user - postgres user demoted in post-setup
4843
for sql in "$db"/migrations/*.sql; do
@@ -54,8 +49,6 @@ else
5449
create role postgres superuser login password '$PGPASSWORD';
5550
alter database postgres owner to postgres;
5651
EOSQL
57-
# run init scripts as postgres user
58-
DBMATE_MIGRATIONS_DIR="$db/init-scripts" DATABASE_URL="postgres://postgres:$connect" dbmate --no-dump-schema migrate
5952
psql -v ON_ERROR_STOP=1 --no-password --no-psqlrc -U postgres -c "ALTER USER supabase_admin WITH PASSWORD '$PGPASSWORD'"
6053
# run migrations as super user - postgres user demoted in post-setup
6154
DBMATE_MIGRATIONS_DIR="$db/migrations" DATABASE_URL="postgres://supabase_admin:$connect" dbmate --no-dump-schema migrate

0 commit comments

Comments
 (0)