Failing make serve and/or circular imports #10802

dabljues · 2022-02-22T02:14:52Z

Describe the bug

So I try to set everything up as described in docs:

$ git clone https://github.com/pypa/warehouse.git
$ cd warehouse
$ make build
$ sudo sysctl -w vm.max_map_count=262144
$ make serve
make: Circular requirements/docs.txt <- .state/env/pyvenv.cfg dependency dropped.
make: Circular requirements/lint.txt <- .state/env/pyvenv.cfg dependency dropped.
[...]

This goes for a while, I have for example localhost:9001 available. But then the database container doesn't work. Errors look like this:

warehouse-notdatadog-1     | b'warehouse.db.session.start:1|c\n'
warehouse-db-1             | 2022-02-22 02:00:00.257 UTC [69] FATAL:  database "warehouse" does not exist
[...]
warehouse-worker-1         | (Background on this error at: https://sqlalche.me/e/14/e3q8)
warehouse-worker-1         | 
warehouse-worker-1         | During handling of the above exception, another exception occurred:
warehouse-worker-1         | 
warehouse-worker-1         | Traceback (most recent call last):
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
warehouse-worker-1         |     R = retval = fun(*args, **kwargs)
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/sentry_sdk/integrations/celery.py", line 200, in _inner
warehouse-worker-1         |     reraise(*exc_info)
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/sentry_sdk/_compat.py", line 54, in reraise
warehouse-worker-1         |     raise value
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/sentry_sdk/integrations/celery.py", line 195, in _inner
warehouse-worker-1         |     return f(*args, **kwargs)
warehouse-worker-1         |   File "/opt/warehouse/src/warehouse/tasks.py", line 88, in __call__
warehouse-worker-1         |     return super().__call__(*(self.get_request(),) + args, **kwargs)
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/celery/app/trace.py", line 734, in __protected_call__
warehouse-worker-1         |     return self.run(*args, **kwargs)
warehouse-worker-1         |   File "/opt/warehouse/src/warehouse/tasks.py", line 71, in run
warehouse-worker-1         |     result = original_run(*args, **kwargs)
warehouse-worker-1         |   File "/opt/warehouse/src/warehouse/manage/tasks.py", line 21, in update_role_invitation_status
warehouse-worker-1         |     request.db.query(RoleInvitation)
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/pyramid/decorator.py", line 40, in __get__
warehouse-worker-1         |     val = self.wrapped(inst)
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/pyramid/util.py", line 123, in <lambda>
warehouse-worker-1         |     wrapped = lambda this: callable(this)
warehouse-worker-1         |   File "/opt/warehouse/src/warehouse/db.py", line 176, in _create_session
warehouse-worker-1         |     raise DatabaseNotAvailableError()
warehouse-worker-1         | warehouse.db.DatabaseNotAvailableError

I tried various combinations of running make initdb from a second terminal. It spits out some information, ends up in errors and hangs. It looks like this:

bcrypt$$2a$16$xq9g/NAFoiOtFkGpk/kgjOiPDPs1/pMG0GHeDHwarehouse=# WHSOd9IY7Fah.yC	2018-11-20 15:44:36.999614	f	ruebot	Nick Ruest	t	2018-11-20 15:44:36.999614	a	2018-11-20 15:44:36.psql:<stdin>:2204980: 999614	03917fd1-68d5-42ba-b2a0-4013b37dad64	\N
bcInvalid command \N. Try \? for help.
rypt$$2a$16$xq9g/NAFoiOtFkGpk/kgjOiPDPs1/pMG0GHeDHWHSOd9warehouse=# IY7Fah.yC	2018-11-20 15:44:36.999614	f	CuberpresKn	CuberpresKn	f	2018-11-20 15:44:36.999614	b	2018-11-20 15:44:36.999614	3e34psql:<stdin>:2204981: 9447-0b91-496b-bda9-6910608f807f	\N
bcrypt$$2a$16$xq9g/NAFoiOtFkGpk/kgjOiPDPs1/pMG0GHeDHWHInvalid command \N. Try \? for help.
SOd9IY7Fah.yC	2018-11-20 15:44:36.999614	f	yuekyang	warehouse=# Jonathan Yang	2018-11-20 15:44:36.999614	d	2018-11-20 15:44:3psql:<stdin>:2204982: 6.999614	be413e9f-f84e-4c93-aa50-77a23508f424	\N
bcrypt$$2Invalid command \N. Try \? for help.
a$16$xq9g/NAFoiOtFkGpk/kgjOiPDPs1/pMG0GHeDHWHSOwarehouse=# d9IY7Fah.yC	2018-11-20 15:44:36.999614

Then, the terminal with make serve starts to output something like:

warehouse-db-1             | 2022-02-22 02:22:41.649 UTC [103] ERROR:  syntax error at or near "0.0" at character 1
warehouse-db-1             | 2022-02-22 02:22:41.649 UTC [103] STATEMENT:  0.0.2	Thibault [email protected]			https://github.com/tducret/ingdirect-python	MIT	Package pour consulter ses comptes bancaires ING Direct	Package pour consulter ses comptes bancaires ING Direct	api	Posix;
warehouse-db-1             | 2022-02-22 02:22:41.649 UTC [103] ERROR:  syntax error at or near "0.0" at character 1
warehouse-db-1             | 2022-02-22 02:22:41.649 UTC [103] STATEMENT:  0.0.3	Thibault [email protected]			https://github.com/tducret/ingdirect-python	MIT	Package pour consulter ses comptes bancaires ING Direct	Package pour consulter ses comptes bancaires ING Direct	api	Posix;
warehouse-db-1             | 2022-02-22 02:22:41.649 UTC [103] ERROR:  syntax error at or near "0.0" at character 1
warehouse-db-1             | 2022-02-22 02:22:41.649 UTC [103] STATEMENT:  0.0.4	Thibault [email protected]			https://github.com/tducret/ingdirect-python	MIT	Package pour consulter ses comptes bancaires ING Direct	Package pour consulter ses comptes bancaires ING Direct	api	Posix;

ends up in:

warehouse-notdatadog-1     | b'warehouse.db.session.finished:1|c\n'
warehouse-worker-1         | LINE 2: FROM admin_flags 
warehouse-worker-1         |              ^
warehouse-worker-1         | 
warehouse-worker-1         | 
warehouse-worker-1         | The above exception was the direct cause of the following exception:
warehouse-worker-1         | 
warehouse-worker-1         | Traceback (most recent call last):
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
warehouse-worker-1         |     R = retval = fun(*args, **kwargs)
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/sentry_sdk/integrations/celery.py", line 200, in _inner
warehouse-worker-1         |     reraise(*exc_info)
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/sentry_sdk/_compat.py", line 54, in reraise
warehouse-worker-1         |     raise value
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/sentry_sdk/integrations/celery.py", line 195, in _inner
warehouse-worker-1         |     return f(*args, **kwargs)
warehouse-worker-1         |   File "/opt/warehouse/src/warehouse/tasks.py", line 88, in __call__
warehouse-worker-1         |     return super().__call__(*(self.get_request(),) + args, **kwargs)
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/celery/app/trace.py", line 734, in __protected_call__
warehouse-worker-1         |     return self.run(*args, **kwargs)
warehouse-worker-1         |   File "/opt/warehouse/src/warehouse/tasks.py", line 71, in run
warehouse-worker-1         |     result = original_run(*args, **kwargs)
warehouse-worker-1         |   File "/opt/warehouse/src/warehouse/packaging/tasks.py", line 111, in update_description_html
warehouse-worker-1         |     request.db.query(Description)
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/pyramid/decorator.py", line 40, in __get__
warehouse-worker-1         |     val = self.wrapped(inst)
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/pyramid/util.py", line 123, in <lambda>
warehouse-worker-1         |     wrapped = lambda this: callable(this)
warehouse-worker-1         |   File "/opt/warehouse/src/warehouse/db.py", line 213, in _create_session
warehouse-worker-1         |     flag = session.query(AdminFlag).get(AdminFlagValue.READ_ONLY.value)
warehouse-worker-1         |   File "<string>", line 2, in get
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/sqlalchemy/util/deprecations.py", line 401, in warned
warehouse-worker-1         |     return fn(*args, **kwargs)
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 943, in get
warehouse-worker-1         |     return self._get_impl(ident, loading.load_on_pk_identity)
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 947, in _get_impl
warehouse-worker-1         |     return self.session._get_impl(
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2892, in _get_impl
warehouse-worker-1         |     return db_load_fn(
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/sqlalchemy/orm/loading.py", line 530, in load_on_pk_identity
warehouse-worker-1         |     session.execute(
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1692, in execute
warehouse-worker-1         |     result = conn._execute_20(statement, params or {}, execution_options)
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1614, in _execute_20
warehouse-worker-1         |     return meth(self, args_10style, kwargs_10style, execution_options)
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
warehouse-worker-1         |     return connection._execute_clauseelement(
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1481, in _execute_clauseelement
warehouse-worker-1         |     ret = self._execute_context(
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1845, in _execute_context
warehouse-worker-1         |     self._handle_dbapi_exception(
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2026, in _handle_dbapi_exception
warehouse-worker-1         |     util.raise_(
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
warehouse-worker-1         |     raise exception
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
warehouse-worker-1         |     self.dialect.do_execute(
warehouse-worker-1         |   File "/opt/warehouse/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
warehouse-worker-1         |     cursor.execute(statement, parameters)
warehouse-worker-1         | sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "admin_flags" does not exist
warehouse-worker-1         | LINE 2: FROM admin_flags 
warehouse-worker-1         |              ^
warehouse-worker-1         | 
warehouse-worker-1         | [SQL: SELECT admin_flags.id AS admin_flags_id, admin_flags.description AS admin_flags_description, admin_flags.enabled AS admin_flags_enabled, admin_flags.notify AS admin_flags_notify 
warehouse-worker-1         | FROM admin_flags 
warehouse-worker-1         | WHERE admin_flags.id = %(pk_1)s]
warehouse-worker-1         | [parameters: {'pk_1': 'read-only'}]
warehouse-worker-1         | (Background on this error at: https://sqlalche.me/e/14/f405)

at the end. And it refreshes periodically, same thing is being logged to the terminal.

Here's how it looks on the localhost:

Expected behavior

make serve should not fail.

To Reproduce

$ git clone https://github.com/pypa/warehouse.git
$ cd warehouse
$ make build
$ sudo sysctl -w vm.max_map_count=262144
$ make serve

My Platform

Arch Linux, 5.16.10 kernel
Python 3.8.9 installed via pyenv
postgresql-libs installed (so pg_config command is available).
docker, docker-compose installed and configured (systemctl enabled etc)
- no proxy or anything

Additional context

I am running this inside a VM, but I get the same results when running from a host machine (also Arch, same configuration). I only used VMs because I wanted a quick way to delete one and start fresh (got snapshots).

Another thing to consider: I also fixed the circular imports (issue described here). Fix comes from another repo - link (I checked out from it). Mainly the fix-sqlalchemy-deps make command. This command just ran some pip-compile command:

fix-sqlalchemy-deps:
	# see fix-google-deps

	pip-compile --allow-unsafe --generate-hashes --output-file=requirements/main.txt \
	   --upgrade-package SQLAlchemy \
	   requirements/main.in

So then it was:

$ pip install pip-compile
$ make build
$ sudo sysctl -w vm.max_map_count=262144
$ make serve

But it ends up the same.

However, when I run all of this ((and I mean all of this) in a separate Docker environment - it worked. So a simple ubuntu:latest container, couple packages installed, like this:

FROM ubuntu:latest

RUN apt update
RUN apt install -yq build-essential git make python3 python3-pip docker-compose python3-venv \
                    curl libcurl4-openssl-dev libssl-dev libpq-dev
RUN pip3 install pip-tools babel

RUN git clone https://github.com/abitrolly/warehouse.git -b fix-devenv
WORKDIR /warehouse

Then running make fix-sqlalchemy-deps, make build and make serve - inside the container (also binding the docker sock and copying /warehouse dir from container to host - so Docker sees it) worked.

The text was updated successfully, but these errors were encountered:

di · 2022-02-22T02:49:09Z

Sorry you're having trouble!

The circular dependency is #10447, although I don't think it's actually causing issues getting started here, just warning messages.

The issue with make initdb is #10772. Disabling Compose V2 will resolve this.

I think given that, this issue is a duplicate of those two issues, but I'll let you decide.

dabljues · 2022-02-22T02:57:34Z

Hi,

Thanks for your reply! I will try to scrap some snippets out of both issues, put them together and check if this solves the issue. Let me get back to you when I check this. I would close the issue now, but I don't know if this would fix the problem.

di · 2022-02-22T03:03:37Z

Again, I think just disabling Compose V2 in Docker will resolve the issues you're having here.

dabljues · 2022-02-22T03:54:41Z

Again, I think just disabling Compose V2 in Docker will resolve the issues you're having here.

Oh, I am sorry. That's what sleep deprivation does to a person, it's like I didn't read what you wrote at all :)

And, holy smokes, this worked. So many hours spent on this thing - just downgraded the docker-compose version and everything works! I still had to make initdb on the 2nd terminal (DatabaseNotAvailableError at first with make serve), but now the db initialization went without any errors, 1st terminal refreshed and I can now search for packages on localhost!

Thanks man!

di · 2022-02-22T04:03:19Z

Glad to see you got it working!

I still had to make initdb on the 2nd terminal

Yep, this is documented here: https://warehouse.pypa.io/development/getting-started.html#running-the-warehouse-container-and-services

di · 2022-02-22T04:20:02Z

(I added a note about #10772 to the docs in ec04074)

dabljues · 2022-02-22T04:59:41Z

Yep, this is documented here: https://warehouse.pypa.io/development/getting-started.html#running-the-warehouse-container-and-services

I know. I just thought that make serve would not throw the database error now, but it doesn't matter. It works!

di · 2022-02-22T13:18:58Z

A possible improvement we could make here: we could catch this exception, check if we're in a development environment, and output a better error message if so.

dabljues added the bug 🐛 label Feb 22, 2022

dabljues closed this as completed Feb 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failing make serve and/or circular imports #10802

Failing make serve and/or circular imports #10802

dabljues commented Feb 22, 2022 •

edited

Loading

di commented Feb 22, 2022

dabljues commented Feb 22, 2022

di commented Feb 22, 2022 •

edited

Loading

dabljues commented Feb 22, 2022 •

edited

Loading

di commented Feb 22, 2022

di commented Feb 22, 2022

dabljues commented Feb 22, 2022

di commented Feb 22, 2022

Failing make serve and/or circular imports #10802

Failing make serve and/or circular imports #10802

Comments

dabljues commented Feb 22, 2022 • edited Loading

di commented Feb 22, 2022

dabljues commented Feb 22, 2022

di commented Feb 22, 2022 • edited Loading

dabljues commented Feb 22, 2022 • edited Loading

di commented Feb 22, 2022

di commented Feb 22, 2022

dabljues commented Feb 22, 2022

di commented Feb 22, 2022

dabljues commented Feb 22, 2022 •

edited

Loading

di commented Feb 22, 2022 •

edited

Loading

dabljues commented Feb 22, 2022 •

edited

Loading