Skip to content

Async Sidecar + maintenance pytest-simcore + docker build-kit/X #1350

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 283 commits into from
Mar 28, 2020
Merged

Async Sidecar + maintenance pytest-simcore + docker build-kit/X #1350

Changes from 1 commit
Commits
Show all changes
283 commits
Select commit Hold shift + click to select a range
2409506
typo
sanderegg Mar 18, 2020
92ee5aa
moving fixtures to package pytest-fixtures
sanderegg Mar 13, 2020
e00740b
moved helpers in common package
sanderegg Mar 13, 2020
48bcd38
back to normal
sanderegg Mar 13, 2020
8e85402
added environs fixture
sanderegg Mar 13, 2020
aa5a1e5
postgres service fixture independent of webserver
sanderegg Mar 13, 2020
e0f1791
celery service independent of webserver
sanderegg Mar 13, 2020
d6e600e
celery config independent of webserver
sanderegg Mar 13, 2020
b2f897b
rabbit fixture independent of webserver
sanderegg Mar 13, 2020
983df67
added retrial policy for redis
sanderegg Mar 13, 2020
d9cf5b9
minor
sanderegg Mar 13, 2020
5168609
linting
sanderegg Mar 15, 2020
c645fc6
added policy for celery retrials
sanderegg Mar 15, 2020
2d9bfd9
annotations
sanderegg Mar 16, 2020
1cb9743
added new command line in pytest to make development faster
sanderegg Mar 16, 2020
3dc75e0
improving test
sanderegg Mar 16, 2020
00ef622
added os environ for rabbit
sanderegg Mar 17, 2020
2a726d1
added minio fixture
sanderegg Mar 17, 2020
03ccac3
linting
sanderegg Mar 18, 2020
bddedf3
linting
sanderegg Mar 18, 2020
bfcec06
make sidecar async and set return values
sanderegg Mar 6, 2020
3f9894d
annotations
sanderegg Mar 6, 2020
487fc8e
async calls, cleanup
sanderegg Mar 13, 2020
5b42942
formatting
sanderegg Mar 13, 2020
beb4098
asyncing + formatting
sanderegg Mar 13, 2020
29f3e5d
refactoring
sanderegg Mar 13, 2020
9326ada
minor
sanderegg Mar 13, 2020
cbb4ee3
moving fixtures to package pytest-fixtures
sanderegg Mar 13, 2020
d573c4d
adding makefile in sidecar
sanderegg Mar 13, 2020
8a77bb4
creating tests for sidecar
sanderegg Mar 13, 2020
935689d
remove pytest-docker
sanderegg Mar 13, 2020
8d48051
moved helpers in common package
sanderegg Mar 13, 2020
5fac359
added aiopika, aiodocker
sanderegg Mar 13, 2020
4e92041
deprecate current sidecar tests
sanderegg Mar 13, 2020
e2375a4
minor
sanderegg Mar 13, 2020
49fd403
typo
sanderegg Mar 13, 2020
0d0ae59
added pytest-aiohttp
sanderegg Mar 13, 2020
b72f3b1
added depenency to servicelib
sanderegg Mar 13, 2020
4f0ebec
removed usage of unnecessary fixtures
sanderegg Mar 13, 2020
a78cc65
add version config
sanderegg Mar 13, 2020
0f71498
aio-pika in tests
sanderegg Mar 13, 2020
5fefd70
do not print the path
sanderegg Mar 13, 2020
615609e
back to normal
sanderegg Mar 13, 2020
d1ec5ba
added environs fixture
sanderegg Mar 13, 2020
117b694
add environs fixture
sanderegg Mar 13, 2020
e4497ae
postgres service fixture independent of webserver
sanderegg Mar 13, 2020
a8309e0
celery service independent of webserver
sanderegg Mar 13, 2020
452f7c6
celery config independent of webserver
sanderegg Mar 13, 2020
d091437
rabbit fixture independent of webserver
sanderegg Mar 13, 2020
dc340cb
add integration testing of sidecar
sanderegg Mar 13, 2020
a608ed5
add rabbitmq utils for retrial policy
sanderegg Mar 13, 2020
7dab132
added retrial policy for redis
sanderegg Mar 13, 2020
8209711
minor
sanderegg Mar 13, 2020
85e92f1
minor
sanderegg Mar 13, 2020
3f404a4
only update aio-pika
sanderegg Mar 13, 2020
39d3bd7
pylint
sanderegg Mar 13, 2020
99bd595
use logged gather to enhance logging
sanderegg Mar 13, 2020
304c222
lintin
sanderegg Mar 13, 2020
aa3f248
update aio-pika
sanderegg Mar 15, 2020
0edc26b
linting
sanderegg Mar 15, 2020
edb18e4
linting
sanderegg Mar 15, 2020
3fc321a
added policy for celery retrials
sanderegg Mar 15, 2020
7a2ae32
remove usage of celery for now
sanderegg Mar 15, 2020
5adbbdb
invalid depend
sanderegg Mar 16, 2020
89bbd19
fixed dockerfile missing libraries
sanderegg Mar 16, 2020
302984f
multi-stage python build
sanderegg Mar 16, 2020
76118fe
format + fixed type
sanderegg Mar 16, 2020
3ef982d
use debian-based image
sanderegg Mar 16, 2020
2e76aa9
add argument to change python version
sanderegg Mar 16, 2020
b449b55
add default value for SWARM_STACK_NAME
sanderegg Mar 16, 2020
515e457
show warning if not running in a container
sanderegg Mar 16, 2020
d9278a8
use common makefile
sanderegg Mar 16, 2020
197a5d5
check if in devel mode before installing dev targets
sanderegg Mar 16, 2020
8b669ab
annotations
sanderegg Mar 16, 2020
40089e5
typo
sanderegg Mar 16, 2020
4fcd09c
fixed issue installing with pip-sync
sanderegg Mar 16, 2020
5af2c98
shellcheck
sanderegg Mar 16, 2020
1ccb2ce
improving test
sanderegg Mar 16, 2020
65e0ac3
adapt call
sanderegg Mar 17, 2020
a079977
linter
sanderegg Mar 17, 2020
fa28b2c
added minio service
sanderegg Mar 17, 2020
836e069
added os environ for rabbit
sanderegg Mar 17, 2020
97b68cb
added minio fixture
sanderegg Mar 17, 2020
4e3da5f
added simcore storage fixture
sanderegg Mar 17, 2020
1ae82d4
added aiofiles
sanderegg Mar 17, 2020
73b3ad4
towards integration test
sanderegg Mar 17, 2020
abe90dc
adding aiofiles
sanderegg Mar 17, 2020
a56e1d6
formatting
sanderegg Mar 17, 2020
5f2cb3a
use f formatting
sanderegg Mar 18, 2020
76109b4
cancel background task
sanderegg Mar 18, 2020
a881646
allow setting host folder
sanderegg Mar 18, 2020
f0492e6
testing sidecar 1st draft
sanderegg Mar 18, 2020
4f0107a
linting
sanderegg Mar 18, 2020
ce90b18
cleanup dockerfile
sanderegg Mar 18, 2020
af8c71f
cleanup
sanderegg Mar 18, 2020
d58f11d
lint
sanderegg Mar 18, 2020
925edae
create rabbit connection with aio-pika
sanderegg Mar 18, 2020
319d957
use rabbit connection fixture
sanderegg Mar 18, 2020
c9fcf83
added queue consumer
sanderegg Mar 18, 2020
2a4572c
build essential package contains gcc
sanderegg Mar 18, 2020
5391ab2
dockerfile update
sanderegg Mar 18, 2020
f7df6b7
cleaning up
sanderegg Mar 18, 2020
e3dceeb
moving sidecar to simcore_service_sidecar
sanderegg Mar 19, 2020
55e9ab8
added pydantic
sanderegg Mar 19, 2020
fc031d1
follow C extensions to pydantic
sanderegg Mar 19, 2020
3294462
till it goes click
sanderegg Mar 19, 2020
2b5805b
deprecated means not used
sanderegg Mar 19, 2020
d953c53
using pydantic
sanderegg Mar 19, 2020
7d9db02
not re-raising exceptioin
sanderegg Mar 19, 2020
4d24ad7
Minor: doc, format and minor changes
Mar 19, 2020
04a224a
collects all requirements of this folder
Mar 19, 2020
ace583e
using a context manager to access rabbit
sanderegg Mar 19, 2020
64506fe
Created pytest-simcore plugin
Mar 19, 2020
76f9d94
Test runs
Mar 20, 2020
10de3a9
renames test
Mar 20, 2020
9455fca
Final fixes in pytest plugin module
Mar 20, 2020
e8a8cdd
Installs and uses pytest simcore plugin in webserver
Mar 20, 2020
7ffc4f0
Doc and cc config
Mar 20, 2020
2769113
Delete pytest-fixtures folder
Mar 20, 2020
f99959b
Added makefile
Mar 20, 2020
45e6d86
Updating ownership
Mar 20, 2020
61a6b13
Adds extra test
Mar 20, 2020
24cc36b
Fixes/improves issue with simcore root
Mar 20, 2020
09fb0fc
Fixed bad merge
Mar 20, 2020
03311ce
use pytest-simcore package
sanderegg Mar 20, 2020
ce1d479
one entrypoint only
sanderegg Mar 20, 2020
a528177
liinting
sanderegg Mar 20, 2020
ac9a939
fixture: add get_ip when multiple network interfaces are around
sanderegg Mar 20, 2020
b5f9f74
fixture: missing connection, exchange and channel
sanderegg Mar 20, 2020
8f2bc81
bad merge
sanderegg Mar 20, 2020
819876c
fixture: bad merge2
sanderegg Mar 20, 2020
18c60d9
fixture: bad merge3
sanderegg Mar 20, 2020
5dc3f17
fixture: docker swarm shall not go down when keep-docker-up set
sanderegg Mar 20, 2020
990c9af
moving to pydantic
sanderegg Mar 20, 2020
df872a4
pydantic issue with assignment of _ prefixed variables
sanderegg Mar 20, 2020
4c4d8c0
typo
sanderegg Mar 20, 2020
7807c53
typo
sanderegg Mar 20, 2020
ae99e63
pydantic does not allow assignment on _ prefixed variables
sanderegg Mar 20, 2020
ebfb163
basic aio-pika works
sanderegg Mar 20, 2020
5e12a67
integration test works again bye bye pydantic
sanderegg Mar 20, 2020
0610665
add sidecar unit testing
sanderegg Mar 22, 2020
e3e33b7
aiodocker
sanderegg Mar 22, 2020
7bc5089
clean up
sanderegg Mar 23, 2020
8a8aa2b
update requirements
sanderegg Mar 23, 2020
4c7facb
docker used in fixtures
sanderegg Mar 23, 2020
d5a6cbc
pydantic
sanderegg Mar 23, 2020
8b98f9a
created aiopg async context manager
sanderegg Mar 23, 2020
54f922c
add depency to simcore-postgres-database package
sanderegg Mar 23, 2020
1cba9f2
cleanup
sanderegg Mar 23, 2020
69867f9
fixed Makefile
sanderegg Mar 24, 2020
11c904d
added bucket fixture
sanderegg Mar 24, 2020
f6cd664
changed endpoint to not contain scheme
sanderegg Mar 24, 2020
09c5cbf
reset change
sanderegg Mar 24, 2020
67bca87
cleanup
sanderegg Mar 24, 2020
2fee460
use pytest-simcore fixtures
sanderegg Mar 24, 2020
f886139
nodeports use aiopg as sql engine
sanderegg Mar 24, 2020
df66eda
fix issue with S3_SECURE not being a 0,1
sanderegg Mar 24, 2020
953e3ae
added sidecar models
sanderegg Mar 24, 2020
7b1f6c8
minor
sanderegg Mar 24, 2020
fbaab8c
adds sidecar exceptions
sanderegg Mar 24, 2020
64d1ca9
minor
sanderegg Mar 24, 2020
b4ca6f2
add callbacks to detect disconnections
sanderegg Mar 24, 2020
7df0d64
only fail in case of exception
sanderegg Mar 24, 2020
4b4ed0a
welcome aiopg instead of synchronous sql access
sanderegg Mar 24, 2020
c2f1d3a
remove the old stuff
sanderegg Mar 24, 2020
ae155be
pylint
sanderegg Mar 24, 2020
f05f9c4
pylint
sanderegg Mar 24, 2020
3b54d78
missing requirements
sanderegg Mar 24, 2020
97036ef
remove unnecessary files
sanderegg Mar 24, 2020
4c4de25
beautify
sanderegg Mar 24, 2020
8b6d7be
Merge branch 'master' into bugfix/sidecar_check_return_value
sanderegg Mar 24, 2020
ec392b9
bad merge
sanderegg Mar 24, 2020
a042c8b
this causes more problems
sanderegg Mar 24, 2020
0ab3618
updated checkout version in github actions
sanderegg Mar 25, 2020
8ed58f6
update docker-compose version to 1.25.3
sanderegg Mar 25, 2020
4478758
missing link
sanderegg Mar 25, 2020
a84cffd
fixed linter
sanderegg Mar 25, 2020
ec7ede8
use pytest-simcore
sanderegg Mar 25, 2020
f5914b3
fix execution flag
sanderegg Mar 25, 2020
09b0b85
remove the container after use
sanderegg Mar 25, 2020
78d7be4
minor
sanderegg Mar 25, 2020
f2e08f6
add check=True
sanderegg Mar 25, 2020
be550fc
docker registry tenacity parameters
sanderegg Mar 25, 2020
4fda38a
use retry to ensure async behaviour
sanderegg Mar 25, 2020
df01a47
docker is already setup, so no need to do it twice
sanderegg Mar 25, 2020
71e7310
show info of docker as well
sanderegg Mar 25, 2020
a69a57e
show docker version complete output
sanderegg Mar 25, 2020
c470649
add click in the requirements
sanderegg Mar 25, 2020
71549c7
add log level in director envs
sanderegg Mar 25, 2020
76172b7
migrate director to debian docker
sanderegg Mar 25, 2020
293cd2f
missing git
sanderegg Mar 25, 2020
a86c43c
minor
sanderegg Mar 25, 2020
19def98
mount the whole package folder in devel mode
sanderegg Mar 25, 2020
d80b56b
typos
sanderegg Mar 25, 2020
c5bf0e0
fix module name
sanderegg Mar 25, 2020
2fc6340
typo
sanderegg Mar 25, 2020
6a73ec2
fix typos
sanderegg Mar 25, 2020
040eb0d
fix call to su
sanderegg Mar 25, 2020
172b52c
move task in celery app
sanderegg Mar 25, 2020
d1b2666
fix message syntax
sanderegg Mar 25, 2020
2c3c396
minor
sanderegg Mar 25, 2020
d2b67e2
fix type
sanderegg Mar 25, 2020
451813c
update the task
sanderegg Mar 25, 2020
d167a4a
sidecar runs without showing the logs
sanderegg Mar 25, 2020
e010b3f
async for in aiofile does not block
sanderegg Mar 25, 2020
387c2b4
minor
sanderegg Mar 26, 2020
0af5476
add docker buildx buildsteps
sanderegg Mar 26, 2020
e430f3c
added separate log_parser
sanderegg Mar 26, 2020
f70c34a
refactoring core
sanderegg Mar 26, 2020
5d09ab1
improve logs
sanderegg Mar 26, 2020
cfd6d28
show who is logged in rabbit UI
sanderegg Mar 26, 2020
e32cefa
enable docker experimental
sanderegg Mar 26, 2020
188e475
test build-x
sanderegg Mar 26, 2020
71d9061
include buildx in github actions
sanderegg Mar 26, 2020
0a885bb
use buildx
sanderegg Mar 26, 2020
949a09e
fix integration tests
sanderegg Mar 26, 2020
e170be3
api-gateway is not pushed to registry
sanderegg Mar 26, 2020
76d3ffe
minor
sanderegg Mar 26, 2020
8474f28
wrong path
sanderegg Mar 26, 2020
d6b6a7c
minor
sanderegg Mar 26, 2020
b274343
use the same local production images as in make up-prod
sanderegg Mar 26, 2020
4e790f4
Merge branch 'master' into bugfix/sidecar_check_return_value
sanderegg Mar 26, 2020
191dca1
make PR faster by not building images in PR mode
sanderegg Mar 26, 2020
083c047
replace celery by the rabbit
sanderegg Mar 26, 2020
d8d9a3a
Merge branch 'bugfix/sidecar_check_return_value' of github.com:sander…
sanderegg Mar 26, 2020
31815f2
make building test images conditional in PRs
sanderegg Mar 26, 2020
b26bc07
only replace environ if none set before
sanderegg Mar 26, 2020
0e0944b
no sidecar here
sanderegg Mar 26, 2020
ded2c35
moved requirements in its own folder
sanderegg Mar 27, 2020
85969da
adds build-kit recipes
sanderegg Mar 27, 2020
3f303c2
use pytest-simcore
sanderegg Mar 27, 2020
cf9999e
better debuggin
sanderegg Mar 27, 2020
37ca090
use pytest-simcore
sanderegg Mar 27, 2020
d77366d
fix path
sanderegg Mar 27, 2020
52675d8
fix path
sanderegg Mar 27, 2020
a9817a7
added aio-pika
sanderegg Mar 27, 2020
21640ec
unnecessary assert
sanderegg Mar 27, 2020
77e3660
@crespov contribution
sanderegg Mar 27, 2020
fa626bf
@pcrespov review
sanderegg Mar 27, 2020
5a9a46e
Merge branch 'master' into bugfix/sidecar_check_return_value
sanderegg Mar 27, 2020
9410456
fix bad merge
sanderegg Mar 27, 2020
a4f29ef
fix execution flag
sanderegg Mar 27, 2020
8acc909
fix path
sanderegg Mar 27, 2020
52a234f
set DOCKER_REGISTRY to default
sanderegg Mar 27, 2020
fad42aa
cannot define in travis and repo settings
sanderegg Mar 27, 2020
caa478b
default to itisfoundation if no docker-registry defined
sanderegg Mar 27, 2020
cfad941
final fix to travis
sanderegg Mar 27, 2020
daf5fb9
test only running on travis
sanderegg Mar 27, 2020
b4dbd2e
sidecar integration testing in travis
sanderegg Mar 27, 2020
3ee7277
minor
sanderegg Mar 27, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion services/sidecar/src/simcore_service_sidecar/db.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,6 @@ async def __aenter__(self):
async def __aexit__(self, exc_type, exc, tb):
self._db_engine.close()
await self._db_engine.wait_closed()
assert self._db_engine.closed
log.debug(
"engine '%s' after shutdown: closed=%s, size=%d",
self._db_engine.dsn,
Expand Down