Skip to content

Commit f2d1972

Browse files
cmyuiCIpre-commit-ci[bot]
authored
Tests running in CI & score submission integration test (#508)
* consolidate & simplify * app is running in new setup * expose on 10000 * flip order of test naming * empty osu test file for itests * basic score submission integration test * test in ci * image on dockerhub & build in tests ci for now * better todo * fix * login into dockerhub * build in ci and fix exec * update test job name * env mapping * fix script path * move pytest install to packages (from dev packages) * fix run-tests script dir * make test to use docker-compose, make test-local for existing functionality * simplify run-tests.sh & print stdout * [CI] Generated new Pipfile.lock * fix volumes * todone * on push for all tests * add respx requirement for mocking http calls * fix mypy err * return charts even when on rx/ap * assert against chart resp * [CI] Generated new Pipfile.lock * Revert "[CI] Generated new Pipfile.lock" This reverts commit 7eabb76e01bc4de1c856ae8ac0f5aa77232b0c2e. * remove OSU_API_KEY from required keys; default none * update snapshot * consistent log msg * remove unused mods * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * mock out calls to 3rd party apis for bmap data * use `respx_mock` fixture name * [CI] Generated new Pipfile.lock * only include status, msg, country code, lat, lon in ip-api resp * mock ip-api call --------- Co-authored-by: CI <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent 112012a commit f2d1972

25 files changed

+1420
-766
lines changed

Diff for: ‎.github/workflows/lock-requirements.yaml

+1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
name: Generate Requirements Lockfile
2+
23
on:
34
push:
45
paths:

Diff for: ‎.github/workflows/require-checklist.yaml

+2
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,11 @@
11
name: Require Checklist
2+
23
on:
34
pull_request:
45
types: [opened, edited, synchronize]
56
issues:
67
types: [opened, edited, deleted]
8+
79
jobs:
810
require-checklist:
911
runs-on: ubuntu-latest

Diff for: ‎.github/workflows/sync-docs.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Sync wiki documentation
1+
name: Sync Wiki Documentation
22

33
on:
44
push:

Diff for: ‎.github/workflows/test.yaml

+57
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
name: Test Application Code
2+
3+
on: push
4+
5+
env:
6+
DB_USER: ${{ vars.DB_USER }}
7+
DB_PASS: ${{ vars.DB_PASS }}
8+
DB_NAME: ${{ vars.DB_NAME }}
9+
DB_HOST: ${{ vars.DB_HOST }}
10+
DB_PORT: ${{ vars.DB_PORT }}
11+
REDIS_PASS: ${{ vars.REDIS_PASS }}
12+
APP_PORT: ${{ vars.APP_PORT }}
13+
APP_HOST: ${{ vars.APP_HOST }}
14+
REDIS_USER: ${{ vars.REDIS_USER }}
15+
REDIS_HOST: ${{ vars.REDIS_HOST }}
16+
REDIS_PORT: ${{ vars.REDIS_PORT }}
17+
REDIS_DB: ${{ vars.REDIS_DB }}
18+
MIRROR_SEARCH_ENDPOINT: ${{ vars.MIRROR_SEARCH_ENDPOINT }}
19+
MIRROR_DOWNLOAD_ENDPOINT: ${{ vars.MIRROR_DOWNLOAD_ENDPOINT }}
20+
DOMAIN: ${{ vars.DOMAIN }}
21+
COMMAND_PREFIX: ${{ vars.COMMAND_PREFIX }}
22+
SEASONAL_BGS: ${{ vars.SEASONAL_BGS }}
23+
MENU_ICON_URL: ${{ vars.MENU_ICON_URL }}
24+
MENU_ONCLICK_URL: ${{ vars.MENU_ONCLICK_URL }}
25+
DATADOG_API_KEY: ${{ vars.DATADOG_API_KEY }}
26+
DATADOG_APP_KEY: ${{ vars.DATADOG_APP_KEY }}
27+
DEBUG: ${{ vars.DEBUG }}
28+
REDIRECT_OSU_URLS: ${{ vars.REDIRECT_OSU_URLS }}
29+
PP_CACHED_ACCS: ${{ vars.PP_CACHED_ACCS }}
30+
DISALLOWED_NAMES: ${{ vars.DISALLOWED_NAMES }}
31+
DISALLOWED_PASSWORDS: ${{ vars.DISALLOWED_PASSWORDS }}
32+
DISALLOW_OLD_CLIENTS: ${{ vars.DISALLOW_OLD_CLIENTS }}
33+
DISCORD_AUDIT_LOG_WEBHOOK: ${{ vars.DISCORD_AUDIT_LOG_WEBHOOK }}
34+
AUTOMATICALLY_REPORT_PROBLEMS: ${{ vars.AUTOMATICALLY_REPORT_PROBLEMS }}
35+
SSL_CERT_PATH: ${{ vars.SSL_CERT_PATH }}
36+
SSL_KEY_PATH: ${{ vars.SSL_KEY_PATH }}
37+
DEVELOPER_MODE: ${{ vars.DEVELOPER_MODE }}
38+
39+
jobs:
40+
run-test-suite:
41+
runs-on: ubuntu-latest
42+
steps:
43+
- name: Checkout repository
44+
uses: actions/checkout@v3
45+
46+
- name: Build application image
47+
run: docker build -t bancho:latest .
48+
49+
- name: Start containers
50+
run: docker-compose up -d bancho mysql redis
51+
52+
- name: Run tests
53+
run: docker-compose exec -T bancho /srv/root/scripts/run-tests.sh
54+
55+
- name: Stop containers
56+
if: always()
57+
run: docker-compose down

Diff for: ‎Makefile

+4-1
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,13 @@ shell:
1818
pipenv shell
1919

2020
test:
21+
docker-compose exec -T bancho /srv/root/scripts/run-tests.sh
22+
23+
test-local:
2124
pipenv run pytest -vv tests/
2225

2326
test-dbg:
24-
pipenv run pytest -vv --pdb tests/
27+
pipenv run pytest -vv --pdb -s tests/
2528

2629
lint:
2730
pipenv run pre-commit run --all-files

Diff for: ‎Pipfile

+4-3
Original file line numberDiff line numberDiff line change
@@ -26,16 +26,17 @@ cryptography = "*"
2626
tenacity = "*"
2727
httpx = "*"
2828
py-cpuinfo = "*"
29+
pytest = "*"
30+
pytest-asyncio = "*"
31+
asgi-lifespan = "*"
32+
respx = "*"
2933

3034
[dev-packages]
31-
pytest = "*"
3235
pre-commit = "*"
3336
black = "*"
3437
reorder-python-imports = "*"
3538
mypy = "*"
3639
autoflake = "*"
37-
asgi-lifespan = "*"
38-
pytest-asyncio = "*"
3940
types-psutil = "*"
4041
types-pymysql = "*"
4142
types-requests = "*"

Diff for: ‎Pipfile.lock

+705-713
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Diff for: ‎app/api/domains/osu.py

+3-31
Original file line numberDiff line numberDiff line change
@@ -36,14 +36,13 @@
3636
from fastapi.responses import RedirectResponse
3737
from fastapi.responses import Response
3838
from fastapi.routing import APIRouter
39-
from py3rijndael import Pkcs7Padding
40-
from py3rijndael import RijndaelCbc
4139
from starlette.datastructures import UploadFile as StarletteUploadFile
4240

4341
import app.packets
4442
import app.settings
4543
import app.state
4644
import app.utils
45+
from app import encryption
4746
from app._typing import UNSET
4847
from app.constants import regexes
4948
from app.constants.clientflags import LastFMFlags
@@ -613,28 +612,6 @@ def parse_form_data_score_params(
613612
)
614613

615614

616-
def decrypt_score_aes_data(
617-
# to decode
618-
score_data_b64: bytes,
619-
client_hash_b64: bytes,
620-
# used for decoding
621-
iv_b64: bytes,
622-
osu_version: str,
623-
) -> tuple[list[str], str]:
624-
"""Decrypt the base64'ed score data."""
625-
aes = RijndaelCbc(
626-
key=f"osu!-scoreburgr---------{osu_version}".encode(),
627-
iv=b64decode(iv_b64),
628-
padding=Pkcs7Padding(32),
629-
block_size=32,
630-
)
631-
632-
score_data = aes.decrypt(b64decode(score_data_b64)).decode().split(":")
633-
client_hash_decoded = aes.decrypt(b64decode(client_hash_b64)).decode()
634-
635-
return score_data, client_hash_decoded
636-
637-
638615
@router.post("/web/osu-submit-modular-selector.php")
639616
async def osuSubmitModularSelector(
640617
request: Request,
@@ -674,7 +651,7 @@ async def osuSubmitModularSelector(
674651
score_data_b64, replay_file = score_parameters
675652

676653
# decrypt the score data (aes)
677-
score_data, client_hash_decoded = decrypt_score_aes_data(
654+
score_data, client_hash_decoded = encryption.decrypt_score_aes_data(
678655
score_data_b64,
679656
client_hash_b64,
680657
iv_b64,
@@ -1083,12 +1060,7 @@ async def osuSubmitModularSelector(
10831060
""" score submission charts """
10841061

10851062
# charts are only displayed for passes vanilla gamemodes.
1086-
if not score.passed or score.mode not in (
1087-
GameMode.VANILLA_OSU,
1088-
GameMode.VANILLA_TAIKO,
1089-
GameMode.VANILLA_CATCH,
1090-
GameMode.VANILLA_MANIA,
1091-
):
1063+
if not score.passed: # TODO: check if this is correct
10921064
response = b"error: no"
10931065
else:
10941066
# construct and send achievements & ranking charts to the client

Diff for: ‎app/commands.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -1224,7 +1224,7 @@ async def server(ctx: Context) -> str | None:
12241224
# current state of settings
12251225
mirror_search_url = urlparse(app.settings.MIRROR_SEARCH_ENDPOINT).netloc
12261226
mirror_download_url = urlparse(app.settings.MIRROR_DOWNLOAD_ENDPOINT).netloc
1227-
using_osuapi = app.settings.OSU_API_KEY != ""
1227+
using_osuapi = bool(app.settings.OSU_API_KEY)
12281228
advanced_mode = app.settings.DEVELOPER_MODE
12291229
auto_logging = app.settings.AUTOMATICALLY_REPORT_PROBLEMS
12301230

Diff for: ‎app/encryption.py

+59
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
from __future__ import annotations
2+
3+
from base64 import b64decode
4+
from base64 import b64encode
5+
6+
from py3rijndael import Pkcs7Padding
7+
from py3rijndael import RijndaelCbc
8+
9+
10+
def encrypt_score_aes_data(
11+
# to encode
12+
score_data: list[str],
13+
client_hash: str,
14+
# used for encoding
15+
iv_b64: bytes,
16+
osu_version: str,
17+
) -> tuple[bytes, bytes]:
18+
"""Encrypt the score data to base64."""
19+
# TODO: perhaps this should return TypedDict?
20+
21+
# attempt to encrypt score data
22+
aes = RijndaelCbc(
23+
key=f"osu!-scoreburgr---------{osu_version}".encode(),
24+
iv=b64decode(iv_b64),
25+
padding=Pkcs7Padding(32),
26+
block_size=32,
27+
)
28+
29+
score_data_joined = ":".join(score_data)
30+
score_data_b64 = b64encode(aes.encrypt(score_data_joined.encode()))
31+
client_hash_b64 = b64encode(aes.encrypt(client_hash.encode()))
32+
33+
return score_data_b64, client_hash_b64
34+
35+
36+
def decrypt_score_aes_data(
37+
# to decode
38+
score_data_b64: bytes,
39+
client_hash_b64: bytes,
40+
# used for decoding
41+
iv_b64: bytes,
42+
osu_version: str,
43+
) -> tuple[list[str], str]:
44+
"""Decrypt the base64'ed score data."""
45+
# TODO: perhaps this should return TypedDict?
46+
47+
# attempt to decrypt score data
48+
aes = RijndaelCbc(
49+
key=f"osu!-scoreburgr---------{osu_version}".encode(),
50+
iv=b64decode(iv_b64),
51+
padding=Pkcs7Padding(32),
52+
block_size=32,
53+
)
54+
55+
score_data = aes.decrypt(b64decode(score_data_b64)).decode().split(":")
56+
client_hash_decoded = aes.decrypt(b64decode(client_hash_b64)).decode()
57+
58+
# score data is delimited by colons (:).
59+
return score_data, client_hash_decoded

Diff for: ‎app/objects/beatmap.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ async def ensure_local_osu_file(
8282
):
8383
# need to get the file from the osu!api
8484
if app.settings.DEBUG:
85-
log(f"Doing osu!api (.osu file) request {bmap_id}", Ansi.LMAGENTA)
85+
log(f"Doing api (.osu file) request {bmap_id}", Ansi.LMAGENTA)
8686

8787
url = f"https://old.ppy.sh/osu/{bmap_id}"
8888
response = await app.state.services.http_client.get(url)

Diff for: ‎app/settings.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@
4242
REDIS_AUTH_STRING = f"{REDIS_USER}:{REDIS_PASS}@" if REDIS_USER and REDIS_PASS else ""
4343
REDIS_DSN = f"redis://{REDIS_AUTH_STRING}{REDIS_HOST}:{REDIS_PORT}/{REDIS_DB}"
4444

45-
OSU_API_KEY = os.environ["OSU_API_KEY"]
45+
OSU_API_KEY = os.environ.get("OSU_API_KEY") or None
4646

4747
DOMAIN = os.environ["DOMAIN"]
4848
MIRROR_SEARCH_ENDPOINT = os.environ["MIRROR_SEARCH_ENDPOINT"]

Diff for: ‎app/state/services.py

+11-6
Original file line numberDiff line numberDiff line change
@@ -201,7 +201,12 @@ async def _fetch_geoloc_from_ip(ip: IPAddress) -> Geolocation | None:
201201
else:
202202
url = "http://ip-api.com/line/"
203203

204-
response = await http_client.get(url)
204+
response = await http_client.get(
205+
url,
206+
params={
207+
"fields": ",".join(("status", "message", "countryCode", "lat", "lon")),
208+
},
209+
)
205210
if response.status_code != 200:
206211
log("Failed to get geoloc data: request failed.", Ansi.LRED)
207212
return None
@@ -216,14 +221,14 @@ async def _fetch_geoloc_from_ip(ip: IPAddress) -> Geolocation | None:
216221
log(f"Failed to get geoloc data: {err_msg}.", Ansi.LRED)
217222
return None
218223

219-
acronym = lines[1].lower()
224+
country_acronym = lines[0].lower()
220225

221226
return {
222-
"latitude": float(lines[6]),
223-
"longitude": float(lines[7]),
227+
"latitude": float(lines[1]),
228+
"longitude": float(lines[2]),
224229
"country": {
225-
"acronym": acronym,
226-
"numeric": country_codes[acronym],
230+
"acronym": country_acronym,
231+
"numeric": country_codes[country_acronym],
227232
},
228233
}
229234

Diff for: ‎docker-compose.yml

+3-1
Original file line numberDiff line numberDiff line change
@@ -33,9 +33,11 @@ services:
3333
environment:
3434
- ALLOW_EMPTY_PASSWORD=yes
3535
- REDIS_PASSWORD=${REDIS_PASS}
36-
## application services
36+
37+
## application services
3738

3839
bancho:
40+
# we also have a public image: osuakatsuki/bancho.py:latest
3941
image: bancho:latest
4042
ports:
4143
- ${APP_PORT}:${APP_PORT}

Diff for: ‎requirements-dev.txt

+8-7
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
-i https://pypi.org/simple
2-
asgi-lifespan==2.1.0; python_version >= '3.7'
32
autoflake==2.2.1; python_version >= '3.8'
43
black==23.9.1; python_version >= '3.8'
54
cfgv==3.4.0; python_version >= '3.8'
@@ -8,32 +7,28 @@ click==8.1.7; python_version >= '3.7'
87
distlib==0.3.7
98
filelock==3.12.4; python_version >= '3.8'
109
identify==2.5.29; python_version >= '3.8'
11-
iniconfig==2.0.0; python_version >= '3.7'
1210
mypy==1.5.1; python_version >= '3.8'
1311
mypy-extensions==1.0.0; python_version >= '3.5'
1412
nodeenv==1.8.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6'
1513
packaging==23.1; python_version >= '3.7'
1614
pathspec==0.11.2; python_version >= '3.7'
1715
platformdirs==3.10.0; python_version >= '3.7'
18-
pluggy==1.3.0; python_version >= '3.8'
1916
pre-commit==3.4.0; python_version >= '3.8'
2017
pyflakes==3.1.0; python_version >= '3.8'
21-
pytest==7.4.2; python_version >= '3.7'
22-
pytest-asyncio==0.21.1; python_version >= '3.7'
2318
pyyaml==6.0.1; python_version >= '3.6'
2419
reorder-python-imports==3.11.0; python_version >= '3.8'
2520
setuptools==68.2.2; python_version >= '3.8'
26-
sniffio==1.3.0; python_version >= '3.7'
2721
types-psutil==5.9.5.16
2822
types-pymysql==1.1.0.1
29-
types-requests==2.31.0.4
23+
types-requests==2.31.0.5
3024
types-urllib3==1.26.25.14
3125
typing-extensions==4.8.0; python_version >= '3.8'
3226
virtualenv==20.24.5; python_version >= '3.7'
3327
aiomysql==0.2.0
3428
akatsuki-pp-py==0.9.8; python_version >= '3.7'
3529
annotated-types==0.5.0; python_version >= '3.7'
3630
anyio==3.7.1; python_version >= '3.7'
31+
asgi-lifespan==2.1.0; python_version >= '3.7'
3732
async-timeout==4.0.3; python_version >= '3.7'
3833
bcrypt==4.0.1; python_version >= '3.6'
3934
certifi==2023.7.22; python_version >= '3.6'
@@ -48,19 +43,25 @@ h11==0.14.0; python_version >= '3.7'
4843
httpcore==0.18.0; python_version >= '3.8'
4944
httpx==0.25.0; python_version >= '3.8'
5045
idna==3.4; python_version >= '3.5'
46+
iniconfig==2.0.0; python_version >= '3.7'
5147
orjson==3.9.7; python_version >= '3.7'
48+
pluggy==1.3.0; python_version >= '3.8'
5249
psutil==5.9.5; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
5350
py-cpuinfo==9.0.0
5451
py3rijndael==0.3.3
5552
pycparser==2.21
5653
pydantic==2.3.0; python_version >= '3.7'
5754
pydantic-core==2.6.3; python_version >= '3.7'
5855
pymysql==1.1.0; python_version >= '3.7'
56+
pytest==7.4.2; python_version >= '3.7'
57+
pytest-asyncio==0.21.1; python_version >= '3.7'
5958
python-dotenv==1.0.0; python_version >= '3.8'
6059
python-multipart==0.0.6; python_version >= '3.7'
6160
pytimeparse==1.1.8
6261
redis==5.0.0; python_version >= '3.7'
6362
requests==2.31.0; python_version >= '3.7'
63+
respx==0.20.2; python_version >= '3.7'
64+
sniffio==1.3.0; python_version >= '3.7'
6465
sqlalchemy==1.4.41; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'
6566
starlette==0.27.0; python_version >= '3.7'
6667
tenacity==8.2.3; python_version >= '3.7'

0 commit comments

Comments
 (0)