[wip] Update to pyodide 0.22.0, use pyodide and serviceworker types #871

bollwyvl · 2022-11-11T16:09:28Z

References

could be moved to new repo if Move the Pyodide kernel to a separate repo #854 lands first
fixes notebook attachment: URLs incorrectly routed to ServiceWorker #792 (it wasn't really broken)
investigates Serviceworker re-registers every time #751

Code changes

User-facing changes

potentially better error messages on startup if Things Go Wrong

Backwards-incompatible changes

n/a
- the sqlite3 change might have been surprising...
  - but we'll always be loading it so IPython can load

github-actions · 2022-11-11T16:09:42Z

👈 Try it on ReadTheDocs

bollwyvl · 2022-11-12T15:06:33Z

packages/pyolite-kernel/src/worker.ts

@@ -582,10 +462,10 @@ export class PyoliteRemoteKernel {
    reject: () => void;
    resolve: () => void;
  } | null = null;
+  protected _pyodide: Pyodide.PyodideInterface = null as any;


not proud of this...

bollwyvl · 2022-11-12T15:08:24Z

packages/pyolite-kernel/src/worker.ts

  /** TODO: real typing */
  protected _localPath = '';
  protected _driveName = '';
-  protected _pyodide: any;
  protected _kernel: any;


These anys still kill me. Among the tasks we should explore in #386 are:

doing a thorough typing of all our custom classes

generating typescript types from the python types Typescript types for Python objects pyodide/pyodide#2162

bollwyvl · 2022-11-12T18:43:17Z

packages/server/src/serviceworker.ts

+      registration =
+        (await serviceWorker.getRegistration(serviceWorker.controller.scriptURL)) ||
+        null;
+      console.info('JupyterLite ServiceWorker was already registered');


bollwyvl · 2022-11-13T19:19:25Z

Seems to be working on CI. Still no progress on #840, even when trying a more complex setup for handling potential race conditions on the BroadcastChannel.

Had a look at a few other things, rolled them back:

workbox for serviceworker
- there's a lot here, like plugins and strategies, but I don't know how we'd get them to work inside of a simple script
- we can however pass some config in via sw-1234.js?param=value, so there might be some things that would work well there in a more configurable way
bringing back importhook Add back importhook for patches #254
- causes IPython to fail loading due to pygments tricksiness, so probably would break other things

bollwyvl · 2022-11-14T14:44:36Z

packages/contents/src/drivefs.ts

+    if (
+      length <= 0 ||
+      stream.file === undefined ||
+      position >= (stream.file.data.length || 0)


ref

Pyolite sqlite3 doesn't work inside the serviceworker-provided FS #840 (comment)

c7c2907#r89808665

There's still something happening in the pyolite case, where doing successive writes without closing between them yields a database disk image is malformed

bollwyvl · 2022-12-12T14:45:48Z

packages/pyolite-kernel/src/worker.ts

@@ -197,6 +78,7 @@ export class PyoliteRemoteKernel {
  protected async initKernel(options: IPyoliteWorkerKernel.IOptions): Promise<void> {
    // from this point forward, only use piplite (but not %pip)
    await this._pyodide.runPythonAsync(`
+      await piplite.install(['sqlite3'], keep_going=True)


This will still be needed in #909

bollwyvl · 2022-12-12T14:47:02Z

dodo.py

+    PYODIDE_CDN_URL = APP_SCHEMA_PYOLITE["properties"]["pyodideUrl"]["default"]
+    PYODIDE_VERSION = _js_version_to_py_version(PYODIDE_JS_VERSION)
+    PYODIDE_JS = PYODIDE_CDN_URL.split("/")[-1]
+    PYODIDE_ARCHIVE = f"pyodide-{PYODIDE_VERSION}.tar.bz2"


this name will need to change, removing -build in #909

Sounds good let's keep track of that in #909.

bollwyvl added 5 commits November 11, 2022 08:51

start pyodide 0.22.0

b71633d

resolve yarn.lock after dependabot merge

a36aa76

use upstream pyodide types, remove some duplicates and anys

d4e37cf

add unvendored sqlite

319911e

add SKIP_ESLINT

c0b2787

sort package.json

9062f13

bollwyvl mentioned this pull request Nov 11, 2022

Typescript types for Python objects pyodide/pyodide#2162

Open

bollwyvl added maintenance api-change dependencies Pull requests that update a dependency file labels Nov 11, 2022

make devDependencies/pyodide source-of-truth

8ff0197

bollwyvl removed the api-change label Nov 11, 2022

bollwyvl commented Nov 12, 2022

View reviewed changes

bollwyvl added 4 commits November 12, 2022 09:41

make serviceworker more optional

d4de412

use more async to wait for registration, etc

2569868

refactor broadcastchannel

208202c

update cytoscape example with attachments from upstream

b1c76f9

bollwyvl mentioned this pull request Nov 12, 2022

notebook attachment: URLs incorrectly routed to ServiceWorker #792

Closed

linting

5499c30

bollwyvl added kernel:pyodide related to the Pyodide webworker kernel kernel:serviceworker ServiceWorker-related challenges labels Nov 12, 2022

linting

fb7eb4b

bollwyvl commented Nov 12, 2022

View reviewed changes

bollwyvl added 5 commits November 12, 2022 12:44

resolve _or_ reject ready promise, not both

76f5ff9

tidy up, lint offline.md for pyodide bz2 link

883be0b

investigate sqlite issues, add northwinds example

8227f94

more work on sqlite example, try setattr again

5bed020

try serviceworker in ts

ef47c96

bollwyvl added 2 commits November 13, 2022 09:52

bump typedoc, etc

c9f519c

start updating typedoc

18e4fe1

bollwyvl changed the title ~~[wip] Update to pyodide 0.22.0~~ [wip] Update to pyodide 0.22.0, use pyodide and serviceworker types Nov 13, 2022

bollwyvl added 3 commits November 13, 2022 10:00

bump resolutions

6c8ef3f

resolve examples

4dba723

lint svg, rework mystify

ebc9782

some more work on drivefs

749b311

bollwyvl commented Nov 14, 2022

View reviewed changes

jtpio added this to the 0.1.0 milestone Nov 15, 2022

Merge remote-tracking branch 'upstream/main' into bumpy-pyodide-0220

a683f1a

This was referenced Dec 7, 2022

Service worker fixes, JupyterLab 3.5.1 #892

Closed

Service worker fixes, JupyterLab 3.5.1 #899

Merged

jtpio merged commit ffa115d into jupyterlite:main Dec 12, 2022

bollwyvl mentioned this pull request Dec 12, 2022

Update to pyodide 0.22.0 #909

Closed

2 tasks

bollwyvl commented Dec 12, 2022

View reviewed changes

jtpio mentioned this pull request Nov 29, 2023

Fixed so hard refresh loads service worker properly #1251

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[wip] Update to pyodide 0.22.0, use pyodide and serviceworker types #871

[wip] Update to pyodide 0.22.0, use pyodide and serviceworker types #871

Uh oh!

bollwyvl commented Nov 11, 2022 •

edited

Loading

Uh oh!

github-actions bot commented Nov 11, 2022

Uh oh!

bollwyvl Nov 12, 2022

Uh oh!

bollwyvl Nov 12, 2022

Uh oh!

bollwyvl Nov 12, 2022

Uh oh!

bollwyvl commented Nov 13, 2022

Uh oh!

bollwyvl Nov 14, 2022

Uh oh!

bollwyvl Dec 12, 2022

Uh oh!

bollwyvl Dec 12, 2022

Uh oh!

jtpio Dec 12, 2022

Uh oh!

Uh oh!

[wip] Update to pyodide 0.22.0, use pyodide and serviceworker types #871

[wip] Update to pyodide 0.22.0, use pyodide and serviceworker types #871

Uh oh!

Conversation

bollwyvl commented Nov 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

References

Code changes

User-facing changes

Backwards-incompatible changes

Uh oh!

github-actions bot commented Nov 11, 2022

Uh oh!

bollwyvl Nov 12, 2022

Choose a reason for hiding this comment

Uh oh!

bollwyvl Nov 12, 2022

Choose a reason for hiding this comment

Uh oh!

bollwyvl Nov 12, 2022

Choose a reason for hiding this comment

Uh oh!

bollwyvl commented Nov 13, 2022

Uh oh!

bollwyvl Nov 14, 2022

Choose a reason for hiding this comment

Uh oh!

bollwyvl Dec 12, 2022

Choose a reason for hiding this comment

Uh oh!

bollwyvl Dec 12, 2022

Choose a reason for hiding this comment

Uh oh!

jtpio Dec 12, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bollwyvl commented Nov 11, 2022 •

edited

Loading