Skip to content

crash on take-jil #194

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
behrica opened this issue Jan 14, 2022 · 83 comments
Closed

crash on take-jil #194

behrica opened this issue Jan 14, 2022 · 83 comments

Comments

@behrica
Copy link
Contributor

behrica commented Jan 14, 2022

see here:

https://clojurians.zulipchat.com/#narrow/stream/215609-libpython-clj-dev/topic/JVM.20crash.20on.20'take.20gil'
@behrica
Copy link
Contributor Author

behrica commented Jan 14, 2022

Repo with reproducible code will follow soon.

@behrica
Copy link
Contributor Author

behrica commented Jan 14, 2022

A repo with code and instructions on how to reproduce the issue is here:
https://github.com/behrica/libpython-clj--194

@behrica
Copy link
Contributor Author

behrica commented Mar 20, 2022

I am still wondering if this is a bug in libpython-clj or if certain python libraries are incompatible with lib-pythonclj and it's using embedded python

@cnuernber
Copy link
Collaborator

Taking a shot in the dark, I turned of automatic gil management in libpython-clj and ran your example w/o using require-python. This is also running with the julia environment which loads the signal handling jvm stub allowing the jvm process to forward signals to python when python has the appropriate signal handler installed.

It worked - there is a lot to dig through here. Note that disabling the gil management means you have to call exactly the java method take-gil once. When doing the java-api I found out that we use check-gil literally many many times per function call. I think there is a race condition in libpython there. With manual gil management we don't ever check the GIL.

user> (System/getProperty "libpython_clj.manual_gil")
"true"
user> (def  train-data  [
                   ["Example sentence belonging to class 1" 1]
                   ["Example sentence belonging to class 0" 0]])



(def eval-data  [
                 ["Example eval sentence belonging to class 1" 1]
                 ["Example eval sentence belonging to class 0" 0]])

#'user/train-data#'user/eval-data
user> (import '[libpython_clj2 python_api])
Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:387).
libpython_clj2.python_api
user> (import '[libpython_clj2 java_api])
libpython_clj2.java_api
user> (java_api/initialize)
Syntax error (IllegalArgumentException) compiling . at (*cider-repl clj-python/libpython-clj-194:localhost:44857(clj)*:61:7).
No matching method initialize found taking 0 args for class libpython_clj2.java_api
user> (java_api/initialize nil)
Mar 21, 2022 7:26:16 AM clojure.tools.logging$eval8515$fn__8518 invoke
INFO: Detecting startup info
Mar 21, 2022 7:26:16 AM clojure.tools.logging$eval8515$fn__8518 invoke
INFO: Startup info {:lib-version "3.9", :java-library-path-addendum "/home/chrisn/miniconda3/lib", :exec-prefix "/home/chrisn/miniconda3", :executable "/home/chrisn/miniconda3/bin/python3", :libnames ("python3.9m" "python3.9"), :prefix "/home/chrisn/miniconda3", :base-prefix "/home/chrisn/miniconda3", :libname "python3.9m", :base-exec-prefix "/home/chrisn/miniconda3", :python-home "/home/chrisn/miniconda3", :version [3 9 1], :platform "linux"}
Mar 21, 2022 7:26:16 AM clojure.tools.logging$eval8515$fn__8518 invoke
INFO: Prefixing java library path: /home/chrisn/miniconda3/lib
Mar 21, 2022 7:26:17 AM clojure.tools.logging$eval8515$fn__8518 invoke
INFO: Loading python library: python3.9
Mar 21, 2022 7:26:17 AM clojure.tools.logging$eval8515$fn__8518 invoke
INFO: Reference thread starting
:ok
user> (java_api/lockGIL)
1
user> (require '[libpython-clj2.python :as py])
nil
user> (def clsmod (py/import-module "simpletransformers.classification"))
#'user/clsmod
user> (def pd (py/import-module "pandas"))
#'user/pd
user> (def train-df (py/call-attr pd "DataFrame" train-data))
#'user/train-df
user> (def eval-df (py/call-attr pd "DataFrame" eval-data))
#'user/eval-df
user> (def model (py/call-attr-kw clsmod "ClassificationModel" ["roberta" "roberta-base"]
                                  {:use_cude false
                                   :args {:use_multiprocessing false
                                          :overwrite_output_dir true
                                          :dataloader_num_workers 1}}))

Downloading:   0%|          | 0.00/481 [00:00<?, ?B/s]
Downloading: 100%|##########| 481/481 [00:00<00:00, 148kB/s]
Execution error at libpython-clj2.python.ffi/check-error-throw (ffi.clj:703).
Traceback (most recent call last):
  File "/home/chrisn/miniconda3/lib/python3.9/site-packages/simpletransformers/classification/classification_model.py", line 361, in __init__
    raise ValueError(
ValueError: 'use_cuda' set to True when cuda is unavailable. Make sure CUDA is available or set use_cuda=False.

user> (def model (py/call-attr-kw clsmod "ClassificationModel" ["roberta" "roberta-base"]
                                  {:use_cuda false
                                   :args {:use_multiprocessing false
                                          :overwrite_output_dir true
                                          :dataloader_num_workers 1}}))

Downloading:   0%|          | 0.00/478M [00:00<?, ?B/s]
Downloading:   0%|          | 929k/478M [00:00<00:52, 9.50MB/s]
Downloading:   0%|          | 1.93M/478M [00:00<00:48, 10.2MB/s]
Downloading:   1%|          | 2.94M/478M [00:00<00:48, 10.3MB/s]
Downloading:   1%|          | 3.92M/478M [00:00<00:48, 10.2MB/s]
Downloading:   1%|1         | 4.90M/478M [00:00<00:48, 10.2MB/s]
Downloading:   1%|1         | 5.94M/478M [00:00<00:47, 10.4MB/s]
Downloading:   1%|1         | 6.94M/478M [00:00<00:47, 10.4MB/s]
Downloading:   2%|1         | 7.96M/478M [00:00<00:47, 10.5MB/s]
Downloading:   2%|1         | 8.96M/478M [00:00<00:50, 9.71MB/s]
Downloading:   2%|2         | 9.95M/478M [00:01<00:49, 9.90MB/s]
Downloading:   2%|2         | 10.9M/478M [00:01<00:49, 9.97MB/s]
Downloading:   3%|2         | 12.0M/478M [00:01<00:47, 10.3MB/s]
Downloading:   3%|2         | 13.0M/478M [00:01<00:47, 10.3MB/s]
Downloading:   3%|2         | 13.9M/478M [00:01<00:47, 10.2MB/s]
Downloading:   3%|3         | 15.0M/478M [00:01<00:46, 10.4MB/s]
Downloading:   3%|3         | 16.0M/478M [00:01<00:46, 10.3MB/s]
Downloading:   4%|3         | 17.0M/478M [00:01<00:47, 10.3MB/s]
Downloading:   4%|3         | 18.0M/478M [00:01<00:46, 10.5MB/s]
Downloading:   4%|3         | 19.0M/478M [00:01<00:46, 10.5MB/s]
Downloading:   4%|4         | 20.0M/478M [00:02<00:46, 10.3MB/s]
Downloading:   4%|4         | 21.0M/478M [00:02<00:46, 10.3MB/s]
Downloading:   5%|4         | 22.0M/478M [00:02<00:46, 10.3MB/s]
Downloading:   5%|4         | 23.0M/478M [00:02<00:45, 10.4MB/s]
Downloading:   5%|5         | 24.0M/478M [00:02<00:46, 10.2MB/s]
Downloading:   5%|5         | 25.0M/478M [00:02<00:45, 10.4MB/s]
Downloading:   5%|5         | 26.0M/478M [00:02<00:46, 10.2MB/s]
Downloading:   6%|5         | 27.0M/478M [00:02<00:45, 10.4MB/s]
Downloading:   6%|5         | 28.0M/478M [00:02<00:46, 10.3MB/s]
Downloading:   6%|6         | 29.0M/478M [00:02<00:45, 10.3MB/s]
Downloading:   6%|6         | 30.1M/478M [00:03<00:44, 10.5MB/s]
Downloading:   7%|6         | 31.1M/478M [00:03<00:44, 10.5MB/s]
Downloading:   7%|6         | 32.1M/478M [00:03<00:44, 10.4MB/s]
Downloading:   7%|6         | 33.1M/478M [00:03<00:45, 10.3MB/s]
Downloading:   7%|7         | 34.1M/478M [00:03<00:44, 10.4MB/s]
Downloading:   7%|7         | 35.1M/478M [00:03<00:44, 10.3MB/s]
Downloading:   8%|7         | 36.1M/478M [00:03<00:44, 10.5MB/s]
Downloading:   8%|7         | 37.1M/478M [00:03<00:44, 10.4MB/s]
Downloading:   8%|7         | 38.1M/478M [00:03<00:45, 10.1MB/s]
Downloading:   8%|8         | 39.1M/478M [00:03<00:44, 10.3MB/s]
Downloading:   8%|8         | 40.1M/478M [00:04<00:44, 10.3MB/s]
Downloading:   9%|8         | 41.1M/478M [00:04<00:44, 10.2MB/s]
Downloading:   9%|8         | 42.2M/478M [00:04<00:43, 10.5MB/s]
Downloading:   9%|9         | 43.2M/478M [00:04<00:43, 10.4MB/s]
Downloading:   9%|9         | 44.2M/478M [00:04<00:44, 10.3MB/s]
Downloading:   9%|9         | 45.2M/478M [00:04<00:43, 10.4MB/s]
Downloading:  10%|9         | 46.2M/478M [00:04<00:43, 10.4MB/s]
Downloading:  10%|9         | 47.2M/478M [00:04<00:43, 10.5MB/s]
Downloading:  10%|#         | 48.2M/478M [00:04<00:43, 10.4MB/s]
Downloading:  10%|#         | 49.2M/478M [00:05<00:43, 10.4MB/s]
Downloading:  11%|#         | 50.2M/478M [00:05<00:42, 10.5MB/s]
Downloading:  11%|#         | 51.2M/478M [00:05<00:42, 10.4MB/s]
Downloading:  11%|#         | 52.2M/478M [00:05<00:42, 10.4MB/s]
Downloading:  11%|#1        | 53.2M/478M [00:05<00:42, 10.4MB/s]
Downloading:  11%|#1        | 54.2M/478M [00:05<00:44, 10.0MB/s]
Downloading:  12%|#1        | 55.1M/478M [00:05<00:44, 9.91MB/s]
Downloading:  12%|#1        | 56.1M/478M [00:05<00:43, 10.1MB/s]
Downloading:  12%|#1        | 57.1M/478M [00:05<00:43, 10.1MB/s]
Downloading:  12%|#2        | 58.1M/478M [00:05<00:43, 10.1MB/s]
Downloading:  12%|#2        | 59.1M/478M [00:06<00:43, 10.1MB/s]
Downloading:  13%|#2        | 60.1M/478M [00:06<00:43, 10.0MB/s]
Downloading:  13%|#2        | 61.1M/478M [00:06<00:42, 10.2MB/s]
Downloading:  13%|#2        | 62.1M/478M [00:06<00:44, 9.88MB/s]
Downloading:  13%|#3        | 63.1M/478M [00:06<00:42, 10.2MB/s]
Downloading:  13%|#3        | 64.1M/478M [00:06<00:42, 10.2MB/s]
Downloading:  14%|#3        | 65.1M/478M [00:06<00:42, 10.3MB/s]
Downloading:  14%|#3        | 66.1M/478M [00:06<00:41, 10.3MB/s]
Downloading:  14%|#4        | 67.1M/478M [00:06<00:41, 10.3MB/s]
Downloading:  14%|#4        | 68.1M/478M [00:06<00:42, 10.2MB/s]
Downloading:  14%|#4        | 69.0M/478M [00:07<00:41, 10.3MB/s]
Downloading:  15%|#4        | 70.1M/478M [00:07<00:40, 10.4MB/s]
Downloading:  15%|#4        | 71.1M/478M [00:07<00:40, 10.4MB/s]
Downloading:  15%|#5        | 72.1M/478M [00:07<00:40, 10.4MB/s]
Downloading:  15%|#5        | 73.1M/478M [00:07<00:40, 10.4MB/s]
Downloading:  15%|#5        | 74.1M/478M [00:07<00:40, 10.3MB/s]
Downloading:  16%|#5        | 75.1M/478M [00:07<00:41, 10.2MB/s]
Downloading:  16%|#5        | 76.0M/478M [00:07<00:41, 10.2MB/s]
Downloading:  16%|#6        | 77.0M/478M [00:07<00:41, 10.2MB/s]
Downloading:  16%|#6        | 78.0M/478M [00:07<00:40, 10.3MB/s]
Downloading:  17%|#6        | 79.0M/478M [00:08<00:40, 10.5MB/s]
Downloading:  17%|#6        | 80.0M/478M [00:08<00:40, 10.4MB/s]
Downloading:  17%|#6        | 81.0M/478M [00:08<00:39, 10.4MB/s]
Downloading:  17%|#7        | 82.0M/478M [00:08<00:40, 10.3MB/s]
Downloading:  17%|#7        | 83.0M/478M [00:08<00:40, 10.3MB/s]
Downloading:  18%|#7        | 84.1M/478M [00:08<00:39, 10.5MB/s]
Downloading:  18%|#7        | 85.1M/478M [00:08<00:39, 10.4MB/s]
Downloading:  18%|#8        | 86.1M/478M [00:08<00:39, 10.3MB/s]
Downloading:  18%|#8        | 87.1M/478M [00:08<00:39, 10.5MB/s]
Downloading:  18%|#8        | 88.1M/478M [00:08<00:39, 10.3MB/s]
Downloading:  19%|#8        | 89.1M/478M [00:09<00:39, 10.4MB/s]
Downloading:  19%|#8        | 90.1M/478M [00:09<00:38, 10.5MB/s]
Downloading:  19%|#9        | 91.1M/478M [00:09<00:38, 10.4MB/s]
Downloading:  19%|#9        | 92.1M/478M [00:09<00:39, 10.3MB/s]
Downloading:  19%|#9        | 93.1M/478M [00:09<00:39, 10.3MB/s]
Downloading:  20%|#9        | 94.1M/478M [00:09<00:38, 10.4MB/s]
Downloading:  20%|#9        | 95.1M/478M [00:09<00:38, 10.3MB/s]
Downloading:  20%|##        | 96.1M/478M [00:09<00:39, 10.3MB/s]
Downloading:  20%|##        | 97.1M/478M [00:09<00:38, 10.4MB/s]
Downloading:  21%|##        | 98.2M/478M [00:09<00:38, 10.5MB/s]
Downloading:  21%|##        | 99.2M/478M [00:10<01:03, 6.27MB/s]
Downloading:  21%|##        | 100M/478M [00:10<01:00, 6.52MB/s] 
Downloading:  21%|##1       | 101M/478M [00:10<00:56, 7.02MB/s]
Downloading:  21%|##1       | 102M/478M [00:10<00:51, 7.65MB/s]
Downloading:  21%|##1       | 103M/478M [00:10<00:47, 8.28MB/s]
Downloading:  22%|##1       | 104M/478M [00:10<00:44, 8.85MB/s]
Downloading:  22%|##1       | 105M/478M [00:10<00:41, 9.42MB/s]
Downloading:  22%|##2       | 106M/478M [00:11<00:40, 9.64MB/s]
Downloading:  22%|##2       | 107M/478M [00:11<00:39, 9.88MB/s]
Downloading:  23%|##2       | 108M/478M [00:11<00:39, 9.95MB/s]
Downloading:  23%|##2       | 109M/478M [00:11<00:37, 10.2MB/s]
Downloading:  23%|##2       | 110M/478M [00:11<00:37, 10.3MB/s]
Downloading:  23%|##3       | 111M/478M [00:11<00:37, 10.2MB/s]
Downloading:  23%|##3       | 112M/478M [00:11<00:36, 10.4MB/s]
Downloading:  24%|##3       | 113M/478M [00:11<00:36, 10.4MB/s]
Downloading:  24%|##3       | 114M/478M [00:11<00:36, 10.4MB/s]
Downloading:  24%|##4       | 115M/478M [00:11<00:36, 10.4MB/s]
Downloading:  24%|##4       | 116M/478M [00:12<00:36, 10.3MB/s]
Downloading:  24%|##4       | 117M/478M [00:12<00:36, 10.4MB/s]
Downloading:  25%|##4       | 118M/478M [00:12<00:36, 10.4MB/s]
Downloading:  25%|##4       | 119M/478M [00:12<00:36, 10.4MB/s]
Downloading:  25%|##5       | 120M/478M [00:12<00:36, 10.3MB/s]
Downloading:  25%|##5       | 121M/478M [00:12<00:35, 10.4MB/s]
Downloading:  25%|##5       | 122M/478M [00:12<00:35, 10.4MB/s]
Downloading:  26%|##5       | 123M/478M [00:12<00:37, 10.0MB/s]
Downloading:  26%|##5       | 124M/478M [00:12<00:36, 10.3MB/s]
Downloading:  26%|##6       | 125M/478M [00:12<00:35, 10.3MB/s]
Downloading:  26%|##6       | 126M/478M [00:13<00:35, 10.3MB/s]
Downloading:  27%|##6       | 127M/478M [00:13<00:36, 10.2MB/s]
Downloading:  27%|##6       | 128M/478M [00:13<00:35, 10.2MB/s]
Downloading:  27%|##6       | 129M/478M [00:13<00:35, 10.2MB/s]
Downloading:  27%|##7       | 130M/478M [00:13<00:35, 10.3MB/s]
Downloading:  27%|##7       | 131M/478M [00:13<00:35, 10.3MB/s]
Downloading:  28%|##7       | 132M/478M [00:13<00:35, 10.3MB/s]
Downloading:  28%|##7       | 133M/478M [00:13<00:34, 10.4MB/s]
Downloading:  28%|##7       | 134M/478M [00:13<00:34, 10.3MB/s]
Downloading:  28%|##8       | 135M/478M [00:13<00:34, 10.4MB/s]
Downloading:  28%|##8       | 136M/478M [00:14<00:35, 10.2MB/s]
Downloading:  29%|##8       | 137M/478M [00:14<00:34, 10.4MB/s]
Downloading:  29%|##8       | 138M/478M [00:14<00:34, 10.4MB/s]
Downloading:  29%|##9       | 139M/478M [00:14<00:33, 10.5MB/s]
Downloading:  29%|##9       | 140M/478M [00:14<00:33, 10.5MB/s]
Downloading:  29%|##9       | 141M/478M [00:14<00:33, 10.5MB/s]
Downloading:  30%|##9       | 142M/478M [00:14<00:34, 10.2MB/s]
Downloading:  30%|##9       | 143M/478M [00:14<00:33, 10.5MB/s]
Downloading:  30%|###       | 144M/478M [00:14<00:33, 10.4MB/s]
Downloading:  30%|###       | 145M/478M [00:14<00:33, 10.5MB/s]
Downloading:  31%|###       | 146M/478M [00:15<00:33, 10.4MB/s]
Downloading:  31%|###       | 147M/478M [00:15<00:33, 10.5MB/s]
Downloading:  31%|###       | 148M/478M [00:15<00:33, 10.4MB/s]
Downloading:  31%|###1      | 149M/478M [00:15<00:34, 9.86MB/s]
Downloading:  31%|###1      | 150M/478M [00:15<00:32, 10.5MB/s]
Downloading:  32%|###1      | 151M/478M [00:15<00:32, 10.6MB/s]
Downloading:  32%|###1      | 152M/478M [00:15<00:32, 10.5MB/s]
Downloading:  32%|###2      | 153M/478M [00:15<00:32, 10.4MB/s]
Downloading:  32%|###2      | 154M/478M [00:15<00:32, 10.5MB/s]
Downloading:  32%|###2      | 155M/478M [00:16<00:32, 10.5MB/s]
Downloading:  33%|###2      | 156M/478M [00:16<00:32, 10.4MB/s]
Downloading:  33%|###2      | 157M/478M [00:16<00:32, 10.4MB/s]
Downloading:  33%|###3      | 158M/478M [00:16<00:32, 10.3MB/s]
Downloading:  33%|###3      | 159M/478M [00:16<00:31, 10.5MB/s]
Downloading:  33%|###3      | 160M/478M [00:16<00:31, 10.5MB/s]
Downloading:  34%|###3      | 161M/478M [00:16<00:31, 10.4MB/s]
Downloading:  34%|###3      | 162M/478M [00:16<00:31, 10.4MB/s]
Downloading:  34%|###4      | 163M/478M [00:16<00:31, 10.4MB/s]
Downloading:  34%|###4      | 164M/478M [00:16<00:31, 10.4MB/s]
Downloading:  35%|###4      | 165M/478M [00:17<00:31, 10.4MB/s]
Downloading:  35%|###4      | 166M/478M [00:17<00:31, 10.3MB/s]
Downloading:  35%|###4      | 167M/478M [00:17<00:31, 10.5MB/s]
Downloading:  35%|###5      | 168M/478M [00:17<00:31, 10.4MB/s]
Downloading:  35%|###5      | 169M/478M [00:17<00:31, 10.3MB/s]
Downloading:  36%|###5      | 170M/478M [00:17<00:31, 10.4MB/s]
Downloading:  36%|###5      | 171M/478M [00:17<00:30, 10.5MB/s]
Downloading:  36%|###6      | 172M/478M [00:17<00:31, 10.3MB/s]
Downloading:  36%|###6      | 173M/478M [00:17<00:30, 10.5MB/s]
Downloading:  36%|###6      | 174M/478M [00:17<00:30, 10.5MB/s]
Downloading:  37%|###6      | 175M/478M [00:18<00:30, 10.4MB/s]
Downloading:  37%|###6      | 176M/478M [00:18<00:30, 10.4MB/s]
Downloading:  37%|###7      | 177M/478M [00:18<00:30, 10.2MB/s]
Downloading:  37%|###7      | 178M/478M [00:18<00:30, 10.5MB/s]
Downloading:  37%|###7      | 179M/478M [00:18<00:30, 10.1MB/s]
Downloading:  38%|###7      | 180M/478M [00:18<00:29, 10.5MB/s]
Downloading:  38%|###7      | 181M/478M [00:18<00:29, 10.5MB/s]
Downloading:  38%|###8      | 182M/478M [00:18<00:29, 10.3MB/s]
Downloading:  38%|###8      | 183M/478M [00:18<00:29, 10.5MB/s]
Downloading:  39%|###8      | 184M/478M [00:18<00:29, 10.5MB/s]
Downloading:  39%|###8      | 185M/478M [00:19<00:29, 10.4MB/s]
Downloading:  39%|###8      | 186M/478M [00:19<00:29, 10.4MB/s]
Downloading:  39%|###9      | 187M/478M [00:19<00:29, 10.4MB/s]
Downloading:  39%|###9      | 188M/478M [00:19<00:29, 10.3MB/s]
Downloading:  40%|###9      | 189M/478M [00:19<00:29, 10.3MB/s]
Downloading:  40%|###9      | 190M/478M [00:19<00:28, 10.4MB/s]
Downloading:  40%|####      | 191M/478M [00:19<00:28, 10.5MB/s]
Downloading:  40%|####      | 192M/478M [00:19<00:28, 10.5MB/s]
Downloading:  40%|####      | 193M/478M [00:19<00:28, 10.4MB/s]
Downloading:  41%|####      | 194M/478M [00:19<00:29, 10.2MB/s]
Downloading:  41%|####      | 195M/478M [00:20<00:28, 10.4MB/s]
Downloading:  41%|####1     | 196M/478M [00:20<00:28, 10.3MB/s]
Downloading:  41%|####1     | 197M/478M [00:20<00:28, 10.5MB/s]
Downloading:  42%|####1     | 198M/478M [00:20<00:27, 10.5MB/s]
Downloading:  42%|####1     | 199M/478M [00:20<00:45, 6.44MB/s]
Downloading:  42%|####1     | 200M/478M [00:20<00:39, 7.36MB/s]
Downloading:  42%|####2     | 201M/478M [00:20<00:36, 8.04MB/s]
Downloading:  42%|####2     | 202M/478M [00:21<00:34, 8.47MB/s]
Downloading:  43%|####2     | 203M/478M [00:21<00:31, 9.09MB/s]
Downloading:  43%|####2     | 204M/478M [00:21<00:30, 9.46MB/s]
Downloading:  43%|####2     | 205M/478M [00:21<00:29, 9.74MB/s]
Downloading:  43%|####3     | 206M/478M [00:21<00:28, 9.87MB/s]
Downloading:  43%|####3     | 207M/478M [00:21<00:28, 9.94MB/s]
Downloading:  44%|####3     | 208M/478M [00:21<00:27, 10.2MB/s]
Downloading:  44%|####3     | 209M/478M [00:21<00:27, 10.3MB/s]
Downloading:  44%|####4     | 210M/478M [00:21<00:27, 10.3MB/s]
Downloading:  44%|####4     | 211M/478M [00:21<00:27, 10.3MB/s]
Downloading:  44%|####4     | 212M/478M [00:22<00:27, 10.2MB/s]
Downloading:  45%|####4     | 213M/478M [00:22<00:26, 10.4MB/s]
Downloading:  45%|####4     | 214M/478M [00:22<00:26, 10.4MB/s]
Downloading:  45%|####5     | 215M/478M [00:22<00:26, 10.4MB/s]
Downloading:  45%|####5     | 216M/478M [00:22<00:26, 10.4MB/s]
Downloading:  45%|####5     | 217M/478M [00:22<00:26, 10.3MB/s]
Downloading:  46%|####5     | 218M/478M [00:22<00:26, 10.2MB/s]
Downloading:  46%|####5     | 220M/478M [00:22<00:25, 10.5MB/s]
Downloading:  46%|####6     | 221M/478M [00:22<00:25, 10.5MB/s]
Downloading:  46%|####6     | 222M/478M [00:22<00:25, 10.5MB/s]
Downloading:  47%|####6     | 223M/478M [00:23<00:25, 10.4MB/s]
Downloading:  47%|####6     | 224M/478M [00:23<00:25, 10.3MB/s]
Downloading:  47%|####6     | 225M/478M [00:23<00:25, 10.5MB/s]
Downloading:  47%|####7     | 226M/478M [00:23<00:25, 10.4MB/s]
Downloading:  47%|####7     | 227M/478M [00:23<00:25, 10.3MB/s]
Downloading:  48%|####7     | 228M/478M [00:23<00:25, 10.5MB/s]
Downloading:  48%|####7     | 229M/478M [00:23<00:25, 10.5MB/s]
Downloading:  48%|####8     | 230M/478M [00:23<00:25, 10.4MB/s]
Downloading:  48%|####8     | 231M/478M [00:23<00:25, 10.3MB/s]
Downloading:  48%|####8     | 232M/478M [00:23<00:24, 10.4MB/s]
Downloading:  49%|####8     | 233M/478M [00:24<00:24, 10.4MB/s]
Downloading:  49%|####8     | 234M/478M [00:24<00:24, 10.4MB/s]
Downloading:  49%|####9     | 235M/478M [00:24<00:24, 10.4MB/s]
Downloading:  49%|####9     | 236M/478M [00:24<00:24, 10.4MB/s]
Downloading:  50%|####9     | 237M/478M [00:24<00:24, 10.4MB/s]
Downloading:  50%|####9     | 238M/478M [00:24<00:24, 10.3MB/s]
Downloading:  50%|####9     | 239M/478M [00:24<00:24, 10.3MB/s]
Downloading:  50%|#####     | 240M/478M [00:24<00:23, 10.5MB/s]
Downloading:  50%|#####     | 241M/478M [00:24<00:23, 10.5MB/s]
Downloading:  51%|#####     | 242M/478M [00:24<00:23, 10.4MB/s]
Downloading:  51%|#####     | 243M/478M [00:25<00:23, 10.3MB/s]
Downloading:  51%|#####     | 244M/478M [00:25<00:23, 10.3MB/s]
Downloading:  51%|#####1    | 245M/478M [00:25<00:23, 10.4MB/s]
Downloading:  51%|#####1    | 246M/478M [00:25<00:23, 10.3MB/s]
Downloading:  52%|#####1    | 247M/478M [00:25<00:23, 10.5MB/s]
Downloading:  52%|#####1    | 248M/478M [00:25<00:23, 10.4MB/s]
Downloading:  52%|#####2    | 249M/478M [00:25<00:22, 10.5MB/s]
Downloading:  52%|#####2    | 250M/478M [00:25<00:23, 10.3MB/s]
Downloading:  52%|#####2    | 251M/478M [00:25<00:22, 10.4MB/s]
Downloading:  53%|#####2    | 252M/478M [00:25<00:22, 10.3MB/s]
Downloading:  53%|#####2    | 253M/478M [00:26<00:22, 10.5MB/s]
Downloading:  53%|#####3    | 254M/478M [00:26<00:22, 10.5MB/s]
Downloading:  53%|#####3    | 255M/478M [00:26<00:22, 10.4MB/s]
Downloading:  54%|#####3    | 256M/478M [00:26<00:22, 10.5MB/s]
Downloading:  54%|#####3    | 257M/478M [00:26<00:22, 10.5MB/s]
Downloading:  54%|#####3    | 258M/478M [00:26<00:22, 10.5MB/s]
Downloading:  54%|#####4    | 259M/478M [00:26<00:22, 10.4MB/s]
Downloading:  54%|#####4    | 260M/478M [00:26<00:22, 10.3MB/s]
Downloading:  55%|#####4    | 261M/478M [00:26<00:21, 10.4MB/s]
Downloading:  55%|#####4    | 262M/478M [00:26<00:21, 10.3MB/s]
Downloading:  55%|#####4    | 263M/478M [00:27<00:21, 10.3MB/s]
Downloading:  55%|#####5    | 264M/478M [00:27<00:21, 10.5MB/s]
Downloading:  55%|#####5    | 265M/478M [00:27<00:21, 10.4MB/s]
Downloading:  56%|#####5    | 266M/478M [00:27<00:21, 10.5MB/s]
Downloading:  56%|#####5    | 267M/478M [00:27<00:21, 10.4MB/s]
Downloading:  56%|#####6    | 268M/478M [00:27<00:21, 10.4MB/s]
Downloading:  56%|#####6    | 269M/478M [00:27<00:20, 10.5MB/s]
Downloading:  56%|#####6    | 270M/478M [00:27<00:21, 10.3MB/s]
Downloading:  57%|#####6    | 271M/478M [00:27<00:20, 10.4MB/s]
Downloading:  57%|#####6    | 272M/478M [00:28<00:20, 10.4MB/s]
Downloading:  57%|#####7    | 273M/478M [00:28<00:20, 10.4MB/s]
Downloading:  57%|#####7    | 274M/478M [00:28<00:20, 10.5MB/s]
Downloading:  58%|#####7    | 275M/478M [00:28<00:20, 10.5MB/s]
Downloading:  58%|#####7    | 276M/478M [00:28<00:20, 10.3MB/s]
Downloading:  58%|#####7    | 277M/478M [00:28<00:20, 10.4MB/s]
Downloading:  58%|#####8    | 278M/478M [00:28<00:19, 10.5MB/s]
Downloading:  58%|#####8    | 279M/478M [00:28<00:19, 10.4MB/s]
Downloading:  59%|#####8    | 280M/478M [00:28<00:20, 10.3MB/s]
Downloading:  59%|#####8    | 281M/478M [00:28<00:19, 10.4MB/s]
Downloading:  59%|#####8    | 282M/478M [00:29<00:19, 10.5MB/s]
Downloading:  59%|#####9    | 283M/478M [00:29<00:19, 10.5MB/s]
Downloading:  59%|#####9    | 284M/478M [00:29<00:19, 10.3MB/s]
Downloading:  60%|#####9    | 285M/478M [00:29<00:19, 10.3MB/s]
Downloading:  60%|#####9    | 286M/478M [00:29<00:19, 10.6MB/s]
Downloading:  60%|######    | 287M/478M [00:29<00:19, 10.5MB/s]
Downloading:  60%|######    | 288M/478M [00:29<00:19, 10.5MB/s]
Downloading:  60%|######    | 289M/478M [00:29<00:18, 10.4MB/s]
Downloading:  61%|######    | 290M/478M [00:29<00:18, 10.4MB/s]
Downloading:  61%|######    | 291M/478M [00:29<00:18, 10.3MB/s]
Downloading:  61%|######1   | 292M/478M [00:30<00:19, 10.2MB/s]
Downloading:  61%|######1   | 293M/478M [00:30<00:18, 10.5MB/s]
Downloading:  62%|######1   | 294M/478M [00:30<00:18, 10.4MB/s]
Downloading:  62%|######1   | 295M/478M [00:30<00:18, 10.5MB/s]
Downloading:  62%|######1   | 296M/478M [00:30<00:18, 10.4MB/s]
Downloading:  62%|######2   | 297M/478M [00:30<00:18, 10.3MB/s]
Downloading:  62%|######2   | 298M/478M [00:30<00:18, 10.3MB/s]
Downloading:  63%|######2   | 299M/478M [00:30<00:18, 10.4MB/s]
Downloading:  63%|######2   | 300M/478M [00:30<00:17, 10.4MB/s]
Downloading:  63%|######3   | 301M/478M [00:30<00:17, 10.3MB/s]
Downloading:  63%|######3   | 302M/478M [00:31<00:17, 10.5MB/s]
Downloading:  63%|######3   | 303M/478M [00:31<00:17, 10.3MB/s]
Downloading:  64%|######3   | 304M/478M [00:31<00:17, 10.5MB/s]
Downloading:  64%|######3   | 305M/478M [00:31<00:17, 10.5MB/s]
Downloading:  64%|######4   | 306M/478M [00:31<00:17, 10.4MB/s]
Downloading:  64%|######4   | 307M/478M [00:31<00:17, 10.4MB/s]
Downloading:  64%|######4   | 308M/478M [00:31<00:17, 10.4MB/s]
Downloading:  65%|######4   | 309M/478M [00:31<00:17, 10.4MB/s]
Downloading:  65%|######4   | 310M/478M [00:31<00:16, 10.5MB/s]
Downloading:  65%|######5   | 311M/478M [00:31<00:16, 10.4MB/s]
Downloading:  65%|######5   | 312M/478M [00:32<00:16, 10.4MB/s]
Downloading:  66%|######5   | 313M/478M [00:32<00:16, 10.5MB/s]
Downloading:  66%|######5   | 314M/478M [00:32<00:16, 10.4MB/s]
Downloading:  66%|######5   | 315M/478M [00:32<00:16, 10.5MB/s]
Downloading:  66%|######6   | 316M/478M [00:32<00:16, 10.5MB/s]
Downloading:  66%|######6   | 317M/478M [00:32<00:16, 10.4MB/s]
Downloading:  67%|######6   | 318M/478M [00:32<00:16, 10.4MB/s]
Downloading:  67%|######6   | 319M/478M [00:32<00:15, 10.4MB/s]
Downloading:  67%|######7   | 320M/478M [00:32<00:15, 10.4MB/s]
Downloading:  67%|######7   | 321M/478M [00:32<00:15, 10.3MB/s]
Downloading:  67%|######7   | 322M/478M [00:33<00:15, 10.5MB/s]
Downloading:  68%|######7   | 323M/478M [00:33<00:15, 10.5MB/s]
Downloading:  68%|######7   | 324M/478M [00:33<00:15, 10.4MB/s]
Downloading:  68%|######8   | 325M/478M [00:33<00:15, 10.4MB/s]
Downloading:  68%|######8   | 326M/478M [00:33<00:15, 10.4MB/s]
Downloading:  68%|######8   | 327M/478M [00:33<00:15, 10.3MB/s]
Downloading:  69%|######8   | 328M/478M [00:33<00:15, 10.4MB/s]
Downloading:  69%|######8   | 329M/478M [00:33<00:14, 10.5MB/s]
Downloading:  69%|######9   | 330M/478M [00:33<00:15, 10.1MB/s]
Downloading:  69%|######9   | 331M/478M [00:33<00:15, 10.2MB/s]
Downloading:  70%|######9   | 332M/478M [00:34<00:14, 10.4MB/s]
Downloading:  70%|######9   | 333M/478M [00:34<00:14, 10.3MB/s]
Downloading:  70%|######9   | 334M/478M [00:34<00:14, 10.4MB/s]
Downloading:  70%|#######   | 335M/478M [00:34<00:14, 10.4MB/s]
Downloading:  70%|#######   | 336M/478M [00:34<00:14, 10.4MB/s]
Downloading:  71%|#######   | 337M/478M [00:34<00:14, 10.3MB/s]
Downloading:  71%|#######   | 338M/478M [00:34<00:13, 10.5MB/s]
Downloading:  71%|#######1  | 339M/478M [00:34<00:13, 10.5MB/s]
Downloading:  71%|#######1  | 340M/478M [00:34<00:13, 10.4MB/s]
Downloading:  71%|#######1  | 341M/478M [00:35<00:13, 10.3MB/s]
Downloading:  72%|#######1  | 342M/478M [00:35<00:13, 10.4MB/s]
Downloading:  72%|#######1  | 343M/478M [00:35<00:13, 10.4MB/s]
Downloading:  72%|#######2  | 344M/478M [00:35<00:13, 10.4MB/s]
Downloading:  72%|#######2  | 345M/478M [00:35<00:13, 10.4MB/s]
Downloading:  72%|#######2  | 346M/478M [00:35<00:13, 10.1MB/s]
Downloading:  73%|#######2  | 347M/478M [00:35<00:13, 10.4MB/s]
Downloading:  73%|#######2  | 348M/478M [00:35<00:13, 10.4MB/s]
Downloading:  73%|#######3  | 350M/478M [00:35<00:12, 10.6MB/s]
Downloading:  73%|#######3  | 351M/478M [00:35<00:12, 10.5MB/s]
Downloading:  74%|#######3  | 352M/478M [00:36<00:12, 10.3MB/s]
Downloading:  74%|#######3  | 353M/478M [00:36<00:12, 10.5MB/s]
Downloading:  74%|#######3  | 354M/478M [00:36<00:12, 10.5MB/s]
Downloading:  74%|#######4  | 355M/478M [00:36<00:12, 10.5MB/s]
Downloading:  74%|#######4  | 356M/478M [00:36<00:12, 10.4MB/s]
Downloading:  75%|#######4  | 357M/478M [00:36<00:12, 10.4MB/s]
Downloading:  75%|#######4  | 358M/478M [00:36<00:12, 10.4MB/s]
Downloading:  75%|#######5  | 359M/478M [00:36<00:12, 10.4MB/s]
Downloading:  75%|#######5  | 360M/478M [00:36<00:12, 10.3MB/s]
Downloading:  75%|#######5  | 361M/478M [00:36<00:11, 10.5MB/s]
Downloading:  76%|#######5  | 362M/478M [00:37<00:11, 10.3MB/s]
Downloading:  76%|#######5  | 363M/478M [00:37<00:11, 10.5MB/s]
Downloading:  76%|#######6  | 364M/478M [00:37<00:11, 10.5MB/s]
Downloading:  76%|#######6  | 365M/478M [00:37<00:11, 10.4MB/s]
Downloading:  76%|#######6  | 366M/478M [00:37<00:11, 10.4MB/s]
Downloading:  77%|#######6  | 367M/478M [00:37<00:11, 10.4MB/s]
Downloading:  77%|#######6  | 368M/478M [00:37<00:11, 10.4MB/s]
Downloading:  77%|#######7  | 369M/478M [00:37<00:10, 10.5MB/s]
Downloading:  77%|#######7  | 370M/478M [00:38<00:17, 6.43MB/s]
Downloading:  78%|#######7  | 371M/478M [00:38<00:15, 7.29MB/s]
Downloading:  78%|#######7  | 372M/478M [00:38<00:13, 8.08MB/s]
Downloading:  78%|#######7  | 373M/478M [00:38<00:12, 8.57MB/s]
Downloading:  78%|#######8  | 374M/478M [00:38<00:12, 9.09MB/s]
Downloading:  78%|#######8  | 375M/478M [00:38<00:11, 9.39MB/s]
Downloading:  79%|#######8  | 376M/478M [00:38<00:10, 9.76MB/s]
Downloading:  79%|#######8  | 377M/478M [00:38<00:10, 9.88MB/s]
Downloading:  79%|#######9  | 378M/478M [00:38<00:10, 10.1MB/s]
Downloading:  79%|#######9  | 379M/478M [00:38<00:10, 10.1MB/s]
Downloading:  79%|#######9  | 380M/478M [00:39<00:10, 10.3MB/s]
Downloading:  80%|#######9  | 381M/478M [00:39<00:09, 10.2MB/s]
Downloading:  80%|#######9  | 382M/478M [00:39<00:09, 10.4MB/s]
Downloading:  80%|########  | 383M/478M [00:39<00:09, 10.4MB/s]
Downloading:  80%|########  | 384M/478M [00:39<00:09, 10.4MB/s]
Downloading:  80%|########  | 385M/478M [00:39<00:09, 10.4MB/s]
Downloading:  81%|########  | 386M/478M [00:39<00:09, 10.5MB/s]
Downloading:  81%|########  | 387M/478M [00:39<00:09, 10.4MB/s]
Downloading:  81%|########1 | 388M/478M [00:39<00:09, 10.4MB/s]
Downloading:  81%|########1 | 389M/478M [00:39<00:09, 10.4MB/s]
Downloading:  82%|########1 | 390M/478M [00:40<00:09, 10.3MB/s]
Downloading:  82%|########1 | 391M/478M [00:40<00:08, 10.4MB/s]
Downloading:  82%|########1 | 392M/478M [00:40<00:08, 10.4MB/s]
Downloading:  82%|########2 | 393M/478M [00:40<00:08, 10.4MB/s]
Downloading:  82%|########2 | 394M/478M [00:40<00:08, 10.4MB/s]
Downloading:  83%|########2 | 395M/478M [00:40<00:08, 10.3MB/s]
Downloading:  83%|########2 | 396M/478M [00:40<00:08, 10.3MB/s]
Downloading:  83%|########3 | 397M/478M [00:40<00:08, 10.5MB/s]
Downloading:  83%|########3 | 398M/478M [00:40<00:08, 10.4MB/s]
Downloading:  83%|########3 | 399M/478M [00:40<00:07, 10.4MB/s]
Downloading:  84%|########3 | 400M/478M [00:41<00:07, 10.4MB/s]
Downloading:  84%|########3 | 401M/478M [00:41<00:07, 10.4MB/s]
Downloading:  84%|########4 | 402M/478M [00:41<00:07, 10.4MB/s]
Downloading:  84%|########4 | 403M/478M [00:41<00:07, 10.4MB/s]
Downloading:  84%|########4 | 404M/478M [00:41<00:07, 10.4MB/s]
Downloading:  85%|########4 | 405M/478M [00:41<00:07, 10.4MB/s]
Downloading:  85%|########4 | 406M/478M [00:41<00:07, 10.4MB/s]
Downloading:  85%|########5 | 407M/478M [00:41<00:07, 10.3MB/s]
Downloading:  85%|########5 | 408M/478M [00:41<00:07, 10.4MB/s]
Downloading:  86%|########5 | 409M/478M [00:42<00:06, 10.4MB/s]
Downloading:  86%|########5 | 410M/478M [00:42<00:06, 10.5MB/s]
Downloading:  86%|########5 | 411M/478M [00:42<00:06, 10.4MB/s]
Downloading:  86%|########6 | 412M/478M [00:42<00:06, 10.4MB/s]
Downloading:  86%|########6 | 413M/478M [00:42<00:06, 10.3MB/s]
Downloading:  87%|########6 | 414M/478M [00:42<00:06, 10.5MB/s]
Downloading:  87%|########6 | 415M/478M [00:42<00:06, 10.5MB/s]
Downloading:  87%|########6 | 416M/478M [00:42<00:06, 10.3MB/s]
Downloading:  87%|########7 | 417M/478M [00:42<00:06, 10.4MB/s]
Downloading:  87%|########7 | 418M/478M [00:42<00:06, 10.4MB/s]
Downloading:  88%|########7 | 419M/478M [00:43<00:05, 10.5MB/s]
Downloading:  88%|########7 | 420M/478M [00:43<00:05, 10.5MB/s]
Downloading:  88%|########8 | 421M/478M [00:43<00:05, 10.4MB/s]
Downloading:  88%|########8 | 422M/478M [00:43<00:05, 10.4MB/s]
Downloading:  88%|########8 | 423M/478M [00:43<00:05, 10.3MB/s]
Downloading:  89%|########8 | 424M/478M [00:43<00:05, 10.5MB/s]
Downloading:  89%|########8 | 425M/478M [00:43<00:05, 10.4MB/s]
Downloading:  89%|########9 | 426M/478M [00:43<00:05, 10.5MB/s]
Downloading:  89%|########9 | 427M/478M [00:43<00:05, 10.5MB/s]
Downloading:  90%|########9 | 428M/478M [00:43<00:05, 10.5MB/s]
Downloading:  90%|########9 | 429M/478M [00:44<00:04, 10.4MB/s]
Downloading:  90%|########9 | 430M/478M [00:44<00:04, 10.3MB/s]
Downloading:  90%|######### | 431M/478M [00:44<00:04, 10.3MB/s]
Downloading:  90%|######### | 432M/478M [00:44<00:04, 10.5MB/s]
Downloading:  91%|######### | 433M/478M [00:44<00:04, 10.4MB/s]
Downloading:  91%|######### | 434M/478M [00:44<00:04, 10.5MB/s]
Downloading:  91%|######### | 435M/478M [00:44<00:04, 10.4MB/s]
Downloading:  91%|#########1| 436M/478M [00:44<00:04, 10.4MB/s]
Downloading:  91%|#########1| 437M/478M [00:44<00:04, 10.4MB/s]
Downloading:  92%|#########1| 438M/478M [00:44<00:04, 10.5MB/s]
Downloading:  92%|#########1| 439M/478M [00:45<00:03, 10.4MB/s]
Downloading:  92%|#########2| 440M/478M [00:45<00:03, 10.4MB/s]
Downloading:  92%|#########2| 441M/478M [00:45<00:03, 10.4MB/s]
Downloading:  92%|#########2| 442M/478M [00:45<00:03, 10.4MB/s]
Downloading:  93%|#########2| 443M/478M [00:45<00:03, 10.4MB/s]
Downloading:  93%|#########2| 444M/478M [00:45<00:03, 10.4MB/s]
Downloading:  93%|#########3| 445M/478M [00:45<00:03, 10.4MB/s]
Downloading:  93%|#########3| 446M/478M [00:45<00:03, 10.4MB/s]
Downloading:  94%|#########3| 447M/478M [00:45<00:03, 10.3MB/s]
Downloading:  94%|#########3| 448M/478M [00:45<00:03, 10.3MB/s]
Downloading:  94%|#########3| 449M/478M [00:46<00:02, 10.5MB/s]
Downloading:  94%|#########4| 450M/478M [00:46<00:02, 10.4MB/s]
Downloading:  94%|#########4| 451M/478M [00:46<00:02, 10.5MB/s]
Downloading:  95%|#########4| 452M/478M [00:46<00:02, 10.4MB/s]
Downloading:  95%|#########4| 453M/478M [00:46<00:02, 10.4MB/s]
Downloading:  95%|#########4| 454M/478M [00:46<00:02, 10.3MB/s]
Downloading:  95%|#########5| 455M/478M [00:46<00:02, 10.4MB/s]
Downloading:  95%|#########5| 456M/478M [00:46<00:02, 10.3MB/s]
Downloading:  96%|#########5| 457M/478M [00:46<00:02, 10.5MB/s]
Downloading:  96%|#########5| 458M/478M [00:46<00:02, 10.4MB/s]
Downloading:  96%|#########6| 459M/478M [00:47<00:01, 10.3MB/s]
Downloading:  96%|#########6| 460M/478M [00:47<00:01, 10.4MB/s]
Downloading:  96%|#########6| 461M/478M [00:47<00:01, 10.4MB/s]
Downloading:  97%|#########6| 462M/478M [00:47<00:01, 10.4MB/s]
Downloading:  97%|#########6| 463M/478M [00:47<00:01, 10.3MB/s]
Downloading:  97%|#########7| 464M/478M [00:47<00:01, 10.4MB/s]
Downloading:  97%|#########7| 465M/478M [00:47<00:01, 10.4MB/s]
Downloading:  98%|#########7| 466M/478M [00:47<00:01, 10.3MB/s]
Downloading:  98%|#########7| 467M/478M [00:47<00:01, 10.3MB/s]
Downloading:  98%|#########7| 468M/478M [00:47<00:00, 10.5MB/s]
Downloading:  98%|#########8| 469M/478M [00:48<00:00, 10.5MB/s]
Downloading:  98%|#########8| 470M/478M [00:48<00:01, 6.37MB/s]
Downloading:  99%|#########8| 471M/478M [00:48<00:00, 7.31MB/s]
Downloading:  99%|#########8| 472M/478M [00:48<00:00, 7.92MB/s]
Downloading:  99%|#########8| 473M/478M [00:48<00:00, 8.61MB/s]
Downloading:  99%|#########9| 474M/478M [00:48<00:00, 9.04MB/s]
Downloading:  99%|#########9| 475M/478M [00:48<00:00, 9.38MB/s]
Downloading: 100%|#########9| 476M/478M [00:49<00:00, 9.75MB/s]
Downloading: 100%|#########9| 477M/478M [00:49<00:00, 9.93MB/s]
Downloading: 100%|##########| 478M/478M [00:49<00:00, 10.2MB/s]
Some weights of the model checkpoint at roberta-base were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.bias', 'lm_head.decoder.weight', 'lm_head.layer_norm.bias', 'lm_head.dense.bias', 'roberta.pooler.dense.weight', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.weight']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at roberta-base and are newly initialized: ['classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.dense.weight', 'classifier.out_proj.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Downloading:   0%|          | 0.00/878k [00:00<?, ?B/s]
Downloading:   5%|5         | 44.0k/878k [00:00<00:01, 450kB/s]
Downloading:  21%|##        | 180k/878k [00:00<00:00, 988kB/s] 
Downloading:  90%|########9 | 788k/878k [00:00<00:00, 3.29MB/s]
Downloading: 100%|##########| 878k/878k [00:00<00:00, 2.84MB/s]

Downloading:   0%|          | 0.00/446k [00:00<?, ?B/s]
Downloading:   6%|6         | 28.0k/446k [00:00<00:01, 283kB/s]
Downloading:  40%|###9      | 178k/446k [00:00<00:00, 945kB/s] 
Downloading: 100%|##########| 446k/446k [00:00<00:00, 1.65MB/s]

Downloading:   0%|          | 0.00/1.29M [00:00<?, ?B/s]
Downloading:   3%|3         | 40.0k/1.29M [00:00<00:03, 408kB/s]
Downloading:  13%|#2        | 172k/1.29M [00:00<00:01, 945kB/s] 
Downloading:  56%|#####6    | 748k/1.29M [00:00<00:00, 3.10MB/s]
Downloading: 100%|##########| 1.29M/1.29M [00:00<00:00, 3.73MB/s]
#'user/model
user> (def x (py/call-attr model "train_model" train-df))
/home/chrisn/miniconda3/lib/python3.9/site-packages/simpletransformers/classification/classification_model.py:585: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.
  warnings.warn(

  0%|          | 0/2 [00:00<?, ?it/s]
 50%|#####     | 1/2 [00:00<00:00,  4.96it/s]
 50%|#####     | 1/2 [00:00<00:00,  4.94it/s]
/home/chrisn/miniconda3/lib/python3.9/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning
  warnings.warn(
Epoch:   0%|          | 0/1 [00:00<?, ?it/s]
Epoch 1 of 1:   0%|          | 0/1 [00:00<?, ?it/s]
Running Epoch 0 of 1:   0%|          | 0/1 [00:00<?, ?it/s]
Epochs 0/1. Running Loss:    0.6930:   0%|          | 0/1 [00:00<?, ?it/s]
Epochs 0/1. Running Loss:    0.6930: 100%|##########| 1/1 [00:01<00:00,  1.68s/it]
Epochs 0/1. Running Loss:    0.6930: 100%|##########| 1/1 [00:01<00:00,  1.69s/it]

Epoch 1 of 1: 100%|##########| 1/1 [00:03<00:00,  3.96s/it]
Epoch 1 of 1: 100%|##########| 1/1 [00:03<00:00,  3.96s/it]
#'user/x
user> (println x)
(1, 0.6929999589920044)
nil
user> (println "finished train")
finished train
nil
user>

@cnuernber
Copy link
Collaborator

Here is the deps.edn file with the jvm opts I used:

{:paths ["src" "resources"]

 :deps {org.clojure/clojure {:mvn/version "1.10.3"}
        clj-python/libpython-clj {:mvn/version "2.018"}}
 :aliases
 {:manual-gil {:jvm-opts ["-Dlibpython_clj.manual_gil=true"]}}}

@behrica
Copy link
Contributor Author

behrica commented Mar 23, 2022 •
edited
Loading

Looks like good progress.
I do not understand enough on python and the GIL, in order to use this correctly, I fear.
I do support your hypothesis of race condition somewhere, as the problem disappears the moment I run the clj file via the clj CLI and not via Cider nrepl.

@behrica
Copy link
Contributor Author

behrica commented Mar 23, 2022

I can confirm that the above code fixes the issue, but I suppose that it is a workaround, is it ?

@behrica
Copy link
Contributor Author

behrica commented Mar 23, 2022

Just for information.
Playing around with the lockGIL I get soetims an python exception:

Traceback (most recent call last):
  File "/opt/conda/envs/st/lib/python3.9/threading.py", line 1486, in _after_fork
    thread._reset_internal_locks(True)
  File "/opt/conda/envs/st/lib/python3.9/threading.py", line 829, in _reset_internal_locks
    self._tstate_lock._at_fork_reinit()
AttributeError: 'NoneType' object has no attribute '_at_fork_reinit'
Exception ignored in: <function _after_fork at 0x7faf5aaf0790>
Traceback (most recent call last):
  File "/opt/conda/envs/st/lib/python3.9/threading.py", line 1486, in _after_fork
    thread._reset_internal_locks(True)
  File "/opt/conda/envs/st/lib/python3.9/threading.py", line 829, in _reset_internal_locks
    self._tstate_lock._at_fork_reinit()
AttributeError: 'NoneType' object has no attribute '_at_fork_reinit'
Exception ignored in: <function _after_fork at 0x7faf5aaf0790>
Traceback (most recent call last):
  File "/opt/conda/envs/st/lib/python3.9/threading.py", line 1486, in _after_fork
    thread._reset_internal_locks(True)
  File "/opt/conda/envs/st/lib/python3.9/threading.py", line 829, in _reset_internal_locks

but code keeps working, so training finshes.

@cnuernber
Copy link
Collaborator

That is fascinating. I will follow up on this soon.

@behrica
Copy link
Contributor Author

behrica commented Sep 4, 2022

Anything new on this ?
I have a reproducible crash (using automatic gil management) and
a working version with manual GIL management using the Java API ?

1 similar comment
@behrica
Copy link
Contributor Author

behrica commented Sep 4, 2022

Anything new on this ?
I have a reproducible crash (using automatic gil management) and
a working version with manual GIL management using the Java API ?

@behrica
Copy link
Contributor Author

behrica commented Sep 4, 2022

I can still not use simpletransformer library with libpython-clj.

@jjtolton
Copy link
Contributor

jjtolton commented Sep 4, 2022

I'll try to take a look Monday, out of town today. I've used simpletransformers a few times so I'll see if I can get a working setup.

@behrica
Copy link
Contributor Author

behrica commented Sep 5, 2022

Just add, it hsppens in REPL only. Working in CLJ file

@jjtolton
Copy link
Contributor

jjtolton commented Sep 6, 2022

I wasn't able to make much progress with this, unfortunately. Very strange. Glad it is working in clj file.

@behrica
Copy link
Contributor Author

behrica commented Sep 6, 2022

I think @cnuernber was investigating this to a certain level, see comments before.

He suspected "I think there is a race condition in libpython there."

I was hoping that the fcat of having it working with java helps to solve it, as apparnetly the clojure API of libpython-clj does something "wrong" / different.

Not being able to use it from REPL, is of course a bit of a blocker.

@behrica
Copy link
Contributor Author

behrica commented Sep 6, 2022

This branch contains the working code using the java API and disable automatic GIL handling via properties:
https://github.com/behrica/libpython-clj--194/tree/withManualGil

@behrica
Copy link
Contributor Author

behrica commented Sep 6, 2022

I have now found by random permutation of the code a (nearly) working clj file.
behrica/libpython-clj--194@2308c84

This file can be loaded one time successfully in a connected repl via the instructions here:
https://github.com/behrica/libpython-clj--194/blob/main/README.md

But repeating step 3. in the same repl, makes it crash again.

@behrica
Copy link
Contributor Author

behrica commented Sep 6, 2022

The code is very "sensitive" about where the "require" statements are.
Replacing for example the ':require' in the ns declaration by the "require" macro makes it crash again.

@behrica
Copy link
Contributor Author

behrica commented Sep 6, 2022

It works as well to execetute line by line in a repl from Emacs.
But only one-time.
Re-execute the "(py/with-gil-stack-rc-context ..." makes it crash again.

@behrica
Copy link
Contributor Author

behrica commented Sep 6, 2022

I think the crash is related to "where" I load the python modules.
If I load them outside the with-... it crashes:

(py/import-as "pandas" pd)
(py/import-as "simpletransformers.classification" classification)
(py/with-gil-stack-rc-context .....
...
do something with teh module

If I load them inside it works, but only ones:

(py/with-gil-stack-rc-context ;; without this it crashes

 (py/import-as "pandas" pd)
 (py/import-as "simpletransformers.classification" classification)

..... do something wit the module

@behrica
Copy link
Contributor Author

behrica commented Sep 6, 2022

Just a detail:

it is not the training of the model which crashes, bu the construcor of teh model (which lodas the model files from internet)
So this line is not even needed:
x (py. model train_model train-df)]

@jjtolton
Copy link
Contributor

jjtolton commented Sep 6, 2022 via email

@behrica
Copy link
Contributor Author

behrica commented Sep 6, 2022

I have it finally fully working.

@behrica
Copy link
Contributor Author

behrica commented Sep 6, 2022

Just pushed it to
behrica/libpython-clj--194@870124c

@behrica
Copy link
Contributor Author

behrica commented Sep 6, 2022

The usage of "require-python" instead of the other function from the ns libpython-clj2.python to load a module, such as_

  • from-import
  • import-module
  • import-as

made it finally work.

@behrica
Copy link
Contributor Author

behrica commented Sep 6, 2022 •
edited
Loading

So it looks to me that only "require-python" behaves well, especially on repeated executions in the same repl instance.

While the other three ways to load a module:

  • libpython-clj2.python/from-import
  • libpython-clj2.python/import-module
  • libpython-clj2.python/import-as

"leak" something or similar.

@jjtolton
Copy link
Contributor

jjtolton commented Sep 6, 2022

That's really interesting and surprising. But now that I think about it, this makes sense -- one of the differences is that require-python uses the Python runtime to load the modules via Python's importlib whereas the others use lower level (I believe) ABIs to load the code. I am guessing that the Python ML code is also using low level ABIs and C code.

@jjtolton
Copy link
Contributor

jjtolton commented Sep 6, 2022

We should probably document advising new users to steer away from the lower level functions and use the methods in the require namespace unless they have a specific reason to do otherwise.

@behrica
Copy link
Contributor Author

behrica commented Sep 6, 2022

I verified again, and indeed all the issue starts the moment I switch from this syntax:

working

(py-require/require-python '[simpletransformers.classification
                              :as classification])
(classification/ClassificationModel

             :use_cuda false
             :model_type "bert"
             :model_name "prajjwal1/bert-tiny"
             :args
             (py/->py-dict
              {:num_train_epochs 1
               :use_multiprocessing false
               :overwrite_output_dir true}))

to

crashing

(py/import-as "simpletransformers.classification" classification)
  (py/py. classification ClassificationModel

             :use_cuda false
             :model_type "bert"
             :model_name "prajjwal1/bert-tiny"
             :args
             (py/->py-dict
              {:num_train_epochs 1
               :use_multiprocessing false
               :overwrite_output_dir true}))

@cnuernber
Copy link
Collaborator

@jjtolton - The manual gil branch works reliably for me. IS that the case for you?

@cnuernber
Copy link
Collaborator

In fact I can train repeatedly and things are fine aside from the at_fork_reinit logging issues noted above:

Exception ignored in: <function _after_fork at 0x7f0173b02790>
Traceback (most recent call last):
  File "/opt/conda/envs/st/lib/python3.9/threading.py", line 1486, in _after_fork
    thread._reset_internal_locks(True)
  File "/opt/conda/envs/st/lib/python3.9/threading.py", line 829, in _reset_internal_locks
    self._tstate_lock._at_fork_reinit()
AttributeError: 'NoneType' object has no attribute '_at_fork_reinit'
Exception ignored in: <function _after_fork at 0x7f0173b02790>
Traceback (most recent call last):
  File "/opt/conda/envs/st/lib/python3.9/threading.py", line 1486, in _after_fork
    thread._reset_internal_locks(True)
  File "/opt/conda/envs/st/lib/python3.9/threading.py", line 829, in _reset_internal_locks
    self._tstate_lock._at_fork_reinit()
AttributeError: 'NoneType' object has no attribute '_at_fork_reinit'
Exception ignored in: <function _after_fork at 0x7f0173b02790>
Traceback (most recent call last):
  File "/opt/conda/envs/st/lib/python3.9/threading.py", line 1486, in _after_fork
    thread._reset_internal_locks(True)
  File "/opt/conda/envs/st/lib/python3.9/threading.py", line 829, in _reset_internal_locks
    self._tstate_lock._at_fork_reinit()
AttributeError: 'NoneType' object has no attribute '_at_fork_reinit'
Exception ignored in: <function _after_fork at 0x7f0173b02790>
Traceback (most recent call last):
  File "/opt/conda/envs/st/lib/python3.9/threading.py", line 1486, in _after_fork
    thread._reset_internal_locks(True)
  File "/opt/conda/envs/st/lib/python3.9/threading.py", line 829, in _reset_internal_locks
    self._tstate_lock._at_fork_reinit()
AttributeError: 'NoneType' object has no attribute '_at_fork_reinit'
Exception ignored in: <function _after_fork at 0x7f0173b02790>
Traceback (most recent call last):
  File "/opt/conda/envs/st/lib/python3.9/threading.py", line 1486, in _after_fork
    thread._reset_internal_locks(True)
  File "/opt/conda/envs/st/lib/python3.9/threading.py", line 829, in _reset_internal_locks
    self._tstate_lock._at_fork_reinit()
AttributeError: 'NoneType' object has no attribute '_at_fork_reinit'
Exception ignored in: <function _after_fork at 0x7f0173b02790>
Traceback (most recent call last):
  File "/opt/conda/envs/st/lib/python3.9/threading.py", line 1486, in _after_fork
    thread._reset_internal_locks(True)
  File "/opt/conda/envs/st/lib/python3.9/threading.py", line 829, in _reset_internal_locks
    self._tstate_lock._at_fork_reinit()
AttributeError: 'NoneType' object has no attribute '_at_fork_reinit'
Exception ignored in: <function _after_fork at 0x7f0173b02790>
Traceback (most recent call last):
  File "/opt/conda/envs/st/lib/python3.9/threading.py", line 1486, in _after_fork
    thread._reset_internal_locks(True)
  File "/opt/conda/envs/st/lib/python3.9/threading.py", line 829, in _reset_internal_locks
    self._tstate_lock._at_fork_reinit()
AttributeError: 'NoneType' object has no attribute '_at_fork_reinit'

I googled this and found some issues but I can't tell if the fix for those are in python 3.9 or not. Also it does appear like the multithreading module is in use which is extremely suspect.

@cnuernber
Copy link
Collaborator

cnuernber commented Sep 23, 2022 •
edited
Loading

@behrica - that branch, manual gil, should not work. The manual gil system variable is a compile time variable for libpython-clj. This means that your with-properties macro body should not work at all - check-gil should be being called all the time regardless. It is utterly confusing why this is repeatedly working from the REPL in this branch for me. Perhaps just chance.

-- and as I said that it crashed --

@cnuernber
Copy link
Collaborator

Getting the manual-gil setting correctly work from my normal repl makes everything work so far.

@cnuernber
Copy link
Collaborator

We should really see if this is robust across the board. We need to figure out how to tell javabridge to start the jvm with a system property set. You can see if manual gil management is enabled by checking the variable libpython-clj2.python.ffi/manual-gil. It wil lbe set to true if the library is compiled with manual gil management.

For some libraries it may just be the case that the automatic check-gil/capture-gil pathway that we use is just not or stable; there could be a race condition there which seems incredible to me but I can't think of another reason why this would be so crazy and infuriating.

@behrica
Copy link
Contributor Author

behrica commented Sep 24, 2022

Getting the manual-gil setting correctly work from my normal repl makes everything work so far.

I do agree, that by using manual gil management in the repl, and being "very careful", things work.
I first though that "repeated training is a problem", but it is not the case.

So I can interactively use simple transformer, which is nice.

@behrica
Copy link
Contributor Author

behrica commented Sep 24, 2022

@jjtolton - The manual gil branch works reliably for me. IS that the case for you?

Only "one time".

If in the same repl sessions you call (load-file "simpletransformers.clj") twice, I get a crash.

But this is not due to "train twice", it is somethiong in the

(java_api/initialize nil)
(java_api/lockGIL)
(require '[libpython-clj2.require :as py-req]
         '[libpython-clj2.python.io-redirect]
         '[libpython-clj2.python :refer [py.- py.] :as py])

(py/from-import simpletransformers.classification ClassificationModel)
(py-req/require-python '[pandas :as pd])

;;do train

(java_api/unlockGIL 1)

which does not like to be called "twice" in the same repl session.

@behrica
Copy link
Contributor Author

behrica commented Sep 24, 2022

We should really see if this is robust across the board. We need to figure out how to tell javabridge to start the jvm with a system property set.

One workaround is to use this in the Docker or outside:
ENV _JAVA_OPTIONS "-Dlibpython_clj.manual_gil=true"

The javabridge JVM picks it up

@behrica
Copy link
Contributor Author

behrica commented Sep 24, 2022

Getting the manual-gil setting correctly work from my normal repl makes everything work so far.

I do agree, that by using manual gil management in the repl, and being "very careful", things work. I first though that "repeated training is a problem", but it is not the case.

So I can interactively use simple transformer, which is nice.

But indeed, only with "manual GIL management".

With automatic GIL management , i can no get it working interactively.

With manual GIL management and being carefull (which means not to repeat the various imports and lock the GIL at least ones)

-> We need to pay attention on both (call lock-gil and not repeat the imports / requires)

Even with manual GIL management, I could bring the sessions in a "state" where the training would crash (by repeating the imports / requires, for example)

@behrica
Copy link
Contributor Author

behrica commented Sep 24, 2022 •
edited
Loading

In a certain way this is satisfactory to me.
Maybe we should document somewhere, that some python libraries work "better" interactively with manual GIL management and avoid to repeat "require-python and similar imports".

@behrica
Copy link
Contributor Author

behrica commented Sep 24, 2022

My current code mixes in some form as well the "java API" and the "clojure API".
Can we uniform this ? So that Clojure supports as well fully the "manual GIL" management ?

@cnuernber
Copy link
Collaborator

Yes of course - the java api is in fact clojure and really just calls down into the clojure code.

The other python bindings from java lock python access to a specific thread. I state this in various places; it isn't helpful for REPL behavior really but if you have to run this in a production context then I think that is reasonable.

@cnuernber
Copy link
Collaborator

Hmm - I think there are a few parts to fix this

  1. The first is clear documentation - I can write a new topic on exactly this.

  2. Could we get a smaller program that always crashes regardless? If you just put (py/import-module "simpletransformers.classification") in a loop does that crash even with manual gil management? What about if (System/gc) followed by (gc/clear-reference-queue) to force module dereferences?

  3. Another direction - could we make sure that require-python is a noop if it has already run and :reload isn't specified? I mean, isn't it already?

Perhaps libpython-clj itself in import-module should track modules and avoid releasing/reimporting them? OR perhaps the reference counting on import-module isn't correct and thus the simpletransformers module is getting completely released which may be something that would just never happen in normal python code.

@behrica
Copy link
Contributor Author

behrica commented Sep 24, 2022

Regarding 2)
One of the issues is that lib-pythonclj behaves differently regarding this, depending on "how and where" you call it from.

  • clj file
  • repl
  • embeded libpython-clj with cljbridge.load_clojure_file(clj_file="simpletransformers.clj")
  • embeded libpython-clj via connected nrepl

@behrica
Copy link
Contributor Author

behrica commented Sep 24, 2022

I have now a "Deadlock" in the "train call", with larger data.
Standard Clojure libpython-clj

Reproducible with my data, every time running as clj src/train.clj

@cnuernber
Copy link
Collaborator

cnuernber commented Sep 24, 2022 •
edited
Loading

With manual gil?

Most likely they are kicking off the multithreading module which is forking the process which is failing.

@behrica
Copy link
Contributor Author

behrica commented Sep 24, 2022

Maybe we are getting somewhere...
This crashes reliably, without manual GIL.

(ns train
  (:require [libpython-clj2.python :refer [py.- py.] :as py]
            [libpython-clj2.require :refer [require-python]]
            [libpython-clj2.python.gc]))


           

(require-python '[simpletransformers.classification
                  :as classification])

(System/gc)
(libpython-clj2.python.gc/clear-reference-queue)

@behrica
Copy link
Contributor Author

behrica commented Sep 24, 2022

Depending if called in repl or command line, it sometimes hangs or crashes, but never runs through.

@cnuernber
Copy link
Collaborator

So there is another variable - object-reference-logging in the ffi namespace. Setting it to true should allow you to see the last object freed before the crash.

The require-python statement really loads a ton of objects and all of these then get released. My guess is if you did (py/import-module "simpletransformers.classification") that would not crash.

@behrica
Copy link
Contributor Author

behrica commented Sep 24, 2022

Changing to import-module `(def classification (py/import-module "simpletransformers.classification"))
gives an exception:

21:24:06.111 [main] ERROR libpython-clj2.python.ffi - Exception while releasing object
java.lang.IllegalStateException: Recursive update
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1739)
	at libpython_clj2.python.ffi$pyobject_type_kwd.invokeStatic(ffi.clj:810)
	at libpython_clj2.python.ffi$pyobject_type_kwd.invoke(ffi.clj:807)
	at libpython_clj2.python.ffi$wrap_obj_ptr$fn__8977.invoke(ffi.clj:845)
	at libpython_clj2.python.gc$track$fn__8386.invoke(gc.clj:37)
	at tech.resource.GCReference.run(GCReference.java:25)
	at libpython_clj2.python.gc$clear_reference_queue.invokeStatic(gc.clj:52)
	at libpython_clj2.python.gc$clear_reference_queue.invoke(gc.clj:49)
	at train$eval14435.invokeStatic(train.clj:11)
	at train$eval14435.invoke(train.clj:11)
	at clojure.lang.Compiler.eval(Compiler.java:7181)
	at clojure.lang.Compiler.load(Compiler.java:7640)
	at clojure.lang.Compiler.loadFile(Compiler.java:7578)
	at clojure.main$load_script.invokeStatic(main.clj:475)
	at clojure.main$script_opt.invokeStatic(main.clj:535)
	at clojure.main$script_opt.invoke(main.clj:530)
	at clojure.main$main.invokeStatic(main.clj:664)
	at clojure.main$main.doInvoke(main.clj:616)
	at clojure.lang.RestFn.applyTo(RestFn.java:137)
	at clojure.lang.Var.applyTo(Var.java:705)
	at clojure.main.main(main.java:40)

but, it continues....

`

@cnuernber
Copy link
Collaborator

huh...what was released? IF the classification variable is bound then that cannot be what is released...I don't think this is the root of the issue but it is certainly a serious issue.

@behrica
Copy link
Contributor Author

behrica commented Sep 24, 2022 

1:30:16.323 [main] INFO libpython-clj2.python.ffi - tracking object  - 0x7f36140ccdc0:3573:dict
21:30:16.323 [main] INFO libpython-clj2.python.ffi - tracking object  - 0x7f34fb76bd90:   6:type
21:30:16.323 [main] INFO libpython-clj2.python.ffi - tracking object  - 0x7f34e31bd3b0:   4:module
21:30:16.324 [main] INFO libpython-clj2.python.ffi - tracking object  - 0x7f34da3c2c70:   4:module
21:30:16.324 [main] INFO libpython-clj2.python.ffi - tracking object  - 0x7f34f57d9090:   4:module
21:30:16.379 [main] INFO libpython-clj2.python.ffi - releasing object - 0x7f34da3e8c10:   1::method-wrapper
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f35ae533f00, pid=158138, tid=158148
#
# JRE version: OpenJDK Runtime Environment (11.0.16.1+1) (build 11.0.16.1+1)
# Java VM: OpenJDK 64-Bit Server VM (11.0.16.1+1, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [libpython3.9.so+0x133f00]  _PyTrash_begin+0x0
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" (or dumping to /home/carsten/Dropbox/sources/predict_disaster_tweet/core.158138)
#
# An error report file with more information is saved as:
# /home/carsten/Dropbox/sources/predict_disaster_tweet/hs_err_pid158138.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

@cnuernber
Copy link
Collaborator

I think the gil has to captured for clear-reference-queue. I don't think that is a method that captures the gil automatically.

@behrica
Copy link
Contributor Author

behrica commented Sep 24, 2022

yes, I did that.
Then the above code (without training), runs.

@behrica
Copy link
Contributor Author

behrica commented Sep 24, 2022

Once I add the training to id, I get again all kind of issues, mainly hanging this time, not crashing.

@behrica
Copy link
Contributor Author

behrica commented Sep 24, 2022

threaddump-1664051627417.tdump.gz

@behrica
Copy link
Contributor Author

behrica commented Sep 24, 2022 •
edited
Loading

I attached a thread dump while it is hanging.

@behrica behrica mentioned this issue Sep 25, 2022
Closed
@behrica
Copy link
Contributor Author

behrica commented Sep 26, 2022

I propose to close this issue.
The "crashing" is solved by using libpython-clj
with:

  1. embedded mode
  2. manual GIL management
  3. configure python library to not use multiprocessing extensively
    (multiprocessing makes apparently as well problems in certain cpython runtimes such as Colab))

I keep here a discussion #219 about a very occasionally blocking and refer to some example code which is 99% working, as reference.

@behrica behrica closed this as completed Sep 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@cnuernber @behrica @jjtolton