Skip to content

Commit 2322d22

Browse files
raviskolliraviskolli
authored
Merge pull request huggingface#5 from microsoft/raviskolli/ort
Raviskolli/ort
2 parents bce3290 + 47e69d4 commit 2322d22

File tree

511 files changed

+39028
-9407
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

511 files changed

+39028
-9407
lines changed

‎.circleci/config.yml

Lines changed: 64 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,6 @@ orbs:
33
gcp-gke: circleci/[email protected]
44
go: circleci/[email protected]
55

6-
76
# TPU REFERENCES
87
references:
98
checkout_ml_testing: &checkout_ml_testing
@@ -69,6 +68,8 @@ jobs:
6968
- image: circleci/python:3.6
7069
environment:
7170
OMP_NUM_THREADS: 1
71+
RUN_PT_TF_CROSS_TESTS: yes
72+
TRANSFORMERS_IS_CI: yes
7273
resource_class: xlarge
7374
parallelism: 1
7475
steps:
@@ -79,13 +80,43 @@ jobs:
7980
- v0.4-{{ checksum "setup.py" }}
8081
- run: sudo apt-get -y update && sudo apt-get install -y libsndfile1-dev
8182
- run: pip install --upgrade pip
82-
- run: pip install .[sklearn,tf-cpu,torch,testing,sentencepiece,speech]
83-
- run: pip install tapas torch-scatter -f https://pytorch-geometric.com/whl/torch-1.8.0+cpu.html
83+
- run: pip install .[sklearn,tf-cpu,torch,testing,sentencepiece,speech,vision]
84+
- run: pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.8.0+cpu.html
85+
- save_cache:
86+
key: v0.4-{{ checksum "setup.py" }}
87+
paths:
88+
- '~/.cache/pip'
89+
- run: python -m pytest -n 8 --dist=loadfile -rA -s --make-reports=tests_torch_and_tf ./tests/ -m is_pt_tf_cross_test --durations=0 | tee tests_output.txt
90+
- store_artifacts:
91+
path: ~/transformers/tests_output.txt
92+
- store_artifacts:
93+
path: ~/transformers/reports
94+
95+
run_tests_torch_and_flax:
96+
working_directory: ~/transformers
97+
docker:
98+
- image: circleci/python:3.6
99+
environment:
100+
OMP_NUM_THREADS: 1
101+
RUN_PT_FLAX_CROSS_TESTS: yes
102+
TRANSFORMERS_IS_CI: yes
103+
resource_class: xlarge
104+
parallelism: 1
105+
steps:
106+
- checkout
107+
- restore_cache:
108+
keys:
109+
- v0.4-torch_and_flax-{{ checksum "setup.py" }}
110+
- v0.4-{{ checksum "setup.py" }}
111+
- run: sudo apt-get -y update && sudo apt-get install -y libsndfile1-dev
112+
- run: pip install --upgrade pip
113+
- run: pip install .[sklearn,flax,torch,testing,sentencepiece,speech,vision]
114+
- run: pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.8.0+cpu.html
84115
- save_cache:
85116
key: v0.4-{{ checksum "setup.py" }}
86117
paths:
87118
- '~/.cache/pip'
88-
- run: RUN_PT_TF_CROSS_TESTS=1 python -m pytest -n 8 --dist=loadfile -rA -s --make-reports=tests_torch_and_tf ./tests/ -m is_pt_tf_cross_test --durations=0 | tee tests_output.txt
119+
- run: python -m pytest -n 8 --dist=loadfile -rA -s --make-reports=tests_torch_and_flax ./tests/ -m is_pt_flax_cross_test --durations=0 | tee tests_output.txt
89120
- store_artifacts:
90121
path: ~/transformers/tests_output.txt
91122
- store_artifacts:
@@ -97,6 +128,7 @@ jobs:
97128
- image: circleci/python:3.7
98129
environment:
99130
OMP_NUM_THREADS: 1
131+
TRANSFORMERS_IS_CI: yes
100132
resource_class: xlarge
101133
parallelism: 1
102134
steps:
@@ -107,13 +139,13 @@ jobs:
107139
- v0.4-{{ checksum "setup.py" }}
108140
- run: sudo apt-get -y update && sudo apt-get install -y libsndfile1-dev
109141
- run: pip install --upgrade pip
110-
- run: pip install .[sklearn,torch,testing,sentencepiece,speech]
111-
- run: pip install tapas torch-scatter -f https://pytorch-geometric.com/whl/torch-1.8.0+cpu.html
142+
- run: pip install .[sklearn,torch,testing,sentencepiece,speech,vision]
143+
- run: pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.8.0+cpu.html
112144
- save_cache:
113145
key: v0.4-torch-{{ checksum "setup.py" }}
114146
paths:
115147
- '~/.cache/pip'
116-
- run: python -m pytest -n 8 --dist=loadfile -s --make-reports=tests_torch ./tests/ | tee tests_output.txt
148+
- run: python -m pytest -n 4 --dist=loadfile -s --make-reports=tests_torch ./tests/ | tee tests_output.txt
117149
- store_artifacts:
118150
path: ~/transformers/tests_output.txt
119151
- store_artifacts:
@@ -125,6 +157,7 @@ jobs:
125157
- image: circleci/python:3.7
126158
environment:
127159
OMP_NUM_THREADS: 1
160+
TRANSFORMERS_IS_CI: yes
128161
resource_class: xlarge
129162
parallelism: 1
130163
steps:
@@ -151,6 +184,7 @@ jobs:
151184
- image: circleci/python:3.7
152185
environment:
153186
OMP_NUM_THREADS: 1
187+
TRANSFORMERS_IS_CI: yes
154188
resource_class: xlarge
155189
parallelism: 1
156190
steps:
@@ -159,9 +193,8 @@ jobs:
159193
keys:
160194
- v0.4-flax-{{ checksum "setup.py" }}
161195
- v0.4-{{ checksum "setup.py" }}
162-
- run: sudo apt-get -y update && sudo apt-get install -y libsndfile1-dev
163196
- run: pip install --upgrade pip
164-
- run: sudo pip install .[flax,sklearn,torch,testing,sentencepiece,speech]
197+
- run: sudo pip install .[flax,testing,sentencepiece]
165198
- save_cache:
166199
key: v0.4-flax-{{ checksum "setup.py" }}
167200
paths:
@@ -178,6 +211,8 @@ jobs:
178211
- image: circleci/python:3.7
179212
environment:
180213
OMP_NUM_THREADS: 1
214+
RUN_PIPELINE_TESTS: yes
215+
TRANSFORMERS_IS_CI: yes
181216
resource_class: xlarge
182217
parallelism: 1
183218
steps:
@@ -188,13 +223,13 @@ jobs:
188223
- v0.4-{{ checksum "setup.py" }}
189224
- run: sudo apt-get -y update && sudo apt-get install -y libsndfile1-dev
190225
- run: pip install --upgrade pip
191-
- run: pip install .[sklearn,torch,testing,sentencepiece,speech]
192-
- run: pip install tapas torch-scatter -f https://pytorch-geometric.com/whl/torch-1.8.0+cpu.html
226+
- run: pip install .[sklearn,torch,testing,sentencepiece,speech,vision]
227+
- run: pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.8.0+cpu.html
193228
- save_cache:
194229
key: v0.4-torch-{{ checksum "setup.py" }}
195230
paths:
196231
- '~/.cache/pip'
197-
- run: RUN_PIPELINE_TESTS=1 python -m pytest -n 8 --dist=loadfile -rA -s --make-reports=tests_pipelines_torch -m is_pipeline_test ./tests/ | tee tests_output.txt
232+
- run: python -m pytest -n 8 --dist=loadfile -rA -s --make-reports=tests_pipelines_torch -m is_pipeline_test ./tests/ | tee tests_output.txt
198233
- store_artifacts:
199234
path: ~/transformers/tests_output.txt
200235
- store_artifacts:
@@ -206,6 +241,8 @@ jobs:
206241
- image: circleci/python:3.7
207242
environment:
208243
OMP_NUM_THREADS: 1
244+
RUN_PIPELINE_TESTS: yes
245+
TRANSFORMERS_IS_CI: yes
209246
resource_class: xlarge
210247
parallelism: 1
211248
steps:
@@ -220,7 +257,7 @@ jobs:
220257
key: v0.4-tf-{{ checksum "setup.py" }}
221258
paths:
222259
- '~/.cache/pip'
223-
- run: RUN_PIPELINE_TESTS=1 python -m pytest -n 8 --dist=loadfile -rA -s --make-reports=tests_pipelines_tf ./tests/ -m is_pipeline_test | tee tests_output.txt
260+
- run: python -m pytest -n 8 --dist=loadfile -rA -s --make-reports=tests_pipelines_tf ./tests/ -m is_pipeline_test | tee tests_output.txt
224261
- store_artifacts:
225262
path: ~/transformers/tests_output.txt
226263
- store_artifacts:
@@ -232,14 +269,15 @@ jobs:
232269
- image: circleci/python:3.7
233270
environment:
234271
RUN_CUSTOM_TOKENIZERS: yes
272+
TRANSFORMERS_IS_CI: yes
235273
steps:
236274
- checkout
237275
- restore_cache:
238276
keys:
239277
- v0.4-custom_tokenizers-{{ checksum "setup.py" }}
240278
- v0.4-{{ checksum "setup.py" }}
241279
- run: pip install --upgrade pip
242-
- run: pip install .[ja,testing,sentencepiece]
280+
- run: pip install .[ja,testing,sentencepiece,jieba]
243281
- run: python -m unidic download
244282
- save_cache:
245283
key: v0.4-custom_tokenizers-{{ checksum "setup.py" }}
@@ -257,6 +295,7 @@ jobs:
257295
- image: circleci/python:3.6
258296
environment:
259297
OMP_NUM_THREADS: 1
298+
TRANSFORMERS_IS_CI: yes
260299
resource_class: xlarge
261300
parallelism: 1
262301
steps:
@@ -272,7 +311,7 @@ jobs:
272311
key: v0.4-torch_examples-{{ checksum "setup.py" }}
273312
paths:
274313
- '~/.cache/pip'
275-
- run: python -m pytest -n 8 --dist=loadfile -s --make-reports=examples_torch ./examples/ | tee examples_output.txt
314+
- run: TRANSFORMERS_IS_CI=1 python -m pytest -n 8 --dist=loadfile -s --make-reports=examples_torch ./examples/ | tee examples_output.txt
276315
- store_artifacts:
277316
path: ~/transformers/examples_output.txt
278317
- store_artifacts:
@@ -282,6 +321,9 @@ jobs:
282321
working_directory: ~/transformers
283322
docker:
284323
- image: circleci/python:3.7
324+
environment:
325+
RUN_GIT_LFS_TESTS: yes
326+
TRANSFORMERS_IS_CI: yes
285327
resource_class: xlarge
286328
parallelism: 1
287329
steps:
@@ -292,7 +334,7 @@ jobs:
292334
git config --global user.name "ci"
293335
- run: pip install --upgrade pip
294336
- run: pip install .[testing]
295-
- run: RUN_GIT_LFS_TESTS=1 python -m pytest -sv ./tests/test_hf_api.py -k "HfLargefilesTest"
337+
- run: python -m pytest -sv ./tests/test_hf_api.py -k "HfLargefilesTest"
296338

297339
build_doc:
298340
working_directory: ~/transformers
@@ -306,7 +348,7 @@ jobs:
306348
- v0.4-{{ checksum "setup.py" }}
307349
- run: sudo apt-get -y update && sudo apt-get install -y libsndfile1-dev
308350
- run: pip install --upgrade pip
309-
- run: pip install ."[all, docs]"
351+
- run: pip install ."[docs]"
310352
- save_cache:
311353
key: v0.4-build_doc-{{ checksum "setup.py" }}
312354
paths:
@@ -328,7 +370,7 @@ jobs:
328370
keys:
329371
- v0.4-deploy_doc-{{ checksum "setup.py" }}
330372
- v0.4-{{ checksum "setup.py" }}
331-
- run: pip install ."[all,docs]"
373+
- run: pip install ."[docs]"
332374
- save_cache:
333375
key: v0.4-deploy_doc-{{ checksum "setup.py" }}
334376
paths:
@@ -356,12 +398,14 @@ jobs:
356398
- '~/.cache/pip'
357399
- run: black --check examples tests src utils
358400
- run: isort --check-only examples tests src utils
401+
- run: python utils/custom_init_isort.py --check_only
359402
- run: flake8 examples tests src utils
360403
- run: python utils/style_doc.py src/transformers docs/source --max_len 119 --check_only
361404
- run: python utils/check_copies.py
362405
- run: python utils/check_table.py
363406
- run: python utils/check_dummies.py
364407
- run: python utils/check_repo.py
408+
- run: python utils/check_inits.py
365409

366410
check_repository_consistency:
367411
working_directory: ~/transformers
@@ -380,6 +424,7 @@ jobs:
380424
- image: circleci/python:3.6
381425
environment:
382426
OMP_NUM_THREADS: 1
427+
TRANSFORMERS_IS_CI: yes
383428
resource_class: xlarge
384429
parallelism: 1
385430
steps:
@@ -418,6 +463,7 @@ workflows:
418463
- run_examples_torch
419464
- run_tests_custom_tokenizers
420465
- run_tests_torch_and_tf
466+
- run_tests_torch_and_flax
421467
- run_tests_torch
422468
- run_tests_tf
423469
- run_tests_flax

‎.circleci/deploy.sh

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,4 +57,9 @@ deploy_doc "818878d" v3.5.1
5757
deploy_doc "c781171" v4.0.1
5858
deploy_doc "bfa4ccf" v4.1.1
5959
deploy_doc "7d9a9d0" v4.2.2
60-
deploy_doc "bae0c79" # v4.3.3 Latest stable release
60+
deploy_doc "bae0c79" v4.3.3
61+
deploy_doc "c988db5" v4.4.0
62+
deploy_doc "c5d6a28" v4.4.1
63+
deploy_doc "6bc89ed" v4.4.2
64+
deploy_doc "4906a29" v4.5.0
65+
deploy_doc "4bae96e" # v4.5.1 Latest stable release

‎.github/ISSUE_TEMPLATE/bug-report.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ Models:
3434
- funnel: @sgugger
3535
- gpt2: @patrickvonplaten, @LysandreJik
3636
- rag: @patrickvonplaten, @lhoestq
37-
- tensorflow: @jplu
37+
- tensorflow: @Rocketknight1
3838
3939
Library:
4040
@@ -48,9 +48,13 @@ Library:
4848
4949
Documentation: @sgugger
5050
51+
Model hub:
52+
53+
- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.
54+
5155
HF projects:
5256
53-
- nlp datasets: [different repo](https://github.com/huggingface/nlp)
57+
- datasets: [different repo](https://github.com/huggingface/datasets)
5458
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
5559
5660
Examples:

‎.github/PULL_REQUEST_TEMPLATE.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ Fixes # (issue)
3030
## Who can review?
3131

3232
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
33-
members/contributors which may be interested in your PR.
33+
members/contributors who may be interested in your PR.
3434

3535
<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @
3636
@@ -46,7 +46,7 @@ Models:
4646
- funnel: @sgugger
4747
- gpt2: @patrickvonplaten, @LysandreJik
4848
- rag: @patrickvonplaten, @lhoestq
49-
- tensorflow: @jplu
49+
- tensorflow: @LysandreJik
5050
5151
Library:
5252
@@ -62,7 +62,7 @@ Documentation: @sgugger
6262
6363
HF projects:
6464
65-
- nlp datasets: [different repo](https://github.com/huggingface/nlp)
65+
- datasets: [different repo](https://github.com/huggingface/datasets)
6666
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
6767
6868
Examples:

0 commit comments

Comments
 (0)