Skip to content

Commit ffeb98f

Browse files
dsikkadsikka
authored
[CLIP] Captioning Pipeline (#1145)
* initial refactor * move BasePipeline to a new file * test fix * anothe test fix * fix import * revert * initial refactor * add tests for BasePipeline * move BasePipeline to a new file * initial refactor * update test; finish off initial refactoring changes post local testing * initial commit for clip zero-shot * add basic structure for text branch and zeroshot * add schema details * update pipelines after running mock engine tests * add zeroshot tests * rebase fix * clean-up comments; add note about onnx export issue * move paths to fixtures * rebase fix * rebase fix * refactor pipelines to separate visual, text, and zeroshot. also add pytest skips until model issues are resolved * fix rebase * initial refactor * move BasePipeline to a new file * initial refactor * move BasePipeline to a new file * initial refactor * rebase fix * move paths to fixtures * initial refactor * initial caption functionality * debugging * more debugging * post debugging code * fix imports * cleanup post model fix * fix variable names, some clean-up * remove image embs loading * update dimensions * rebase * remove extra param * remove typo * update README instructions; fix linalg import * clean-up pipelines, updatetyping and descriptions * rebase fix * expose pipeline engine args
1 parent d7f037c commit ffeb98f

File tree

9 files changed

+587
-60
lines changed

9 files changed

+587
-60
lines changed

Diff for: ‎setup.py

+5-1
Original file line numberDiff line numberDiff line change
@@ -162,7 +162,11 @@ def _parse_requirements_file(file_path):
162162
"haystack_reqs.txt",
163163
)
164164
_haystack_integration_deps = _parse_requirements_file(_haystack_requirements_file_path)
165-
_clip_deps = ["open_clip_torch==2.20.0", "scipy==1.10.1"]
165+
_clip_deps = [
166+
"open_clip_torch==2.20.0",
167+
"scipy==1.10.1",
168+
f"{'nm-transformers' if is_release else 'nm-transformers-nightly'}",
169+
]
166170

167171
_torch_deps = ["torch>=1.7.0,<=2.0"]
168172

Diff for: ‎src/deepsparse/clip/README.md

+54-5
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ DeepSparse allows inference on [CLIP](https://github.com/mlfoundations/open_clip
44

55
The CLIP integration currently supports the following task:
66
- **Zero-shot Image Classification** - Classifying images given possible classes
7+
- **Caption Generation** - Generate a caption given an image
78

89
## Getting Started
910

@@ -13,24 +14,38 @@ Before you start your adventure with the DeepSparse Engine, make sure that your
1314
```pip install deepsparse[clip]```
1415

1516
### Model Format
16-
By default, to deploy CLIP models using the DeepSparse Engine, it is required to supply the model in the ONNX format. This grants the engine the flexibility to serve any model in a framework-agnostic environment. To see examples of pulling CLIP models and exporting them to ONNX, please see the [sparseml documentation](https://github.com/neuralmagic/sparseml/tree/main/integrations/clip). For the Zero-shot image classification workflow, two ONNX models are required, a visual model for CLIP's visual branch, and a text model for CLIP's text branch. Both of these model should be produced through the sparseml integration linked above.
17+
By default, to deploy CLIP models using the DeepSparse Engine, it is required to supply the model in the ONNX format. This grants the engine the flexibility to serve any model in a framework-agnostic environment. To see examples of pulling CLIP models and exporting them to ONNX, please see the [sparseml documentation](https://github.com/neuralmagic/sparseml/tree/main/integrations/clip).
18+
19+
For the Zero-shot image classification workflow, two ONNX models are required, a visual model for CLIP's visual branch, and a text model for CLIP's text branch. Both of these models can be produced through the sparseml integration linked above. For caption generation, specific models called CoCa models are required and instructions on how to export CoCa models are also provided in the sparseml documentation above. The CoCa exporting pathway will generate one additional decoder model, along with the text and visual models.
1720

1821
### Deployment examples:
19-
The following example uses pipelines to run the CLIP models for inference. As input, the pipeline ingests a list of images and a list of possible classes. A class is returned for each of the provided images.
22+
The following example uses pipelines to run the CLIP models for inference. For Zero-shot prediction, the pipeline ingests a list of images and a list of possible classes. A class is returned for each of the provided images. For caption generation, only an image file is required.
2023

2124
If you don't have images ready, pull down the sample images using the following commands:
2225

2326
```bash
2427
wget -O basilica.jpg https://raw.githubusercontent.com/neuralmagic/deepsparse/main/src/deepsparse/yolo/sample_images/basilica.jpg
28+
```
2529

30+
```bash
2631
wget -O buddy.jpeg https://raw.githubusercontent.com/neuralmagic/deepsparse/main/tests/deepsparse/pipelines/sample_images/buddy.jpeg
2732
```
2833

29-
This will pull down two images, one with a happy dog and one with St.Peter's basilica.
34+
```bash
35+
wget -O thailand.jpg https://raw.githubusercontent.com/neuralmagic/deepsparse/main/src/deepsparse/yolact/sample_images/thailand.jpg
36+
```
37+
38+
<p float="left">
39+
<img src="https://raw.githubusercontent.com/neuralmagic/deepsparse/main/src/deepsparse/yolo/sample_images/basilica.jpg" width="300" />
40+
<img src="https://raw.githubusercontent.com/neuralmagic/deepsparse/main/tests/deepsparse/pipelines/sample_images/buddy.jpeg" width="300" />
41+
<img src="https://raw.githubusercontent.com/neuralmagic/deepsparse/main/src/deepsparse/yolact/sample_images/thailand.jpg" width="300" />
42+
</p>
43+
44+
This will pull down 3 images, a happy dog, St.Peter's basilica, and two elephants.
3045

3146
#### Zero-shot Prediction
3247

33-
Let's run an example to clasify the images. We'll provide the images in a list with their file names as well as a list of possible classes. We'll also provide paths to the exported ONNX models.
48+
Let's run an example to clasify the images. We'll provide the images in a list with their file names as well as a list of possible classes. We'll also provide paths to the exported ONNX models under the `zeroshot_research` root folder.
3449

3550
```python
3651
import numpy as np
@@ -43,7 +58,7 @@ from deepsparse.clip import (
4358
)
4459

4560
possible_classes = ["ice cream", "an elephant", "a dog", "a building", "a church"]
46-
images = ["basilica.jpg", "buddy.jpeg"]
61+
images = ["basilica.jpg", "buddy.jpeg", "thailand.jpg"]
4762

4863
model_path_text = "zeroshot_research/text/model.onnx"
4964
model_path_visual = "zeroshot_research/visual/model.onnx"
@@ -72,4 +87,38 @@ DeepSparse, Copyright 2021-present / Neuralmagic, Inc. version: 1.6.0.20230727 C
7287
7388
Image basilica.jpg is a picture of a church
7489
Image buddy.jpeg is a picture of a dog
90+
Image thailand.jpg is a picture of an elephant
91+
```
92+
93+
#### Caption Generation
94+
Let's try a caption generation example. We'll leverage the `thailand.jpg` file that was pulled down earlier. We'll also provide the 3 exported CoCa ONNX models under the `caption_models` folder.
95+
96+
```python
97+
from deepsparse import BasePipeline
98+
from deepsparse.clip import CLIPCaptionInput, CLIPVisualInput
99+
100+
root = "caption_models"
101+
model_path_visual = f"{root}/clip_visual.onnx"
102+
model_path_text = f"{root}/clip_text.onnx"
103+
model_path_decoder = f"{root}/clip_text_decoder.onnx"
104+
engine_args = {"num_cores": 8}
105+
106+
kwargs = {
107+
"visual_model_path": model_path_visual,
108+
"text_model_path": model_path_text,
109+
"decoder_model_path": model_path_decoder,
110+
"pipeline_engine_args": engine_args
111+
}
112+
pipeline = BasePipeline.create(task="clip_caption", **kwargs)
113+
114+
pipeline_input = CLIPCaptionInput(image=CLIPVisualInput(images="thailand.jpg"))
115+
output = pipeline(pipeline_input).caption
116+
print(output[0])
117+
```
118+
Running the code above, we get the following caption:
119+
120+
```
121+
DeepSparse, Copyright 2021-present / Neuralmagic, Inc. version: 1.6.0.20230727 COMMUNITY | (3cb4a3e5) (optimized) (system=avx2, binary=avx2)
122+
123+
an adult elephant and a baby elephant .
75124
```

Diff for: ‎src/deepsparse/clip/__init__.py

+6-16
Original file line numberDiff line numberDiff line change
@@ -11,21 +11,11 @@
1111
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
1212
# See the License for the specific language governing permissions and
1313
# limitations under the License.
14-
1514
# flake8: noqa
15+
from deepsparse.clip.decoder_pipeline import *
16+
from deepsparse.clip.text_pipeline import *
17+
from deepsparse.clip.visual_pipeline import *
18+
1619

17-
from deepsparse.clip.text_pipeline import (
18-
CLIPTextInput,
19-
CLIPTextOutput,
20-
CLIPTextPipeline,
21-
)
22-
from deepsparse.clip.visual_pipeline import (
23-
CLIPVisualInput,
24-
CLIPVisualOutput,
25-
CLIPVisualPipeline,
26-
)
27-
from deepsparse.clip.zeroshot_pipeline import (
28-
CLIPZeroShotInput,
29-
CLIPZeroShotOutput,
30-
CLIPZeroShotPipeline,
31-
)
20+
from deepsparse.clip.zeroshot_pipeline import * # isort:skip
21+
from deepsparse.clip.captioning_pipeline import * # isort:skip

0 commit comments

Comments
 (0)