Skip to content

Commit 8d73d4f

Browse files
authored
Specify that exported ONNX/OpenVINO models don't include pooling/normalization (#3307)
for compatibility reasons
1 parent a51d9c9 commit 8d73d4f

File tree

1 file changed

+11
-1
lines changed

1 file changed

+11
-1
lines changed

docs/sentence_transformer/usage/efficiency.rst

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,11 @@ To convert a model to ONNX format, you can use the following code:
9999
sentences = ["This is an example sentence", "Each sentence is converted"]
100100
embeddings = model.encode(sentences)
101101
102-
If the model path or repository already contains a model in ONNX format, Sentence Transformers will automatically use it. Otherwise, it will convert the model to ONNX the format.
102+
If the model path or repository already contains a model in ONNX format, Sentence Transformers will automatically use it. Otherwise, it will convert the model to the ONNX format.
103+
104+
.. note::
105+
106+
If you wish to use the ONNX model outside of Sentence Transformers, you'll need to perform pooling and/or normalization yourself. The ONNX export only converts the Transformer component, which outputs token embeddings, not sentence embeddings. To get sentence embeddings, you'll need to apply the appropriate pooling strategy (like mean pooling) and any normalization that the original model uses.
103107

104108
All keyword arguments passed via ``model_kwargs`` will be passed on to :meth:`ORTModel.from_pretrained <optimum.onnxruntime.ORTModel.from_pretrained>`. Some notable arguments include:
105109

@@ -291,6 +295,12 @@ To convert a model to OpenVINO format, you can use the following code:
291295
sentences = ["This is an example sentence", "Each sentence is converted"]
292296
embeddings = model.encode(sentences)
293297
298+
If the model path or repository already contains a model in OpenVINO format, Sentence Transformers will automatically use it. Otherwise, it will convert the model to the OpenVINO format.
299+
300+
.. note::
301+
302+
If you wish to use the OpenVINO model outside of Sentence Transformers, you'll need to perform pooling and/or normalization yourself. The OpenVINO export only converts the Transformer component, which outputs token embeddings, not sentence embeddings. To get sentence embeddings, you'll need to apply the appropriate pooling strategy (like mean pooling) and any normalization that the original model uses.
303+
294304
.. raw:: html
295305

296306
All keyword arguments passed via <code>model_kwargs</code> will be passed on to <a href="https://huggingface.co/docs/optimum/intel/openvino/reference#optimum.intel.openvino.modeling_base.OVBaseModel.from_pretrained"><code style="color: #404040; font-weight: 700;">OVBaseModel.from_pretrained()</code></a>. Some notable arguments include:

0 commit comments

Comments
 (0)