DeepSparse has an official integration within LangChain. It is broken into two parts: installation and then examples of DeepSparse usage.
- Install the Python packages with
pip install deepsparse-nightly langchain - Choose a SparseZoo model or export a support model to ONNX using Optimum
- Models hosted on HuggingFace are also supported by prepending
"hf:"to the model id, such as"hf:mgoin/TinyStories-33M-quant-deepsparse"
There exists a DeepSparse LLM wrapper, which you can access with:
from langchain.llms import DeepSparseIt provides a simple, unified interface for all models:
from langchain.llms import DeepSparse
llm = DeepSparse(model='zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base-none')
print(llm('def fib():'))And provides support for per token output streaming:
from langchain.llms import DeepSparse
llm = DeepSparse(
model="zoo:nlg/text_generation/codegen_mono-350m/pytorch/huggingface/bigpython_bigquery_thepile/base_quant-none",
streaming=True
)
for chunk in llm.stream("Tell me a joke", stop=["'","\n"]):
print(chunk, end='', flush=True)It has arguments to control the model loaded, any configs for how the model should be loaded, configs to control how tokens are generated, and then whether to return all tokens at once or to stream them one-by-one.
model: str
"""The path to a model file or directory or the name of a SparseZoo model stub."""
model_config: Optional[Dict[str, Any]] = None
"""Keyword arguments passed to the pipeline construction.
Common parameters are sequence_length, prompt_sequence_length"""
generation_config: Union[None, str, Dict] = None
"""GenerationConfig dictionary consisting of parameters used to control
sequences generated for each prompt. Common parameters are:
max_length, max_new_tokens, num_return_sequences, output_scores,
top_p, top_k, repetition_penalty."""
streaming: bool = False
"""Whether to stream the results, token by token."""