top_k for openai compatible server

[artist-genai]

I am using vLLM in a docker container and inferencing using langchain community VLLMOpenAI. Neither the engine arguments or this package has a parameter to change the top_k value. Does anyone have any idea about how can this be changed?

[zuxfoucault]

same question!

[artist-genai]

Found the answer use this

llm = VLLMOpenAI(openai_api_key="EMPTY", openai_api_base="http://127.0.0.1:8000/v1", model=f"/models/{model_name}", top_p=top_p, max_tokens=1024, frequency_penalty=fp, temperature=temp, extra_body={"top_k":top_k, "stop":["Answer:", "Note:", "Note", "Step", "Answered", "Answered by","Answered By", "The final answer"], "seed":42, "repetition_penalty":rp})

I am passing the values based on certain conditions. The accepted dtype for top_p is float, top_k is int, temp is float, rp is float, fp is float. You can find that in the VLLMOpenAI docs.

[artist-genai]

Found the answer use this

llm = VLLMOpenAI(openai_api_key="EMPTY", openai_api_base="http://127.0.0.1:8000/v1", model=f"/models/{model_name}", top_p=top_p, max_tokens=1024, frequency_penalty=fp, temperature=temp, extra_body={"top_k":top_k, "stop":["Answer:", "Note:", "Note", "Step", "Answered", "Answered by","Answered By", "The final answer"], "seed":42, "repetition_penalty":rp})

I am passing the values based on certain conditions. The accepted dtype for top_p is float, top_k is int, temp is float, rp is float, fp is float. You can find that in the VLLMOpenAI docs.

[zuxfoucault]

yes, it works for me as well. Thanks!