Skip to content

Commit a745d15

Browse files
authored
Fix initialization of stream decoder (#288)
Fixes #287 Given the following code: ```python import replicate def go(): for event in replicate.stream( "meta/meta-llama-3-8b-instruct", input={ "top_p": 0.9, "prompt": "Hi! Help me please:)", "max_tokens": 512, "min_tokens": 0, "temperature": 0.01, "stop_sequences": "<|end_of_text|>", "prompt_template": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n", "presence_penalty": 0, "frequency_penalty": 1, }, ): print(str(event), end="") print() go() print("---------------------------------") go() print("---------------------------------") go() print("---------------------------------") ``` The latest release repeats the same token once for each previous invocation. <details> ``` Hi there! I'd be happy to help you with whatever you need. What's on your mind?assistant I'm glad you asked! I'm here to help you with any questions or problems you might have. Whether it's something big or something small, I'm here to help. So, what's on your mind? Do you have a specific question or problem you'd like to talk about?assistant I'm all ears! Take your time, and feel free to share as much or as little as you'd like. Remember, everything we discuss is confidential and just between us. So, --------------------------------- Hi Hi there! I'd be happy to help you with whatever you need. What's on your mind?assistant I'm glad you asked! I'm here to help you with any questions or problems you might have. Whether it's something big or something small, I'm here to help. So, what's on your mind? Do you have a specific question or problem you'd like to talk about?assistant I'm all ears! Take your time, and feel free to share as much or as little as you'd like. Remember, everything we discuss is confidential and just between us. So, --------------------------------- Hi Hi Hi there! I'd be happy to help you with whatever you need. What's on your mind?assistant I'm glad you asked! I'm here to help you with any questions or problems you might have. Whether it's something big or something small, I'm here to help. So, what's on your mind? Do you have a specific question or problem you'd like to talk about?assistant I'm all ears! Take your time, and feel free to share as much or as little as you'd like. Remember, everything we discuss is confidential and just between us. So, --------------------------------- ``` </details> This is caused by incorrect initialization of the stream `Decoder` object. After applying this change, the code produces the correct behavior: <details> ``` Hi there! I'd be happy to help you with whatever you need. What's on your mind?assistant I'm glad you asked! I'm here to help you with any questions or problems you might have. Whether it's something big or something small, I'm here to help. So, what's on your mind? Do you have a specific question or problem you'd like to talk about?assistant I'm all ears! Take your time, and feel free to share as much or as little as you'd like. Remember, everything we discuss is confidential and just between us. So, --------------------------------- Hi there! I'd be happy to help you with whatever you need. What's on your mind?assistant I'm glad you asked! I'm here to help you with any questions or problems you might have. Whether it's something big or something small, I'm here to help. So, what's on your mind? Do you have a specific question or problem you'd like to talk about?assistant I'm all ears! Take your time, and feel free to share as much or as little as you'd like. Remember, everything we discuss is confidential and just between us. So, --------------------------------- Hi there! I'd be happy to help you with whatever you need. What's on your mind?assistant I'm glad you asked! I'm here to help you with any questions or problems you might have. Whether it's something big or something small, I'm here to help. So, what's on your mind? Do you have a specific question or problem you'd like to talk about?assistant I'm all ears! Take your time, and feel free to share as much or as little as you'd like. Remember, everything we discuss is confidential and just between us. So, --------------------------------- ``` Signed-off-by: Mattt Zmuda <[email protected]>
1 parent 4cc05ad commit a745d15

File tree

1 file changed

+11
-4
lines changed

1 file changed

+11
-4
lines changed

replicate/stream.py

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -79,10 +79,16 @@ class Decoder:
7979
A decoder for server-sent events.
8080
"""
8181

82-
event: Optional["ServerSentEvent.EventType"] = None
83-
data: List[str] = []
84-
last_event_id: Optional[str] = None
85-
retry: Optional[int] = None
82+
event: Optional["ServerSentEvent.EventType"]
83+
data: List[str]
84+
last_event_id: Optional[str]
85+
retry: Optional[int]
86+
87+
def __init__(self) -> None:
88+
self.event = None
89+
self.data = []
90+
self.last_event_id = None
91+
self.retry = None
8692

8793
def decode(self, line: str) -> Optional[ServerSentEvent]:
8894
"""
@@ -134,6 +140,7 @@ def decode(self, line: str) -> Optional[ServerSentEvent]:
134140

135141
def __iter__(self) -> Iterator[ServerSentEvent]:
136142
decoder = EventSource.Decoder()
143+
137144
for line in self.response.iter_lines():
138145
line = line.rstrip("\n")
139146
sse = decoder.decode(line)

0 commit comments

Comments
 (0)