Sharing common KV/prompt for parallel batches but without vram duplication - possible ?

[cmp-nct]

About a year ago llama.cpp was upgraded with the better batching API, that included the functions to copy kv cache between sequences.
That avoids having to process the same prompt multiple times if it's shared among parallel clients but it does not avoid the memory duplication issue.
That's a minor issue with a small system prompt but it's a showstopper for large shared context.

Wouldn't it be possible to process the shared context from one shared k/v sequence and "private" k/v tokens in their separate memory slots ?
Is it computationally impossible to mask the k/v that way?

[ggerganov]

This functionality has been available since the very beginning of the batching support - see the llama-batched, llama-batched-bench and llama-parallel examples. It works by simply assigning the tokens in the batch to multiple sequences.

The llama-server does not use it, because it's very unlikely that 2 clients would have a common prompt at the same time. But to compensate, it has a prompt cache and cache reuse (--cache-reuse) functionality.

[cmp-nct]

I thought it literally copies them., I've had no idea that does just an assignment, thanks!