Skip to content

Commit a0e99a3

Browse files
authored
doc: Add a note about int32/int64 datatypes to the kv_layout tutorial (#737)
Not sure the best place to put this, but this is where I was looking when i was experiencing the error in this [issue](#725). Note might need to be removed if the library adds dispatch to kernels with idtype=int64, or type validation in the kernels.
1 parent 9f5fbee commit a0e99a3

File tree

1 file changed

+6
-0
lines changed

1 file changed

+6
-0
lines changed

docs/tutorials/kv_layout.rst

+6
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,9 @@ shape ``(indptr[-1], num_heads, head_dim)`` when the layout is ``NHD``.
4141

4242
We can use ``data[indptr[i]:indptr[i+1]]`` to slice the keys (or values) of request ``i``.
4343

44+
.. note::
45+
``indptr`` arrays across the flashinfer library should be of type ``int32``. Arrays of type ``int64`` can cause indexing errors.
46+
4447
FlashInfer APIs
4548
~~~~~~~~~~~~~~~
4649

@@ -137,6 +140,9 @@ when stored in a tuple of tensors, ``kv_data = (k_data, v_data)``, and each one
137140
where ``max_num_pages`` is the maximum number of pages used by all requests, ``page_size`` is the number of tokens
138141
we fit into each page. ``2`` in single tensor storage means K/V (first one for keys, the second one for values).
139142

143+
.. note::
144+
``indptr`` arrays across the flashinfer library should be of type ``int32``. Arrays of type ``int64`` can cause indexing errors. This is also true of the ``kv_page_indices`` and ``kv_last_page_lens`` arrays.
145+
140146
FlashInfer APIs
141147
~~~~~~~~~~~~~~~
142148

0 commit comments

Comments
 (0)