Skip to content

Commit 9ccfefa

Browse files
tot0sofiar-msft
authored andcommitted
Set use_rcts in build_index helper used by local index build. (Azure#34110)
* Set `use_rcts` in `build_index` helper used by local index build. Recursive Text Splitter is enabled by default for cloud index builds and effects how Markdown files are chunked. It preferences chunk length being close to the given `chunk_size` over splitting by heading sections. * Udpate changelog. * Move changelog to latest unreleased version.
1 parent 400fcd9 commit 9ccfefa

File tree

2 files changed

+5
-3
lines changed

2 files changed

+5
-3
lines changed

sdk/ai/azure-ai-generative/CHANGELOG.md

+1
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
### Other Changes
66

7+
- Recursive Text Splitter is now enabled by default for local index builds which mimics behavior of cloud index builds
78
- Bug fixes
89

910
## 1.0.0b5 (2024-02-02)

sdk/ai/azure-ai-generative/azure/ai/generative/index/_build_mlindex.py

+4-3
Original file line numberDiff line numberDiff line change
@@ -4,14 +4,14 @@
44

55
from pathlib import Path
66
from typing import Dict, Optional, Union
7-
from packaging import version
87

98
import yaml # type: ignore[import]
9+
from packaging import version
1010

11+
from azure.ai.resources._utils._open_ai_utils import build_open_ai_protocol
1112
from azure.ai.resources.entities.mlindex import Index
12-
from azure.ai.resources.operations._index_data_source import ACSSource, LocalSource
1313
from azure.ai.resources.operations._acs_output_config import ACSOutputConfig
14-
from azure.ai.resources._utils._open_ai_utils import build_open_ai_protocol
14+
from azure.ai.resources.operations._index_data_source import ACSSource, LocalSource
1515

1616

1717
def build_index(
@@ -60,6 +60,7 @@ def build_index(
6060
splitter_args= {
6161
'chunk_size': chunk_size,
6262
'chunk_overlap': chunk_overlap,
63+
'use_rcts': True
6364
}
6465
if max_sample_files is not None:
6566
splitter_args["max_sample_files"] = max_sample_files

0 commit comments

Comments
 (0)