> For the complete documentation index, see [llms.txt](https://docs.ixhello.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.ixhello.com/knowledge-bot/functional-documentation/functional-capabilities/data-sources/chunking-strategies-guide.md). # Chunking Strategies Guide ### What is chunking? When you upload a document to Knowledge Bot, the system breaks it into smaller pieces called "chunks." These chunks are what the bot searches through when answering a question. The way your document is split directly affects how accurate and relevant the bot's answers will be. Think of it like organizing a reference book: if you tear out random half-pages and file them, finding the right answer later becomes difficult. But if you file by topic or section, retrieval is fast and precise. Knowledge Bot uses **two different chunking modes** depending on the type of document you upload: * **Text documents** (PDF, DOCX, PPTX, TXT) — split by words, page, or paragraph * **Table documents** (Excel, CSV) — split by rows with header preservation The system shows you the appropriate settings based on the file type. The sections below cover each mode in detail. ### 1. Text document chunking (PDF, DOCX, PPTX, TXT) When uploading text-based documents, you choose how the system divides the content into chunks. Three strategies are available. #### Split by words The document is divided into chunks of a fixed number of words, regardless of where sentences, paragraphs, or sections start and end. **How it works:** You set a word count (e.g. 300 words). The system splits the text every 300 words, creating evenly-sized chunks. **When to use:** * Long, continuous narrative text with no clear structural markers (e.g. transcripts, meeting notes, unformatted text dumps) * When you want predictable, uniform chunk sizes for cost or performance reasons **When to avoid:** * Structured documents (FAQs, procedures, policies) — splitting mid-sentence or mid-paragraph will break the meaning * Documents with tables, numbered lists, or step-by-step instructions **Example:** A 1,200-word company policy with no headings or sections. Split by words at 300 words creates 4 equal chunks. Each chunk may cut mid-paragraph, but the overlap setting (see below) helps preserve context at the boundaries. #### Split by page The document is divided using page breaks as the boundary. Each page becomes one chunk. **How it works:** The system uses the page structure of the uploaded file (PDF pages, Word page breaks) to create one chunk per page. **When to use:** * Slide decks converted to PDF — each slide is a self-contained topic * Documents where each page covers a distinct topic or section * Scanned documents where page boundaries are meaningful **When to avoid:** * Documents where content flows across pages without regard to page breaks (e.g. a 50-page contract where a clause starts on page 12 and continues on page 13) * Documents with very dense pages — a single chunk may contain too many topics, reducing retrieval precision **Example:** A product training deck exported as PDF (30 slides). Each slide covers one feature. Split by page creates 30 chunks, each containing one feature's content. When an agent asks about a specific feature, the bot retrieves the right slide's content. #### Split by paragraph The document is divided using paragraph breaks as the boundary. Each paragraph (or group of related paragraphs) becomes one chunk. **How it works:** The system uses line breaks and paragraph markers in the document to create chunks. Short consecutive paragraphs may be grouped together. **When to use:** * FAQ documents — each question + answer pair is typically one or more paragraphs * Policy documents with numbered sections * Knowledge base articles with clear topic separation per paragraph * Any document where each paragraph is a self-contained unit of information **When to avoid:** * Documents with very short paragraphs (e.g. bullet lists where each bullet is a paragraph) — creates too many tiny chunks that lack context * Narrative text where meaning spans multiple paragraphs without clear breaks **Example:** An FAQ document with 11 questions and answers (like the Axis Multi Asset Active FoF FAQ). Each Q\&A pair is a separate paragraph block. Split by paragraph keeps each Q\&A together as a chunk, so when an agent asks "What is the exit load?" the bot retrieves the complete answer without mixing in unrelated content. #### Overlap (text documents only) Overlap defines how many words from the end of one chunk are repeated at the beginning of the next chunk. **Why it matters:** When a document is split, the boundary between two chunks may cut through a sentence or idea. Overlap ensures that context at the boundary isn't lost. **Recommended values:** * **50–100 words** for most documents — enough to preserve context without excessive duplication * **0** if chunks are naturally self-contained (e.g. FAQ split by paragraph, where each chunk is a complete Q\&A) * **Higher (150+)** for dense technical content where context dependencies span multiple paragraphs **Example:** A technical manual split by words at 300 words, with 100-word overlap. The last 100 words of chunk 1 are also the first 100 words of chunk 2. If a user asks a question whose answer spans the boundary, both chunks contain enough context to produce a relevant answer. *** ### 2. Table document chunking (Excel, CSV) When uploading spreadsheets or tabular data, the text-splitting strategies (words, page, paragraph) don't apply. Instead, kBot offers table-specific settings that control how rows and headers are handled during chunking. #### Table header type Defines how the system identifies headers in your spreadsheet. This is critical — without the correct setting, the bot will lose the relationship between data rows and their column/row labels. **No header** The spreadsheet has no header row or column. All rows are treated as data. When to use: Raw data dumps with no labels. Rare in practice — most spreadsheets have headers. **Columns** The first N rows of the spreadsheet are column headers (e.g. "Agent Name", "Region", "Score"). These headers are preserved and prepended to each chunk so that every chunk retains the column context. When to use: Most spreadsheets — this is the standard tabular format where the top row(s) contain column names and all rows below are data. **Rows** The first N columns of the spreadsheet are row headers (e.g. the first column contains category names, and subsequent columns contain values). These headers are preserved per chunk. When to use: Pivot-style or matrix-style spreadsheets where the leftmost column(s) are labels and the data extends to the right. #### Header size The number of rows (if "Columns" is selected) or columns (if "Rows" is selected) that make up the header. **Recommended values:** * **1** for most spreadsheets (single header row) * **2–3** for spreadsheets with multi-level headers (e.g. a merged header row on top + a sub-header row below) **Why it matters:** If set too low, the system misses part of the header and chunks lose column context. If set too high, actual data rows get treated as headers and are excluded from chunks. **Example:** An escalation matrix with a merged header row ("Contact Details") and a sub-header row ("Name", "Phone", "Email", "Region"). Set header type to "Columns" and header size to 2. #### Max rows per chunk The maximum number of data rows included in each chunk. The header is automatically prepended to every chunk. **How it works:** The system creates chunks of N data rows each, with the header repeated at the top of every chunk. This ensures the bot always knows what each column/row represents. **Recommended values:** * **10–20 rows** for most spreadsheets — small enough for precise retrieval, large enough to preserve grouped context * **5–10 rows** for wide spreadsheets with many columns (to keep chunk size manageable) * **Higher (30–50)** for simple spreadsheets with few columns where rows are closely related **Why it matters:** Too many rows per chunk means the bot retrieves a lot of irrelevant data alongside the answer. Too few means the header overhead dominates the chunk content. **Example:** An agent directory with 200 rows and columns: Name, Region, Skill, Phone, Email. Set table header type to "Columns", header size to 1, max rows per chunk to 15. The system creates \~14 chunks, each containing the header row + 15 agent rows. When someone asks "What is the phone number for Agent X?", the bot retrieves the chunk containing that agent's row — with full column context. #### The table header problem (and why this matters) This is the single most common issue with spreadsheet ingestion. When table headers are separated from their data rows during chunking, the bot sees rows of values with no labels. It doesn't know that "John Smith" is a name, "EMEA" is a region, or "92%" is a quality score. The result is fabricated or interpolated answers — the bot guesses what the columns mean instead of knowing. **Always configure the table header type and header size when uploading spreadsheets.** Leaving these at "No header" on a spreadsheet with headers is the most common cause of poor answer quality for tabular content. *** ### 3. Common settings (both document types) These settings apply to both text documents and table documents. #### Data source description A free-text field where you describe what this document contains. This description helps the bot understand the context of the chunks when matching them against user queries. **Best practices:** * Be specific: "Axis Multi Asset Active FoF FAQ — fund structure, allocation process, exit load, peer comparison" is better than "Fund FAQ" * Include the domain and key topics covered * Keep it under 2–3 sentences **When to check "Add data source description to each chunk":** Enable this when you have multiple documents in the same knowledge base covering different topics. The description is prepended to each chunk, helping the bot distinguish between similar content from different sources. **Example:** You upload three documents to the same bot: a product FAQ, a pricing guide, and a troubleshooting manual. Without the description added to chunks, the bot may confuse pricing content with troubleshooting content. With descriptions enabled, each chunk carries its source context. #### Data source priority A value from 1 to 100 that determines which data source takes precedence when multiple sources contain relevant content. **How it works:** When the bot retrieves chunks from multiple data sources, higher-priority sources are weighted more heavily in the answer. **Recommended values:** * **Default (50)** when you have a single data source or all sources are equally important * **Higher (70†“100)** for the most authoritative source (e.g. the official product documentation) * **Lower (1†“30)** for supplementary content (e.g. meeting notes, informal guides) **Example:** A bot has two data sources: an official product manual (priority 80) and a collection of internal team notes (priority 30). When both sources contain relevant content for a question, the bot favors the official manual. *** ### 4. Quick reference: which strategy for which document type? #### Text documents | Document Type | Recommended Strategy | Overlap | Why | | ----------------------------------- | -------------------------- | ---------------- | --------------------------------- | | FAQ (Q\&A format) | Split by paragraph | 0†“50 words | Each Q\&A is self-contained | | Policy / SOP with numbered sections | Split by paragraph | 50†“100 words | Sections are self-contained units | | Slide deck (PDF) | Split by page | 0 | Each slide is one topic | | Technical manual (long, structured) | Split by paragraph | 100†“150 words | Preserve section context | | Meeting transcript | Split by words (300–500) | 100 words | No structural markers to rely on | | Unformatted text dump | Split by words (300–500) | 100 words | No structural markers to rely on | | Knowledge base article | Split by paragraph | 50†“100 words | Topic-per-paragraph structure | | Legal contract | Split by paragraph | 100†“150 words | Clause boundaries matter | | Sales playbook | Split by page or paragraph | 50†“100 words | Topic-per-section structure | #### Table documents | Document Type | Header Type | Header Size | Max Rows per Chunk | Why | | ----------------------- | ----------- | ----------- | ------------------ | ------------------------------------------- | | Agent directory | Columns | 1 | 10†“20 | Keep agent rows with column headers | | Escalation matrix | Columns | 1†“2 | 10†“15 | Keep escalation paths with their labels | | Pricing / rate card | Columns | 1 | 10†“20 | Keep rates paired with their column context | | Pivot / matrix report | Rows | 1 | 15†“20 | Row labels on the left define the data | | Scorecard / KPI tracker | Columns | 1†“2 | 10 | Keep metrics with their column definitions | *** ### 5. Common mistakes #### Text documents **1. Using "Split by words" on a structured FAQ** The bot ends up with chunks that start mid-answer and end mid-question. Retrieval returns partial answers. Use "Split by paragraph" instead. **2. Setting overlap to 0 on narrative text split by words** Without overlap, context at chunk boundaries is lost. The bot may miss relevant content that spans two chunks. Always use at least 50–100 words of overlap with word-based splitting. **3. Uploading a large document without considering chunk size** A 100-page PDF split by page creates 100 large chunks. If each page contains multiple unrelated topics, the bot retrieves too much irrelevant content alongside the answer. Consider splitting by paragraph for dense, multi-topic pages. **4. Using "Split by page" on a Word document with no page breaks** If the document has no explicit page breaks, the system may treat the entire document as one page, creating a single massive chunk. Use "Split by paragraph" or "Split by words" instead. #### Table documents **5. Leaving table header type as "No header" on a spreadsheet with headers** The bot loses the relationship between data rows and column labels. It sees raw values with no context, leading to fabricated or guessed answers. Always set the correct header type and header size for spreadsheets. **6. Setting max rows per chunk too high on large spreadsheets** A chunk with 100+ rows contains too many unrelated entries. The bot retrieves the entire block when only one row is relevant, diluting answer quality. Keep max rows at 10–20 for most spreadsheets. #### Both document types **7. Leaving data source description empty** When a bot has multiple data sources, empty descriptions make it harder for the bot to distinguish between sources. Always describe what the document contains. *** ### 6. Tips for best results 1. **Match the strategy to your document type.** Text documents use word/page/paragraph splitting. Spreadsheets use table chunking with header preservation. The system shows you the right settings based on the file type — use them. 2. **Test with real questions.** After uploading, ask the bot the same questions your agents would ask. If the answers are incomplete or mix content from different topics, try a different strategy or adjust the settings. 3. **Use data source descriptions.** Even if you only have one document, a good description helps the bot understand the domain context. 4. **Prioritise your most authoritative source.** If you have multiple documents covering similar topics, set the most trustworthy one to a higher priority. 5. **Keep documents focused.** A single, focused document (e.g. "Product X FAQ") produces better results than a catch-all document covering multiple unrelated topics. If possible, split large multi-topic documents into separate uploads. 6. **Always configure table headers for spreadsheets.** This is the #1 cause of poor answers from tabular content. If your spreadsheet has headers (and most do), tell the system. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.ixhello.com/knowledge-bot/functional-documentation/functional-capabilities/data-sources/chunking-strategies-guide.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.