Data Ingestion

To add new data sources, users should select the Upload tab, which offers four options corresponding to supported data types:

• Document: for PDF, TXT, XLSX, PPTX, DOCX files

• URL: for websites

• Confluence: for Confluence pages

• SharePoint: for SharePoint documents and pages

After selecting the data source type, additional fields specific to that type will appear. Once the data source is saved, it remains in a Pending state until processing begins. After processing starts, the system checks every 5 minutes for new chunks. If none are created within this interval, the data source will be marked as Failed. This process ensures that large files are processed efficiently without timeout issues; if a file remains stuck, it will be marked as failed after retries.

5.3.2.1. Documents ingestion

The knowledge bot can retrieve information from the internal repository where knowledge materials are uploaded in formats such as PDF (.pdf), TXT (.txt), XLSX (.xlsx), PPTX (.pptx), and DOCX (.docx).

• Users can upload up to 10 documents at once, all the same type.

• The maximum file size per document is 30MB.

• Ingested data is extracted as plain text (excluding images or comments).

Data Masking Status

• Displays whether masking is enabled

• Lists all default masking entities selected during Kbot creation

Viewing Masked Data

Once ingestion is complete:

1. Open a processed document

2. Navigate to the Masking Entities Section

Compare with the uploaded document

Example Masked Entities

• Phone numbers

• E-Mail addresses

• Locations

• Date and time

Behavior

• Masking is applied only to the selected entity types

• Unselected entities remain visible

Example:

• Phone, email → masked

• Contact person name → not masked (if not selected)

Chunking and ingestion configuration: Depending on the file type, users can set additional settings:

• Divide text by – provides options to divide the document into manageable chunks, for optimal processing:

  • Split by words: Divides content into chunks of 200 to 4,500 words (PDF, DOCX, TXT).

  • Split by page: Separates content based on document pages (PDF).

  • Split by paragraph: Separates by paragraphs (PDF, PPTX, DOCX).

  • Split by slide: For PowerPoint documents.

• Chunk settings:

  • Data Source Description: A brief description of the content to improve accuracy and relevance of information retrieval.

  • Overlap: Number of words repeated between chunks to enhance retrieval context. Higher overlap enhances accuracy but may increase processing load. The option is not available for Excel files.

  • Add description to each chunk: If enabled, the description is prepended to each chunk, to improve retrieval and accuracy.

  • Excel-specific options:

    • Header type:

      • No header

      • Columns – the header is applied to columns, and the header section is displayed horizontally

      • Rows – the header is applied to rows, and the header section is displayed vertically

    • The header is added to each chunk

    • Header size: Number of rows or columns comprising the header.

• Priority: A value between 1 (low) and 100 (high), indicating the importance of the content for retrieval, to ensure critical information is emphasized in responses.

When adding Excels as data source, if the user does not select a specific chunking strategy then the default strategy is used: to check the maximum number of rows / columns that can fit in a chunk as long as the size is smaller than the embedding model max token input (ADA Models have a max token input of 8191 tokens).

For PDFs data sources, the files are processed with Azure Document Intelligence with the Markdown content format.