Our Approach to Table Chunking

Ragie employs a specialized approach to chunking tabular data extracted from various document formats, such as Word, PDF, CSVs, and spreadsheets, for improved semantic retrieval. The strategy aims to address common issues with naive table chunking, such as losing contextual information when chunks end in the middle of columns or rows, and the problem of invalid data in formats like XML, JSON, or YAML when data exceeds chunk size. The Ragie table chunker begins with a structured representation of the data and creates markdown-formatted table chunks, ensuring that table data remains associated with headers and rows are not split mid-record. If a table's size exceeds the chunk size, it is processed by row to fit within the limits, and for tables with many columns, the chunk size is adjusted to accommodate the data without excessive repetition of headers, ensuring that the data remains coherent and effective for hybrid search results.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.