Introducing AI chunking to semchunk

The introduction of AI chunking mode to the semchunk semantic chunking algorithm, powered by the Kanon 2 Enricher model, marks a significant advancement in improving Retrieval-Augmented Generation (RAG) systems. This AI-driven mode enhances performance by increasing RAG correctness significantly over traditional chunking methods, such as LangChain's recursive chunking and fixed-size chunking. The semchunk algorithm works by preserving syntactic and semantic divisions within chunks, while the Kanon 2 Enricher creates structured knowledge graphs from unstructured documents. The AI chunking mode demonstrates superior accuracy in context-constrained environments by effectively managing document segmentation and maintaining essential context, which is crucial for applications like legal RAG systems. This development underscores the importance of AI-based chunking in optimizing data retrieval and accuracy, offering a 15.6% improvement over the worst-performing algorithms.