February 2023 Summaries

### Vectara InstantIndex` Vectara has introduced "InstantIndex", a new feature that enables near real-time processing of large amounts of data, reducing the time between when a document is requested and when it is available to serve queries. This feature solves the challenge of incremental indexing by short-circuiting the traditional event streaming system and journaling process, allowing documents to be instantly indexed in-memory and sent to search nodes for immediate application. InstantIndex reduces latency and supports use cases such as real-time call center transcripts, product launches, employee terminations, and filtering by index time metadata. It is available to customers without configuration and can be used alongside batch and incremental indexing modes.

Vectara offers a versatile indexing solution that supports both batch and incremental indexing, catering to diverse data processing needs in today's fast-paced digital environment. While batch indexing is advantageous for cost and performance optimizations by processing large data sets at once, incremental indexing addresses the demand for real-time updates in search systems by allowing frequent modifications to indexes. Vectara enhances this capability with its InstantIndex feature, which minimizes the delay between document submission and availability for queries, crucial for applications requiring immediate data accessibility, such as chat applications, product catalogs, and real-time call center transcripts. By utilizing in-memory data structures and bypassing conventional event streaming, InstantIndex ensures that documents are instantly available for querying, offering strategic advantages in scenarios like product launches and content updates. As a multi-tenant system, Vectara's infrastructure ensures efficient data distribution across search nodes, enhancing the search experience by providing relevant answers in natural language and supporting cross-language hybrid search.

Vectara's web crawler sample application is designed to handle the most challenging scenarios for data ingestion, without relying on upstream semi-structured data or rendered tags. The crawler offers four modes of link discovery: single URL, sitemap, RSS feed, and recursive crawl. Each mode has its strengths and limitations, and the recursive mode requires careful consideration due to potential issues with rendering timeouts, uniqueness of links, memory usage, and discovering hidden content. Once a link is found, the crawler renders it using either Chrome or Qt WebKit, depending on the `–pdf-driver` parameter, which can impact accuracy and security. Finally, the rendered PDFs are submitted to Vectara's file upload API for processing, ensuring good search results.

The blog post delves into Vectara's web crawler application, which is designed to handle scenarios where traditional data access methods are unavailable, by providing four modes of link discovery: Single URL, Sitemap, RSS, and Recursive. Each mode offers varying levels of complexity and functionality, with Recursive being the most intense as it attempts to discover and index all links from a starting URL. The crawler operates in-memory, utilizes a bloom filter to manage visited sites, and renders pages using either Chrome or Qt WebKit before submitting URLs to Vectara's file upload API for improved search results. The application emphasizes flexibility and adaptability, encouraging users to experiment with different settings and renderers to optimize performance for their specific needs. The overarching goal of Vectara is to enhance user interaction with information by providing relevant, language-agnostic search results that cater to the demands of modern AI-era users.

When deciding whether to crawl a website for search, it's essential to consider if there are better alternatives, such as machine-readable sources of semi-structured documents like raw JSON. Websites were designed for humans, not machines, and traditional HTML parsers may struggle with dynamic elements like heavy use of JavaScript and CSS. A real browser can be used to render the page and extract specific content, but this approach has limitations, especially with PDF generation still in its early stages. If crawling is the right choice, bespoke crawlers or headless browsers like Selenium, Playwright, or Puppeteer can be used to automate the process, while rendering a page as a PDF can provide a more stable output and "get what the user gets."

Part one of a two-part blog series on web crawling discusses the methods, appropriateness, and strategies for effectively crawling websites. It emphasizes that while crawling can make web content searchable, it is often not the best approach if machine-readable data like raw JSON is available, due to the human-centric design of web pages. The blog highlights the importance of using bespoke crawlers and real browsers to handle dynamic web content and suggests rendering pages as PDFs when access to structured data is limited. It also introduces Vectara, a search engine that enhances user interaction by providing relevant, language-independent search results, and hints at a forthcoming exploration of Vectara's web crawler application in the subsequent post.