March 2025 Summaries

The gap between planning and implementation of generative AI (GenAI) solutions is a significant challenge that many companies face. To overcome this, it's essential to create a solid foundation by laying down a comprehensive AI stack. This includes infrastructure for creating both traditional and GenAI applications, training, fine-tuning, and providing context to ML pipelines or AI models. A well-designed AI stack should include essential components such as high-quality data, large language models (LLMs), parametric memory, non-parametric memory, agents, tools for prototyping and productizing AI apps, and monitoring and observability systems. By identifying and commoditizing the common components of most AI app solutions, developers can save time and resources, enabling them to build more complex and accurate GenAI applications.

Apache Cassandra 5.0 introduces Storage-attached indexing (SAI), which enhances query patterns, reduces coding requirements, and simplifies adding application functionality. With SAI, developers can modernize their applications with semantic search capabilities using large language models (LLMs) and generative AI to retrieve data based on contextual meaning rather than just text matching. Semantic search uses natural language processing (NLP) and machine learning algorithms to understand the underlying meaning of a user's query and deliver more accurate results. Vector embeddings are used to encode data in a database, allowing for mathematical operations to measure similarity between vectors and quickly locate relevant records. Astra DB, a vector database, enables storage of embeddings alongside movie data, facilitating semantic search capabilities. The introduction of SAI with semantic search capabilities aims to replace traditional full-text search solutions like Solr.

Agents aren't a one-size-fits-all solution for workflows needing structured decision-making, teamwork, or autonomy. Different architectures suit different needs, with single-agent setups ideal for simple tasks, multi-agent architectures suitable for complex processes requiring validation and structured decision-making, hierarchical agents mirroring management structures, and hybrid-sequential architectures blending structure with flexibility. Observability is crucial for debugging and scaling agents, but even the smartest agents need guidance to deliver precise results. The best architecture depends on workflow needs, starting simple and scaling as needed.

Restworld's pioneering platform uses Astra DB's vector search capabilities to transform job searching by enabling developers to bring context into their data search, transforming the approach to job searching and focusing on Restworld's use of collaborative filtering algorithm and real-time, serverless, scalable platform. The company leverages AI-powered tools to explore their database of workers, providing customer success managers with AI-driven features to filter candidates based on semantic search, tailored to match specific parameters, and automatic interview questions and assessment tools that leverage data context to see what's been effective in similar job positions. Restworld's partnership with DataStax offers strong support and collaboration, and the company has seen a fivefold improvement in retrieval conversion rate for internal job seekers. The use of Astra DB's vector search capabilities enables Restworld to store their knowledge representation of job positions, focusing on job data instead of worker profiles, without relying on potentially incomplete worker profiles.

RAG applications use retrieval-augmented generation (RAG) to provide users with natural language interfaces by searching for relevant information and then passing it to a large language model (LLM) to generate the best response. The quality of context is crucial in RAG systems, as without the right context, answers won't be as useful. Building a RAG application is fundamentally different from building an LLM interface, requiring custom rules and limitations to ensure accurate responses. To enable search, documents are embedded as vectors, which requires deciding on chunk size and using techniques such as window strategy to capture unique keywords, meaning, and synonyms. Model temperature and accuracy thresholds also need to be considered when setting up LMs for RAG applications, with parameters controlling the behavior of the model and its ability to generate responses. ANN search is used in RAG applications to find embedded vectors most relevant to a query, providing relevance scores critical for generating meaningful responses.

Cassandra has long struggled with data modeling, leading to questions in the Cassandra tag on Stack Overflow. The author, who answered over 1,175 questions about Cassandra data modeling, wrote a blog post in 2015 about result set ordering and how it can be solved using "query tables." However, this approach had limitations, as it required creating new query tables for each different query. In recent years, Cassandra has introduced storage-attached indexes (SAI) in version 5.0, which allow developers to query by non-key columns without the need for separate query tables. SAI can also enforce a sort order on indexed columns, improving upon previous indexing paradigms. The author explores how to use SAI to solve the problem of result set ordering and discusses DataStax's new Data API, which allows developers to work with data in a document paradigm and exposes their data as a RESTful API. This simplifies development and keeps developers focused on their application rather than the database.

Retrieval-augmented generation (RAG) is a new wave in AI development that allows language models to access relevant documents through external retrieval, enabling them to answer queries beyond their original training timeline and scope. RAG is based on the concept of grounding in evidence, where a retriever and generative model work together to solve problems, with the goal of optimizing cost-quality tradeoffs in enterprise systems. The system involves backpropagating into the query encoder to learn about specific documents, allowing for improved retrieval algorithms. While current implementations often use frozen RAG without adapting to provided documentation, future developments aim to incorporate backpropagation across the entire model, enabling more flexible and dynamic AI systems. As RAG advances, it has the potential to revolutionize language models by grounding knowledge in relevant information, preventing hallucination, and improving multimodal interactions with humans. The technology holds great promise for transforming industries, but still faces challenges such as hallucination, attribution, compliance, data privacy, and cost-quality trade-offs.

Astra DB Hybrid Search is an enhanced search feature that combines vector search and lexical search to improve retrieval accuracy in production AI systems, including those powered by generative AI. The new feature uses the NVIDIA NeMo Retriever reranking microservices to reorder top results based on fine-tuned LLM models, significantly improving relevance. By leveraging Astra DB's hybrid search approach, developers can achieve up to 45% improvement in search relevance and ensure that their GenAI applications return accurate responses. The feature is available on Langflow and in the Data API in April, and users can sign up for a webinar to learn more about its capabilities.

To celebrate PI Day, a team built a weather application on a Raspberry Pi that uses Astra DB and/or Langflow. The goal is to use the National Weather Service's data API to get the most recent weather data, store it in Astra DB, and display it on a simple front-end. The project uses Java 17 Spring Boot Maven Vaadin with an Astra DB account and Langflow instance. The application can pull the latest reading from Astra DB for a specific station and year/month combination, and also refresh its data view from either the Astra DB Data API or from a Langflow API endpoint. To build the project, developers need to assemble the Raspberry Pi, install Java Virtual Machine (JVM) and Java Development Kit (JDK), install Maven, create an SSH key, and clone the project repository. After building and running the application, it can be accessed by navigating to `http://127.0.0.1:8080/` and clicking on "Astra DB Refresh" or "Langflow Refresh". The application displays the latest weather data from Astra DB.

CodeBeasts is an innovative application that transforms users' GitHub activity into unique mythical creatures, representing their most-used programming languages as animals, using AI models like Stability AI or DALL-E for image generation. This app, inspired by displays at GitHub Universe, is constructed with a robust backend of Python, Flask, and Langflow, while the frontend is crafted using Lovable.dev and React. By entering a GitHub handle, users activate a process where an AI agent scans repositories to determine the primary languages, linking each to an animal based on O'Reilly's animal references. Langflow facilitates the creation of complex AI workflows with ease, allowing for swift iteration without extensive code modifications. The app's efficiency is enhanced by storing user data in Astra DB, which reduces redundant GitHub API calls and speeds up subsequent interactions. Developer Relations Engineer David Jones-Gilardi, who has a rich background in programming and databases, emphasizes the app's potential for personalizing user experiences and managing scalable data, ultimately blending creativity with technology to offer an engaging way to visualize coding history.

AI agents have gained significant attention due to their ability to interact with external tools, maintain state, and execute complex workflows, akin to a personal assistant managing tasks like emailing, documenting, and scheduling. However, developing effective agents poses challenges, as seen in the author's experience of building a personal assistant app. Key pitfalls included underestimating the importance of clear, detailed instructions in prompts, overloading a single agent with too many tools, and failing to properly name and describe each tool. The solution involved adopting a multi-agent architecture where specialized agents focus on specific tasks, with an orchestrator agent managing and delegating tasks to ensure efficiency. The author emphasizes the critical role of precise tool descriptions and structured architectures in maximizing agent effectiveness, highlighting that while agents appear powerful, they require thoughtful design and clear information to perform optimally. Langflow, a visual IDE, is noted for aiding in developing generative and agentic AI workflows, allowing for quick iterations and seamless integration with applications.

DataStax has launched several products that support application development in the generative AI and retrieval-augmented generation (RAG) space, including JVector for vector storage and Langflow, a low-code visual tool for rapid GenAI development. However, this led to new testing challenges, particularly assessing semantic performance of modern application stacks. To address these challenges, DataStax integrated the RAGChecker framework with its own products due to its comprehensive capabilities and alignment with strategic goals. RAGChecker evaluates the accuracy, relevance, and completeness of responses generated by large language models (LLMs), which falls outside the typical realm of testing performance features such as latency and throughput. The framework addresses challenges like diverse RAG architectures, dynamic landscape, human involvement, and complex metrics, providing fine-grained diagnostics, supporting dynamic datasets, and aligning with GenAI product stacks. It uses a configurable LLM provider to perform claim-level entailment checks and generates detailed metrics to provide insight into the strengths and weaknesses of various components in the application under test. The integration with Langflow enables varying components of the RAG application prior to evaluation, allowing for iteration over choices made and modification at runtime to come up with repeatable and detailed metrics. This framework establishes a systematic approach to evaluating the effectiveness of RAG applications using clear and understandable scoring metrics, enabling refinement of applications for greater semantic accuracy in the fast-moving world of AI and RAG applications.

Astra DB over MCP is a new way of interacting with the flagship vector database, Astra DB, by leveraging the Model Context Protocol (MCP) to enable agentic interactions between tools and language models. This innovation allows developers to create and manage databases without writing code, using popular MCP clients such as Claude Desktop and Cursor. By integrating Astra DB over MCP, users can perform complex database operations like creating collections and building entire applications without coding, while also accelerating time to production and democratizing innovation. The protocol is an open standard, first pioneered by Anthropic in late 2024, and allows AI systems to share context and execute functions agentically on behalf of the user. Astra DB over MCP is an open-source project, with detailed documentation, a GitHub repository, and a Discord community for users and developers to share advice and best practices.

IBM's acquisition of DataStax aims to enhance the development of production AI and NoSQL data at scale, addressing challenges faced by AI app developers in delivering accurate responses to maintain customer trust. The detailed guide outlines processes for improving AI accuracy using generative AI techniques, specifically through a Retrieval-Augmented Generation (RAG) chatbot example. It emphasizes the importance of experimenting with various tools, including Langflow, NVIDIA NIM microservices, Astra DB, and Arize Phoenix, to evaluate and improve AI applications. By utilizing the Stanford Question Answering Dataset (SQuAD) for benchmarking and integrating NVIDIA's reranking models, developers can refine their AI workflows to achieve better accuracy. The post also highlights the collaboration between Langflow and Arize Phoenix, showcasing how developers can integrate NVIDIA's AI Enterprise services to enhance their generative AI applications. Alejandro Cantarero, Field CTO of AI at DataStax, emphasizes the use of state-of-the-art tools to drive up accuracy in AI applications, inviting developers to discuss advancements at the NVIDIA GTC 2025 event.

Astra DB is a high-performance NoSQL database that simplifies the process of creating vector embeddings for text chunks at the point of inserting them into the collection, allowing users to build accurate and low-latency retrieval-augmented generation (RAG) powered generative AI apps. It supports various embedding models from different providers, including OpenAI, Voyage AI, Mistral AI, Jina AI, Upstage, and NVIDIA embedding models. Astra Vectorize can create vector embeddings for text chunks at the point of inserting them into the collection, simplifying the process of ingesting data for RAG applications. The database also supports graph RAG via LangChain, which takes documents, extracts links between them, and uses those links to retrieve extra contextual information at the retrieval stage, making it easier for large language models (LLMs) to answer certain queries accurately. Astra DB's vector indexing capabilities are a combination of Cassandra's storage-attached indexing (SAI) and JVector, allowing for high throughput and accuracy even under mixed loads of reads and writes. It is available in LangChain and Langflow, which provides a visual way to build agents, making it easy to build RAG or agentic RAG within the platforms. Astra DB supports various applications, including AI resume assistants, voice agents, music recognition apps, and chatbots, and can be used with alternative vector searches like ColBERT for improved accuracy.

IBM is acquiring DataStax to accelerate production of artificial intelligence (AI) and NoSQL data at scale. Langflow, a tool for building flows and AI agents, can be integrated into applications through its API, which allows developers to experiment with generative AI. The @datastax/langflow-client npm module makes it easy to use the Langflow API in Node.js, and streaming GenAI outputs can improve user experience. To stream results from a Langflow flow in Node.js, developers need a flow set up for streaming responses, which requires a model with streaming capabilities and a chat output connected to it. The Langflow client can be used with both self-hosted and DataStax-hosted versions of the platform, and provides a way to log out just the text from a flow response or make API requests from an Express server.

IBM is planning to acquire DataStax, a company that provides NoSQL database solutions, in an effort to accelerate the development of production AI and NoSQL data at scale. The acquisition aims to help organizations migrate their applications from Apache Cassandra to Astra DB, a new database service offered by IBM, without experiencing any downtime or interruption to users. To facilitate this migration, DataStax has developed tools such as the Zero Downtime Migration (ZDM) Proxy, which bridges operational database traffic between Cassandra and Astra DB, allowing data to be moved in an offline manner. The ZDM Proxy is configured with credentials that enable it to communicate with both databases, and its configuration requires the use of IBM's Astra token. The migration process involves connecting the application to the ZDM Proxy, specifying the proxy's IP address along with port 14002, and verifying various connection properties before cutover. While the process can be complex, DataStax has developed tools and resources to ease the migration, including a Modernize Apache Cassandra Workloads page for developers looking to migrate their applications.

IBM to Acquire DataStax, Accelerating Production AI & NoSQL Data at Scale` IBM is acquiring DataStax, a company that provides an AI development platform, to accelerate the production of AI and NoSQL data. The acquisition highlights the growing importance of prioritizing AI projects, as only 9% of organizations have successfully shipped AI products to production. A comprehensive AI strategy is crucial for overcoming technical and legal roadblocks, providing a roadmap for teams across the organization. Key priorities should include identifying overall priorities and potential opportunities, creating guidelines and tools for AI implementation, forming a Center of Excellence (COE) that assists teams with tooling, strategy, data, and other facets of GenAI apps, and ensuring responsible AI practices. The DataStax AI platform provides a scalable, serverless AI development system that combines vector database technology with no-code/low-code approaches to developing GenAI apps.