April 2024 Summaries

As businesses increasingly incorporate large language models (LLMs) and generative AI into their operations, maintaining customer trust and safety becomes challenging due to unpredictable behaviors from AI agents. Automated benchmarks often fall short in capturing the complexities of real-world interactions, especially in specialized domains, necessitating a hybrid evaluation approach that combines human and automated techniques. LangSmith and Labelbox address this need by offering enterprise-grade solutions for LLM monitoring, human evaluation, and data labeling. LangSmith provides a platform for developing, testing, and monitoring LLM applications, offering features like dataset management and prompt experimentation, while Labelbox focuses on optimizing data labeling and supervision to enhance model performance. The integration of these platforms aims to improve the reliability and performance of generative AI applications by leveraging human feedback and advanced monitoring, ultimately enhancing the quality of interactions in production environments.

As generative AI applications move from prototypes to production, evaluating large language models (LLMs) is essential for success, with current state-of-the-art techniques combining automated and human assessments, although human evaluation remains resource-intensive. To address this, Labelbox and Google Cloud have collaborated to offer an integrated, managed service within the Vertex AI platform, allowing users to efficiently evaluate LLMs through a streamlined process that includes selecting evaluation types and criteria, and receiving quality-reviewed results swiftly. This service gives customers access to human raters for extensive evaluation across various criteria and integrates seamlessly with existing Google Cloud services such as BigQuery and CloudSQL. Furthermore, Labelbox offers a full suite of products available on the Google Cloud Marketplace, including AI-assisted labeling, data curation, and model diagnostics, which enhance the development of intelligent applications by blending AI assistance with human oversight. This partnership aims to simplify and enhance LLM development by incorporating human evaluation seamlessly, allowing organizations to focus on creating and shipping AI products with reduced manual effort.

Labelbox has introduced a new data warehouse integration tool powered by Census, simplifying the synchronization of data pipelines for AI teams by offering a no-code setup that connects with over 25 data storage options. This tool supports full change data capture from major data warehouses to simple spreadsheets and allows data engineers to build data pipelines in under five minutes without the need for custom Python scripts, thereby reducing both time and cost. It also provides flexible synchronization options with upserts to reflect data changes accurately in Labelbox Catalog datasets. This integration can result in significant cost savings, particularly for large customers managing complex data pipelines, and supports synchronization with platforms like Google Sheets and various cloud storage providers. The tool is available for free with limitations, and additional Syncs or private Census workspaces are available through an Enterprise Beta program.

Labelbox has transitioned from a traditional continuous delivery (CD) pipeline to a GitOps approach using ArgoCD, enhancing automation, collaboration, and deployment reliability. This shift addresses challenges faced with their previous system, such as complex version control and the need for credentials in each target environment, by employing a declarative configuration where the desired state of infrastructure and applications is defined in Git. ArgoCD automates deployments by monitoring changes in the Git repository, reducing deployment time, improving reliability, and fostering stronger collaboration between development and operations teams. The GitOps approach also enhances scalability and security, as well as provides auditability through Git commit histories and Kubernetes Events emitted by ArgoCD. As a leading data-centric AI platform, Labelbox supports teams in integrating generative AI and LLMs with human oversight, serving major enterprises like Walmart and Adobe, and is open to new talent for advancing AI development.