September 2024 Summaries

Prompt design and engineering, a crucial yet complex task for effectively utilizing Large Language Models (LLMs), can be optimized using Google's Vertex AI Prompt Optimizer, now available in Public Preview. This service aids users in crafting the most effective prompts by iteratively refining instructions and demonstrations for any chosen model on Vertex AI, drawing on methods outlined in Google Research's paper on automatic prompt optimization. By incorporating a systematic approach to prompt refinement, users can save significant time and effort, ensuring high-performance prompts for their GenAI applications. The service employs an optimization algorithm where separate models generate and evaluate candidate prompts, selecting the best based on user-defined metrics. The blog illustrates the process through the example of enhancing a cooking assistant’s prompt template for better performance using the Vertex AI SDK for Python, ultimately demonstrating how the tool streamlines prompt engineering and enhances the quality of LLM-based applications.

Google has released updated versions of its Gemini models, namely Gemini-1.5-Pro-002 and Gemini-1.5-Flash-002, offering improved performance and significant cost reductions. These updates include over 50% price reductions for input and output tokens under 128K, increased rate limits, faster output, and reduced latency, making them more efficient and cost-effective for various tasks such as synthesizing information from extensive PDFs, answering complex code-related questions, and video content creation. The models now exhibit superior performance in math, long context, and vision-related tasks, with improvements noted in benchmarks like MMLU-Pro, MATH, and HiddenMath, and enhancements in visual understanding and Python code generation. The updated models respond more efficiently, with shorter default outputs for tasks like summarization and question answering, while maintaining content safety standards. Google has also announced a new experimental version, Gemini-1.5-Flash-8B-Exp-0924, which promises enhanced performance across various use cases and is accessible via Google AI Studio and the Gemini API. These advancements reflect Google's commitment to incorporating developer feedback and optimizing its experimental-to-production release pipeline.

Google has announced updates to its Photos APIs aimed at enhancing security and integration capabilities for both users and developers. The introduction of the new Google Photos Picker API allows for a secure and intuitive way to select photos and albums through seamless integration with the Google Photos app. Changes to the existing Library API, taking effect on March 31, 2025, will focus on managing photos and videos uploaded by applications, prioritizing user privacy and data protection. Developers currently using the Library API for selecting content not uploaded by their apps must migrate to the Picker API to maintain a smooth user experience. Additionally, the new Photos API User Data and Developer Policy will replace the existing Acceptable Use Policy after the same date. These updates aim to promote a more secure and user-centric approach to integrating with Google Photos, encouraging developers to build creative solutions using the new APIs.

PaliGemma is a versatile vision-language model (VLM) that integrates both image and text inputs to generate text responses, inspired by PaLI-3 and utilizing components such as the SigLIP vision model and the Gemma language model. This model architecture employs a specialized Gemma 2B model in conjunction with an image encoder to process inputs, with the vision model handling image segmentation and object detection by breaking images into patches and encoding spatial information. PaliGemma uses a multi-modal projector to combine vision and language representations, enabling it to generate coherent outputs from both image and text data. The model's tokenizer extends its vocabulary to include tokens for coordinates and segmentation tasks, facilitating advanced image processing capabilities. Released by Google, the Gemma family, including PaliGemma, offers a broad range of functionalities for modern language model systems, making it suitable for a diverse array of tasks and fostering collaborative development within the Google Developer Community.

LiteRT, previously known as TensorFlow Lite (TFLite), is a rebranded high-performance runtime for on-device AI that now supports models from various frameworks such as PyTorch, JAX, and Keras. Part of Google's AI Edge suite, LiteRT facilitates seamless deployment of machine learning models on platforms like Android, iOS, and embedded devices, leveraging robust model conversion and optimization tools. Although the name has changed, LiteRT maintains the same functionality and provides continuity for existing TFLite users, requiring minimal code adjustments. The transition to LiteRT is progressive, with updates reflected in developer documentation and dependencies, while the main TensorFlow brand remains unaffected. This rebranding underscores Google's commitment to a multi-framework ecosystem and aims to improve model deployment with GPU and NPU acceleration across platforms, ensuring that developers can continue using their current workflows with minor updates to package dependencies.

Google has introduced Controlled Generation for its Gemini 1.5 Pro and Flash models, which allows developers to generate AI responses that conform to a predefined schema, ensuring consistency and reducing post-processing time. This feature, showcased at Google I/O, has been rapidly adopted and positively received, facilitating the integration of AI into software development by enabling seamless data handoff and integration into existing systems. Controlled Generation supports formats like JSON and is built on OpenAPI 3.0 standards, making it compatible with existing workflows and the API economy. It introduces predictability to AI outputs, allowing developers to create structured, machine-readable data and enabling applications like a meal planning app to generate recipes in a structured format. The feature is based on Google's controlled decoding advancements, adding minimal latency to API calls, and supports "enum" as a type, which allows for the classification of product conditions within a set of predefined values. While it enforces output format, the actual response depends on the model's reasoning capabilities, and limitations include dependency on model training data and potential output of responses based on insufficient prompts.