May 2024 Summaries

Google has announced significant updates to its Gemini API and Google AI Studio, including the release of Gemini 1.5 Flash and 1.5 Pro, which are designed for high-volume, cost-efficient tasks with increased rate limits. These models aim to address developers' needs for lower latency and cost, and tuning support for Gemini 1.5 Flash will be available soon without additional costs. Developers can access these models for free or unlock higher API rate limits through a billing account in Google AI Studio. Additional features include JSON schema mode for controlling model outputs, and improvements in Google AI Studio's user interface with options for light mode and mobile support. For enterprise-grade needs, these models are also accessible via Vertex AI.

Google has introduced the AI Edge Torch Generative API, designed to facilitate the development of high-performance language models using PyTorch for deployment on edge devices via the TensorFlow Lite runtime. This API allows developers to bring generative AI capabilities, such as summarization and content generation, directly onto devices, enhancing performance and developer efficiency. The release is part of a series of updates from Google AI Edge, following the initial introduction of AI Edge Torch for PyTorch model inference on mobile devices. The Generative API offers an intuitive authoring experience, compatibility with existing deployment flows, and support for models like TinyLlama, Phi-2, and Gemma 2B, with future enhancements planned for GPU and NPU support. It also includes features like multi-signature export for efficient large language model inference, quantization for optimized performance, and cross-platform deployment capabilities. The announcement highlights ongoing collaboration across Google teams and invites developers to engage with the library as it continues to evolve.

Hundreds of millions worldwide utilize Google Pay for secure transactions, and recent updates unveiled at Google I/O 2024 highlight enhancements aimed at enriching user experience and simplifying merchant integration. New features include the integration of "buy now, pay later" options with Affirm and Zip in the US, which reportedly boost customer acquisition and retention. Google Pay's dynamic button now offers improved contextual information by displaying card art and issuer names, which has shown to increase transaction rates by up to 40%. Security is enhanced with zero fraud liability for eligible transactions and a merchant liability shift for Visa online transactions. Improved developer productivity is supported by enhanced APIs, a preview feature in Jetpack Compose for easier design experimentation, and an extended test card suite for comprehensive payment flow validation. Enhanced error messaging also aids in debugging and user communication during payment processes, while documentation and sample applications are available for developers to facilitate integration.

KotlinConf 2024 served as a platform for Google to highlight its commitment to Kotlin Multiplatform (KMP), a JetBrains-developed tool that facilitates cross-platform app development by compiling Kotlin code into platform-native binaries. Google has been integrating KMP into its projects, such as Google Workspace, to enhance code sharing and efficiency across mobile, web, server, and desktop platforms. The conference featured keynotes and technical sessions, including Google's Jeffrey van Gogh discussing KMP's role in streamlining development, and presentations by the Android Jetpack team on KMP's integration into Jetpack libraries. Other sessions highlighted the evolution of Java and Kotlin, performance optimizations in Kotlin, and Kotlin Lint Checks. Attendees had the opportunity to engage with engineers and explore KMP resources, with Google expressing its ongoing plans to expand KMP support to more AndroidX libraries, reflecting its broader adoption of Kotlin for efficient app development.

Google's introduction of the Google Wallet API has received strong support from the developer community, significantly expanding the reach of digital valuables globally. Now available in over 80 countries, Google Wallet continues to extend its accessibility, with plans to launch in all major regions. Recently, Google Wallet became available in India, collaborating with over 20 leading Indian brands, thus offering one of the most comprehensive partner networks for a digital wallet in the country. The API enables Indian developers to create and distribute secure digital assets such as boarding passes, event tickets, and loyalty cards to users. Google Wallet's launch in India complements Google Pay, which has been active in the country since 2017, contributing to the evolution of digital payments in India and providing Google with insights into digital transformation in technologically advanced societies.

At Google I/O 2024, a range of new features and tools were announced to enhance Android development, primarily through the integration of AI capabilities and improvements in Android Studio. The updates include the introduction of Gemini, an AI model that offers code suggestions and crash report insights, which developers can leverage using a starter app template or integrate with Google Cloud's Vertex AI for advanced workflows. The Android Studio Koala Feature Drop also introduces productivity enhancements such as a faster and improved Profiler, USB cable speed detection, and a new streamlined Google sign-in process. Additional features support the development of Wear OS apps, including preview tiles and synthetic sensor data generation for testing. The update also includes an IntelliJ Platform update, bringing improvements like a revamped terminal and sticky lines in the editor for easier navigation. These innovations aim to make Android app development more efficient and are available in the Android Studio canary channel for developers to explore and provide feedback.

Google is enhancing the Google Wallet API to offer a more comprehensive, convenient, and approachable platform for managing digital valuables, integrating with other Google services for a seamless user experience. Announced at Google I/O, the updates focus on security, wearables, push notifications, Auto Linked Passes, Generic Private Passes, Gmail integration, and developer experience improvements. The security enhancements include an extension to the Android Credential Manager API to allow the use of digital credentials like government-issued IDs, ensuring privacy by letting users authorize data sharing. Wearables now support more passes on Wear OS devices, while Auto Linked Passes enable automatic addition of related passes for users. Expanded push notifications keep users informed of changes, and improved Gmail integration allows boarding pass creation from confirmation emails. Developer tools include client libraries for multiple programming languages, a Flutter plugin, and framework integrations for Java, alongside new analytics available in the Google Pay & Wallet Console. These advancements aim to streamline user engagement and provide developers with enhanced tools to create integrated digital experiences.

Firebase Genkit is an open-source framework designed to simplify the integration of generative AI into applications by providing developer-friendly tools and patterns. It assists developers in building, testing, deploying, and monitoring AI workloads, currently supporting JavaScript/TypeScript with Go support on the horizon. Genkit offers a comprehensive tooling experience through a dedicated CLI and a browser-based developer UI that allows for efficient exploration and evaluation of AI solutions. Its components are equipped with Open Telemetry for enhanced observability and monitoring, enabling detailed inspection of AI workflows. The framework also introduces the dotprompt file format for managing prompt engineering alongside code, facilitating easier testing and organization. Genkit supports a plugin ecosystem that integrates with Google Cloud, Firebase, and Vertex AI, among others, providing access to pre-built components for various AI tasks. By using Genkit in the Compass travel planning app, developers were able to enhance user experiences with AI-powered features like semantic search and prompt refinement, demonstrating Genkit's capabilities in streamlining AI development from prototype to production.

Location-based augmented reality (AR) experiences are revolutionizing user interaction by seamlessly integrating digital attributes into physical environments, making it easier for developers to create engaging content with tools like Geospatial Creator powered by ARCore and Photorealistic 3D Tiles from Google Maps Platform. These tools have allowed over 2,700 participants to develop immersive AR experiences for various sectors, including entertainment and travel, while Google is testing a pilot program to showcase AR content in Google Maps, enabling users to discover AR experiences through Street View and Lens. The initiative, starting in Singapore and Paris, aims to enhance tourism and cultural discovery, as seen in collaborations with the Singapore Tourism Board and Google Arts & Culture, offering immersive journeys through landmarks and historical events. Recent updates to Geospatial Creator improve content creation efficiency and expand coverage to include India, reflecting Google's commitment to advancing AR technology and supporting creators globally.

Google is transforming the smart home landscape by reimagining Google Home as a platform for developers, leveraging the Matter standard to enhance connectivity and user experience. The introduction of Home APIs and Home runtime allows developers to access a vast ecosystem of over 600 million devices, enabling the creation of innovative app experiences on both Android and iOS. These APIs facilitate seamless device integration and automation using Google's intelligence, ensuring privacy and security by requiring user consent for access. The new Device, Structure, and Automation APIs empower developers to build smart home solutions with improved local control and responsiveness. Google's expansion of hubs, including integrating with TVs like Chromecast with Google TV, enhances remote access and local device control. Collaborations with partners across various industries, such as ADT, LG, and Eve Systems, showcase diverse applications of these APIs, from enabling secure home access to automating home environments, opening opportunities for developers to bridge digital experiences with physical devices.

Project Gameface, unveiled at I/O 2023, is an open-source, hands-free gaming tool that enables computer control through head movements and facial gestures, designed to enhance accessibility for individuals with disabilities. Initially inspired by quadriplegic gamer Lance Carr, this innovation has evolved through collaboration with various organizations, including playAbility and Incluzza, to expand its use in gaming, education, and workplace settings. Utilizing the Android Accessibility Service and MediaPipe's Face Landmarks Detection API, Project Gameface allows users to perform actions like cursor movement, clicks, and drag functions through a virtual cursor on Android devices, offering customizable and intuitive control options. The project aims to provide a cost-effective, scalable solution that leverages previous learnings to be user-friendly and customizable, and its code is now available on GitHub to encourage further development.

Google AI Edge Torch is a new initiative by Google that facilitates the integration of PyTorch models into the TensorFlow Lite (TFLite) runtime, enhancing model coverage and CPU performance. This tool, part of the Google AI Edge suite, marks Google's commitment to framework flexibility, adding PyTorch compatibility to existing support for Jax, Keras, and TensorFlow. Released in Beta, AI Edge Torch offers seamless PyTorch integration, excellent CPU performance, and initial GPU support, validated on over 70 models from platforms like torchvision and HuggingFace. It supports more than 70% of core_aten operators in PyTorch and allows for easy conversion to TFLite models without deployment code changes, featuring a PyTorch-centric experience. The AI Edge Torch project aims to reduce developer friction and boost performance, offering significant improvements over existing workflows like ONNX2TF. Collaborations with companies like Shopify and hardware partners such as Qualcomm have enhanced performance and coverage, with the introduction of Qualcomm's new TFLite delegate providing notable speedups. Future plans include expanding model coverage, improving GPU support, and enabling new quantization modes, with ongoing contributions and feedback from the PyTorch community and hardware partners.

The Gemini API Developer Competition invites developers of all skill levels to create innovative applications using the Gemini API, with the aim of shaping the future of AI by addressing real-world challenges and enhancing accessibility, sustainability, and joy. Participants can integrate the Gemini API into new or existing mobile, web, or other applications, with the potential to win a range of prizes, including a custom electric 1981 DeLorean and awards for innovation, impact, and creativity. The competition emphasizes the transformative potential of AI, likening it to the early days of the web, and encourages developers to explore applications such as AI-driven disaster response systems, adaptive educational games, and advanced customer service chatbots. Resources such as the Gemini API cookbook and prompt gallery are provided to assist participants, and the competition runs until August 12, 2024, culminating in a People's Choice Award voted by the developer community.

Google I/O 2024 has highlighted significant updates to Project IDX, now in beta, showcasing its role in creating an integrated workspace for developing full-stack, AI-powered applications. With the removal of the waitlist, users can access features like AI-assisted code completion, collaboration tools, and seamless integration with Google services such as Flutter and Firebase. Enhanced AI assistance through the Gemini model offers contextual code suggestions and slash commands for efficient workflow management. The platform's integration panel allows developers to add generative AI features, deploy apps using Google Maps and Firebase, and utilize debugging tools directly within the environment. Project IDX also provides new templates to facilitate quick project setup and plans to incorporate more features based on user feedback, further streamlining development processes.

Google's Gemma models, a family of lightweight, state-of-the-art open models, have quickly gained widespread adoption, inspiring developers to create diverse projects like multilingual variants and on-device action models. Building on this momentum, Google has introduced CodeGemma for code completion and RecurrentGemma for efficient inference, while expanding the lineup with PaliGemma, a powerful open vision-language model designed for tasks such as image captioning and object detection. PaliGemma is accessible through platforms like GitHub, Hugging Face, and Kaggle, and Google provides cloud credits for academic research. They are also preparing to launch Gemma 2, which promises greater performance and efficiency due to its new architecture, offering developers versatile tuning across various platforms. In line with their commitment to responsible AI, Google is enhancing its Responsible Generative AI Toolkit with tools like the LLM Comparator to ensure safe and effective model evaluations. Google's ongoing initiatives reflect their dedication to fostering innovation in AI while ensuring ethical development practices.

Developers often face the challenge of choosing the right languages, frameworks, and tools for building cross-platform apps, with Google's offerings providing several options. For Android-specific app development, Kotlin, combined with Jetpack Compose, is recommended to leverage the latest Android features and services, supporting a wide range of devices from phones to cars. Kotlin Multiplatform (KMP) is suggested for sharing business logic across various platforms, allowing compilation into platform-specific binaries and reducing code duplication. For those wanting to share both UI code and business logic across platforms, Flutter, utilizing the Dart programming language, is Google's suggested SDK, enabling consistent app experiences across mobile, web, and desktop. Both Flutter and Kotlin are being improved for better interoperability, offering developers flexibility in selecting the best approach for their projects.

At this year's Google I/O, the focus is on making AI accessible and beneficial for developers by introducing new tools and models across the entire development stack. Key highlights include the public preview of the Gemini 1.5 Flash and Pro models, which streamline AI-powered applications and are available in over 200 countries. The Gemini API now supports new features like context caching and parallel function calling, enhancing workflows for large prompts. Google also introduced PaliGemma for multimodal tasks and previewed Gemma 2, a highly efficient model. An open ecosystem is emphasized, allowing developers to leverage various tools such as TensorFlow, PyTorch, and Keras to optimize AI workloads. Google AI Edge provides solutions for deploying machine learning models to mobile and web platforms, with expanded support for TensorFlow Lite. Developers are encouraged to participate in the Gemini API Developer Competition, and there are numerous enhancements in Android and web development, including the integration of Gemini Nano in Chrome. Firebase is evolving with AI-powered features, and Google's AI-powered compliance platform, Checks, is available for iOS and Android developers. The program aims to foster innovation and collaboration, with ongoing resources and events to support developers worldwide.

Google has announced the general availability of Checks, an AI-powered compliance platform designed to help Android and iOS developers manage data privacy compliance efficiently. By utilizing Google's advanced technology, Checks provides automated privacy compliance reports that offer insights into data collection and sharing practices, identifying potential privacy issues and offering actionable recommendations. It integrates seamlessly into developers' workflows, enhancing visibility and alignment across product, engineering, legal, and privacy teams, thereby accelerating app launches and ensuring compliance. Additionally, Checks Code Compliance, currently in private preview, aids in maintaining code compliance through real-time analysis, while Checks AI Safety focuses on ensuring safe generative AI outputs through customizable safety policies. This platform empowers developers to innovate responsibly, protecting user data and adhering to evolving regulations.

Google I/O 2024 introduced several enhancements to Firebase, aiming to streamline the development of AI-driven applications across platforms. Among the highlights is the introduction of Firebase Data Connect, which allows developers to connect their apps directly to PostgreSQL databases hosted on Cloud SQL, facilitating an integrated approach to data management and AI feature development. Firebase's evolution includes Genkit, an AI integration framework in beta, designed to simplify the incorporation of sophisticated AI features using various libraries and tools. Additionally, the Vertex AI for Firebase SDKs enable direct calls to AI models from app clients, with security features to mitigate threats like fraud and impersonation. The new Firebase App Hosting framework supports modern web app deployment and scalability, while updated monitoring tools like the Release Monitoring dashboard and Gemini in Firebase provide real-time insights and AI-assisted crash analyses to enhance app performance and user experience. These updates are part of Firebase’s ongoing commitment to support developers in creating robust, modern applications.

Kaggle Models, which hosts thousands of pre-trained models, is now open for user contributions with Keras model uploads, allowing users to share fine-tuned models easily. This process involves importing the required libraries, fine-tuning the model, saving it as a KerasNLP preset, and uploading it to Kaggle using a specific URI format. Users can make their models discoverable by adding descriptions and dataset details, with a usability rating system to guide improvements. Keras is preferred for its consistent user experience, supporting frameworks like JAX, TensorFlow, and PyTorch, and it has shown significant download success compared to other formats. The integration of KerasNLP with Hugging Face allows models to be uploaded and accessed in a similar manner, further enhancing user accessibility and engagement. This development transforms model sharing on platforms like Kaggle and Hugging Face, making it more efficient by eliminating the need for lengthy fine-tuning processes and expanding community interaction through competitions and innovative applications.

Google I/O is set to begin with a keynote on May 14, followed by various sessions and over 150 technical deep dives available on-demand from May 16, aimed at equipping developers with the latest in technology innovations. The event will highlight the Gemini era in AI, showcasing new features in the Gemini API and Google AI Studio, as well as pre-trained models from Kaggle and open-source libraries like Keras and JAX. Developers can expect updates on Android 15, advancements in generative AI, and tools in the Jetpack and Compose ecosystem, alongside breakthroughs in web development with Baseline, which offers real-time information on web features and API interoperability. The future of ChromeOS will also be a focal point, discussing investments in app capabilities and operating system integrations to enhance Chromebook experiences. Participants are encouraged to register online and prepare their 'My I/O' agenda to fully engage with the event's offerings.