December 2022 Summaries

An end-of-year newsletter challenges several common predictions for 2023, arguing that ChatGPT will not replace Google because conversational AI still lacks sufficient accuracy, transparency, and understanding of user needs. It also predicts that fully autonomous vehicles will remain limited by technical challenges involving weather, construction, complex road conditions, and regulation, preventing widespread adoption. Stable Diffusion and similar generative-image models are expected to remain primarily artistic tools, as specialized applications such as medicine require domain-specific knowledge and reliable clinical validation. The ongoing conflict between artists and AI companies over training data, copyright, and consent is unlikely to be resolved quickly, although opt-out mechanisms may become more common. Finally, the piece anticipates expanding AI regulation in response to growing public and governmental concern over the technology’s societal effects.

Dynamic batching and autoscaling can substantially improve production serving capacity for Stable Diffusion by combining incoming requests into parallel GPU batches and adding or removing model servers according to demand. In the described deployment, batching requests up to 12 at a time increased throughput by 50% and raised per-process concurrency from about 30 to 60 users, with the optimal batch size depending on available GPU memory. Requests were queued, processed by a background coroutine, and returned through a results mechanism, while parallel model servers ran across multiple GPUs. An autoscaling job evaluated queue traffic every 30 seconds, maintaining a configurable minimum server count and scaling capacity as needed; increasing from one to four servers raised concurrency from 60 to roughly 400 users. Performance testing with Locust showed that four servers combined with batches of 12 increased supported concurrent users more than tenfold and could be extended to support thousands of requests.

Lightning is presented as an MLOps platform that extends PyTorch Lightning from distributed model training to deployment, serving, monitoring, and data engineering, aiming to reduce the complexity of fragmented machine learning workflows. Built around the open-source Lightning App framework, it allows Python applications to run locally, on Lightning Cloud infrastructure backed by AWS, or on private clusters with minimal code changes. Its core components, LightningWork and LightningFlow, support the construction and orchestration of long-running tasks such as training, inference, ETL, and application demos, while enabling developers to specify compute resources including CPUs, GPUs, memory, and storage through Python APIs. The example architecture uses a Flow to coordinate separate training and deployment Works, allowing each task to use different infrastructure, such as GPUs for training and CPUs for serving. The platform can be launched locally or in the cloud through command-line options and can be expanded with production capabilities including autoscaling, dynamic batching, hyperparameter optimization, drift detection, and model monitoring.

A machine learning newsletter highlights Sebastian Raschka’s free, university-style deep learning course, which offers practical lessons, quizzes, code examples, and downloadable resources. Research coverage includes the NFL’s AWS-supported challenge to use machine learning and computer vision to analyze player contact and help prevent injuries, a Nature Medicine study identifying four reproducible long-COVID sub-phenotypes, and a privacy-preserving federated learning project using brain scans from 6,314 glioblastoma patients across 71 sites. Engineering updates feature Apple Music Sing’s on-device vocal reduction, Canva Docs’ generative AI tools based on Stable Diffusion and GPT-3, and Lightning’s Echo transcription app powered by OpenAI Whisper alongside new cloud logging features. Community projects include tools for remote-sensing segmentation, 3D machine learning, and Stable Diffusion fine-tuning, while the newsletter also notes upcoming 2023 submission deadlines for IJCAI, ACL, ICML, IROS, and ICCV.

OpenAI’s open-source Whisper speech-recognition model, including its published weights, creates an opportunity to build lower-cost and more privacy-conscious transcription products than proprietary captioning services, particularly for regulated sectors such as healthcare and finance. The author describes building Echo, a transcription app, with the Lightning Framework, using a GPU-backed LightningWork component to integrate and deploy Whisper locally or in the cloud with minimal code. After validating the core idea, the project expanded through Lightning-generated command-line tools, FastAPI-based REST APIs with OpenAPI client generation, and a React frontend. Enterprise-oriented capabilities included autoscaling GPU transcription workers through a Python load balancer, environment-variable and secret-based configuration, Sentry exception monitoring, and GitHub Actions deployment. Echo was launched as a minimal product with its source code available for customization, while potential future additions include real-time transcription, completion notifications, and speaker identification.

NeurIPS 2022 highlighted advances in foundation models, generative AI, datasets, and embodied agents, with notable papers including DeepMind’s data-efficient 70B-parameter Chinchilla language model, Google’s photorealistic Imagen text-to-image system, AI2’s ProcTHOR home simulator, NVIDIA’s Minecraft-based MineDojo agent framework, and LAION-5B, the large image-text dataset behind Stable Diffusion. Other developments included OpenAI’s launch of ChatGPT, trained with reinforcement learning from human feedback to support conversational interactions; AWS’s Amazon DataZone data-management service; and SeeByte’s expanded US Navy contract for autonomous maritime systems. Open-source announcements featured Petoi’s programmable Bittle robot dog and Google’s Rust-based KataOS security-focused operating system, while Lightning AI released a component for deploying Stable Diffusion 2.0 servers quickly. The update also noted community events at NeurIPS and provided submission deadlines for major 2023 AI conferences, including IJCAI, ACL, ICML, IROS, and ICCV.