July 2023 Summaries

Developers often face challenges and rising costs when adding chat functionality to games and apps, but a new solution using Momento Topics, a serverless publish and subscribe messaging service, aims to simplify this process. By utilizing a sample app built with the Next.js framework and Momento's Web SDK, developers can easily integrate chat features without the need for websockets or additional hosting services, as everything can be deployed quickly on platforms like Vercel or Netlify. The app allows the creation of multiple disposable chat rooms by simply instantiating new topic clients and ensures secure communication through direct connections from the browser to Momento Topics, without intermediary data services. A built-in token vending machine provides short-lived, narrowly-scoped tokens to manage user access and maintain security, making it a convenient option for developers seeking an efficient and scalable chat solution.

In the second episode of the Cache-it podcast, Khawaja discusses the intricacies of search system indexing with Manju Rajashkhar, Etsy's VP of Engineering. They explore traditional and modern indexing methods, including the use of inverted indexes in search engines like Lucene and Elasticsearch, emphasizing the importance of selecting the appropriate indexing type based on query nature and the need for continuous learning to enhance search systems. The episode also previews Manju's talk at MoCon 2023 in Seattle, where he shared his expertise on designing adaptable search systems. Manju's background includes leading machine learning and personalization efforts at Etsy, co-founding Blackbird AI, and contributing to Twitter's early engineering efforts, where he developed scalable caching systems that were open-sourced and remain in industry use.

In the first episode of the Cache-it podcast, host Khawaja Shams interviews Yao Yue, CEO and co-founder of IOP Systems, about applying caching lessons to machine learning feature stores. Yao, who has extensive experience from her time at Twitter focusing on distributed caching and performance engineering, discusses the critical role caching played in optimizing Twitter's performance during high-profile incidents. She elaborates on machine learning feature stores as platforms for discovering, storing, and serving features for ML training and inference, while also addressing the challenges of performance engineering at scale and managing tail latencies. Yao is also known for creating Pelikan, a unified cache framework, and her work extends to co-founding IOP Systems to enhance software efficiency and reduce costs.

Tail latency, representing high percentile latency in dynamic, cloud-based applications, is a crucial factor affecting performance, user experience, and client confidence. Though infrequent, these latencies can significantly degrade performance when applications handle vast numbers of operations per second. Monitoring and addressing these latencies, even those affecting a small fraction of requests, is vital because they can snowball into larger system issues, especially during traffic spikes or viral marketing events. Proactively managing tail latency helps maintain site reliability, ensuring that both user experience and client confidence are preserved, while also meeting stringent service-level agreements (SLAs) and objectives (SLOs). By focusing on high percentile latencies, teams can enhance performance robustness, cater to their most demanding users, and protect against potential outages during high-load scenarios.

Momento has announced a new feature that allows users to access Momento Cache in edge environments through an HTTP API, demonstrated with a Cloudflare worker example. Users need a Momento cache, an authentication token, and the associated HTTP endpoint to utilize this feature, which is currently available in AWS regions. The setup involves cloning example code from the Momento JavaScript SDK, configuring Cloudflare worker files, and securely storing authentication tokens as Cloudflare secrets. The provided example code demonstrates basic cache operations such as setting and retrieving data, and users can run a local development server or deploy the setup to their Cloudflare account. The company encourages users to explore advanced applications of Momento Cache in edge functions for enhanced data reliability and functionality.

Momento, an early-stage startup focused on high-performance caching, has successfully optimized its open-core Pelikan caching engine for Google's new Arm-based Tau T2A VMs, achieving significant improvements in queries per second (qps) and cost efficiency. Initially expecting a complex and costly transition, the team found the adaptation process surprisingly straightforward, resulting in a 25% increase in performance goals and the ability to handle over one million queries per second at their 2ms p999 Service Level Objective (SLO). The T2A VMs offer advantages such as cheaper DRAM costs and increased core availability, which allow for larger in-memory working sets and higher throughput. Through head-to-head benchmarking and tuning, Momento realized improvements across both Arm and x86 VMs, proving the investment to be beneficial even at their current scale. The company emphasizes a flexible approach to adopting new processor technologies, encouraging businesses to evaluate cost and performance benefits without committing fully, and highlights their partnership with Google Cloud to enhance the price-to-performance ratio for their customers.