January 2020 Summaries

The Amazon Builders' Library article discusses implementing health checks for scalable and resilient systems. The authors highlight the importance of balancing thorough health checks that quickly mitigate single-server failures with the harm of false positives that affect the entire fleet. They recommend using a combination of liveness, local, and dependency health checks to measure system health. However, they also caution against over-reliance on health checks, particularly when it comes to dependencies, as this can lead to cascade failures. The article also shares real-world examples of failures with health checks at Amazon and provides guidance on how to react safely to health check failures.

Serverless development offers numerous benefits, with cost efficiency being a primary advantage. By choosing the right tool for the task, developers can save money and optimize their architecture. In the case of Lumigo's data processing flow, the team initially chose Kinesis Streams due to its real-time processing capabilities and scalability features. However, as the company grew and data volumes increased, the cost of Streams became too high, prompting a reevaluation of the decision. After analyzing the costs and benefits of both Streams and Firehose, Lumigo decided to switch to Firehose, which offered significant cost savings, improved scalability, and reduced latency. By utilizing both services together, the team achieved a balance between real-time processing and cost efficiency, demonstrating the importance of careful consideration when making serverless architecture decisions.

The Amazon Builders' Library article "Static stability using availability zones" by Becky Weiss and Mike Furr explains how to achieve static stability in systems using availability zones. Static stability refers to a system's ability to function even when some components are impaired or unavailable. This is achieved by separating the control plane (changes to a system) from the data plane (daily business of resources), ensuring that the data plane is scaled independently and can continue to operate even if the control plane is impaired. The article discusses two patterns for achieving static stability: active-active on availability zones, where services are deployed across multiple AZs, and active-standby on availability zones, where some services require a leader node in one AZ while replicated writes go to another AZ. Amazon's EC2 service uses zonal deployment calendars, keeps network traffic local to the AZ, and designs foundational services to be AZ-independent. This design allows for highly available regional services but introduces additional complexity in managing zonal configurations.

The CEO of Lumigo, Erez Berkner, shares seven guiding principles that shaped his development team's serverless CICD pipeline. The DevOps infinity loop is still relevant in the serverless paradigm, but its details differ from traditional development approaches. The team uses a multi-repo approach due to its better match for a distributed CICD pipeline, despite potential dependency correlation challenges. They emphasize testing in cloud environments, integration tests, automated gates, predictable deployment scripts, and immutable infrastructure as crucial components of their serverless CICD pipeline. The company adopts CircleCI for automation and uses Lumigo's monitoring tools to address the unique challenges of serverless computing. Despite making progress, there is still no clearly defined rulebook for serverless CICD, and the team continues to optimize their processes through experimentation and community engagement.

The Amazon Builders' Library is a collection of articles written by principal engineers at Amazon that explain how Amazon builds scalable and resilient systems. The library focuses on various topics, including avoiding insurmountable queue backlogs, which is the main topic of the latest article in this series. To measure availability and latency for SQS, metrics such as the number of messages going into the dead-letter queue (DLQ) are used to gauge availability, while message age can be used to measure latency. In multi-tenant systems, fairness throttling is crucial to prevent a single customer from monopolizing resources and affecting other customers' workloads. Amazon's strategies for building multi-tenant systems include separating workloads into separate queues, shuffle-sharding, sidelining excess traffic, and dropping old messages by specifying message time-to-live. These strategies aim to avoid backlogs, limit threads and resources per workload, send backpressure upstream, and use delay queues to put off work until later.

Jan 14 2020, Lumigo has achieved ISO 27001 and ISO 27799 (HIPAA compliance) certification, demonstrating its commitment to achieving high standards of data security and privacy in its technology and internal processes. The company will undergo annual auditing to ensure it meets exacting standards for information security procedures and handling of Protected Health Information. This milestone is a testament to Lumigo's investment in providing leading-edge technology and secure solutions to its clients, who view the company as trusted partners.

In a recent article from the Amazon Builders' Library, Jacob Gabrielson highlights the importance of avoiding fallbacks in distributed systems. Fallbacks are strategies used to handle critical failures, but they can be challenging to test and may introduce unpredictable load or latent bugs. Instead, Amazon focuses on improving the reliability of non-fallback cases, letting callers handle errors, pushing data proactively, converting fallbacks into failovers, and ensuring retries and timeouts don't become fallbacks. By adopting these strategies, distributed systems can achieve greater resilience and scalability.

The Amazon Builders' Library article "Using load shedding to avoid overload" by David Yanacek explains the concept of load shedding as a mechanism to prevent system overload and maintain predictable performance. Load shedding involves intentionally rejecting excess incoming requests to reduce the system's availability, but allowing it to maintain consistent performance for accepted requests. This approach can help prevent client-side timeouts and improve overall system reliability. The article discusses various testing strategies, including load tests, and provides guidance on implementing load shedding mechanisms, such as tracking request latency and using timeout hints. It also highlights the importance of visibility and monitoring in optimizing load shedding and suggests using operating system features to limit server resource usage.