September 2023 Summaries

The company recently moved its infrastructure into Google Cloud and experienced a spike in connection timeouts, particularly with Postgres and Memcache. They initially tried doubling the maximum connection lifespan and making connection pools static, which improved the situation but did not completely resolve it. They then switched to their own memcached instance running inside Kubernetes, but this also did not solve the problem entirely. Upon further investigation, they discovered that a bad query was causing thousands of duplicate network calls to an external third party, leading to increased Postgres and Memcache connection and request timeouts. The issue seemed to be related to high volume of TCP connections being opened and closed rapidly on the node due to GKE Dataplane V2 agent Pods (anetd). They mitigated this by implementing keep-alives for HTTP calls and connection pooling for relevant workloads. The company learned that they should not focus their hypotheses too quickly, and limit outbound network concurrency in the future to avoid similar issues.

Incident.io has introduced Status Pages, designed to improve communication between companies and their customers during ongoing incidents. The platform addresses issues with existing status page solutions, such as cumbersome interfaces and standalone products that are not well-integrated into the incident response tech stack. Key features of Status Pages include setting up system components, customer subscriptions via email and RSS, publishing incidents directly from Slack or the incident.io dashboard, transferring settings and subscribers in a few clicks, and customizing brand logos and domains. Since launching, improvements have been made to enhance user experience, such as sub-pages for different audiences, internal status pages for teams like Sales and Support, subscription options via Slack, templates for consistent messaging, automated updates, and retrospective incidents for clarity after events. Upcoming features include Customer Pages for private, authenticated status pages.

Product reliability and availability are crucial factors for modern organizations, directly impacting user satisfaction and trust. While it's impossible to guarantee 100% uptime due to the complexity of technology infrastructure, companies can strive to maintain high levels of both by balancing costs and service quality. Key metrics for measuring reliability include error rate, response time, and crash-free sessions, while availability is often expressed as "9's" (e.g., 99.9% uptime). To improve reliability and availability, teams can implement best practices such as monitoring, testing, automation, fault tolerance, redundancy, load balancing, failover mechanisms, and capacity planning.

Incident management and problem management, once distinct practices carried out by separate teams, have become increasingly intertwined due to movements like DevOps. While they overlap significantly, it's still important to understand the differences between them. Incident management focuses on mitigating impact as quickly and efficiently as possible, while problem management aims to identify and address underlying causes of issues to reduce future incidents. Both are crucial in maintaining efficient software development processes, with incident management handling immediate resolution and problem management preventing recurring problems. The introduction of a more integrated approach has led to the convergence of these concepts under the broader term "incident management."

The text provides practical guidance for getting started as a Site Reliability Engineer (SRE) at a new company, especially when starting with little context about the company's systems and infrastructure. It emphasizes the importance of understanding the current setup, talking to various stakeholders, keeping an eye on customer feedback, seeking external validation for ideas, and creating a roadmap based on common themes from diverse inputs. The suggested roadmap includes improving compute, database, and observability aspects of the system.