June 2022 Summaries

Large parts of the internet rely on third-party providers like Google, AWS, Fastly, and Cloudflare for their services. While these providers boast high availability rates, they occasionally experience outages. To ensure a product or service is available to customers, all systems required to deliver it must be working, including those controlled by the provider and any dependencies on third-party providers. Achieving higher availability beyond consumer needs can be expensive and may not provide significant value. Instead of building multi-cloud platforms, purchasing high availability products from best-in-class providers is recommended. In case of a big provider outage, prioritizing things within control, choosing high availability products, understanding the impact, communicating with customers, mitigating the impact, and preparing for future incidents are crucial steps to manage such situations effectively.

A survey was conducted to understand how organizations structure and compensate their on-call teams. The study included responses from over 200 companies, including tech giants like Google, Amazon, and Airbnb. Key findings include that in 70% of the organizations, each team is responsible for its own on-call rota rather than having a centralized on-call team. Over 40% of participants were not compensated for being on-call, with this trend more common in larger organizations. The most commonly cited challenges faced by on-callers include disrupted personal life, dealing with issues without sufficient context or knowledge, lack of sleep, and false alerts. To address the issue of accurately calculating time spent on-call, a new on-call compensation calculator has been launched that can automatically generate reports detailing how much each team member is owed based on their hours spent holding the pager.

Charlie has recently joined incident.io in Business Operations after working at Palantir where he led product development for their integrated development environment. He also managed global supply chain initiatives and national COVID-19 responses in several countries, as well as contributed to making Formula 1 cars faster. Excited about joining incident.io, Charlie looks forward to scaling the product to more customers. In his free time, he enjoys golfing, drinking craft beer, and attempting to make the perfect sourdough pizza.

Andy has recently joined incident.io as their new Product Engineer after working as Lead Backend Engineer at Cuvva where he developed a subscription product and fell in love with Golang. Prior to that, he worked at Bloomberg in both financial and developer experience teams. He finds the team at incident.io amazing and believes it would have been beneficial during his career. In his leisure time, Andy enjoys running, tennis, taking care of house plants, and playing board games such as Werewolf, Pandemic, and Terraforming Mars.

The article discusses the importance of declaring incidents early and often in a new job, despite common fears such as the amount of work involved or potential trouble for oneself or the team. It emphasizes that addressing small issues proactively is beneficial for both individuals and organizations, as it helps to prevent larger problems from arising later on. The article also highlights the value of learning incident response processes through game days and applying this knowledge in real-life situations.

The text discusses strategies for running effective meetings that are engaging and productive. Key points include defining a clear purpose for the meeting, creating shared context with pre-read documents, limiting the number of attendees based on the type of meeting, allocating sufficient time, avoiding scheduling during inconvenient times, choosing a facilitator to guide the discussion, taking notes, keeping the meeting on track, and wrapping up with clear next steps. The facilitator should be empowered to effectively manage the meeting and ensure its success.