May 2023 Summaries

May 22, 2023: A company like Lumigo uses a set of tests to deploy code changes fast and relies on integration tests to ensure new changes work and don't break existing functionality. However, these tests can sometimes fail due to issues such as test code being too fragile or caching problems. To address this, the company recently switched from traditional logging methods to OpenTelemetry traces graphs, allowing them to better understand system behavior and identify issues that may be difficult to uncover through logging alone. By using OpenTelemetry, they can create meaningful graphs and timelines of events, including tracing test execution, external calls, logs, and more.

The software technology space is rapidly evolving, particularly in the area of Observability, which refers to gaining insight into the internal workings and performance of cloud-based software. The rapid change in this field has been driven by changes in application deployment, such as the shift from monolithic architecture to microservices and containerization. This has resulted in a dramatic change in the role and process around observability, with the lack of proper observability affecting engineering teams in many ways, including having zero transparency, limiting capacity planning and troubleshooting, and resulting in inefficient systems. To improve understanding of software systems, especially those running on Kubernetes, it is essential to focus on improving observability. Observability can be thought of at three levels: application-level, Kubernetes itself, and infrastructure parameters. Designing for optimal observability requires understanding how these three pieces interact with each other and form a complete stack. Various architectural patterns in software development, such as event-driven applications, can provide insights into designing for observability. Monitoring event-driven applications involves tracking events, their processing, and the overall health of the system. Ingesting application metrics data is also crucial, and Kubernetes components like kube-apiserver, kube-controller-manager, and kube-scheduler have built-in instrumentation that can be used to gauge the health of Kubernetes clusters. Observability has a significant impact on many areas across an organization, including improving performance, reducing mean time to resolution for issues, optimizing resource utilization, and aiding in cost savings. A troubleshooting platform like Lumigo can help developers monitor, trace, and troubleshoot issues fast, providing end-to-end virtual stack traces, API visibility, distributed tracing, and a unified platform to explore and query across microservices.

The new features in Lumigo aim to enhance collaboration among development teams for debugging and troubleshooting complex application issues. The tool offers message threading, which allows team members to start a new thread or add to an existing one on an issue, making it easier to investigate the root cause of problems. Additionally, threads can be marked as resolved, allowing the discussion to be closed and keeping the comments section tidy. Lumigo also includes screenshot annotation, enabling teams to attach annotated screenshots to their comments, providing valuable context for team members. These features contribute to a more successful and productive development team, better equipped to tackle challenging issues.

This article discusses the problem of slow-draining queues in microservice-based applications, particularly those using AWS Lambda functions and containers with AWS Simple Queue Service (SQS) message queueing. Repeated failures in processing SQS messages can lead to increased costs, reduced reliability, and hindered performance downstream components. Lumigo is introduced as a cloud-native observability platform that helps debug and solve this issue by providing distributed tracing capabilities, code error analysis, monitoring key metrics, log analysis, and more. A coded example using Node.js Lambda function demonstrates how to implement Lumigo tracing capabilities into the application, allowing developers to gain insight into message processing behavior and quickly pinpoint issues when they arise.