October 2023 Summaries

Honeycomb offers a generous free plan allowing users to send up to 20 million events monthly, with techniques available to manage data volumes without sacrificing observability. One such method, tail-based sampling, can be implemented using Honeycomb's Refinery, a container-based solution that reduces data sent to the backend, thus lowering costs. Tail-based sampling retains more critical data while discarding less useful spans by intelligently sampling traces, allowing users to maintain effective observability even at high data volumes. Refinery has been made more accessible with its recent updates and can be deployed using Docker, making it suitable even for those without extensive technical expertise. By implementing Refinery, users can manage observability costs effectively while maintaining the accuracy of trace data, although statistical accuracy improves with larger datasets. While the free tier provides significant functionality, Honeycomb's Pro tiers offer additional features like SLOs and Single Sign-On, which can further enhance data value. The decision to invest in paid plans should consider the total cost of ownership and the scale of data management needs, with Honeycomb's support teams available to assist in optimizing configurations.

Large Language Models (LLMs) are increasingly significant in software development due to their capability to enhance software systems, as demonstrated by Honeycomb's Query Assistant, which allows engineers to query systems using plain English. However, challenges arise due to the nondeterministic nature of LLMs, which can produce varying outputs for the same input. This has led to the development of prompt engineering, a technique that involves crafting specific prompts to guide LLMs towards desired outputs, albeit imperfectly. At Honeycomb, careful experimentation and iteration with prompts were necessary to optimize results for specific use cases, such as returning relevant product tags based on customer descriptions. Despite the added costs associated with more detailed prompts, this approach is crucial for improving the accuracy of LLM outputs. By incorporating observability, tracing, and A/B testing, developers can measure the effectiveness of prompts and iterate on them to enhance user experiences, while being mindful of potential issues like prompt injection attacks and the limitations of user input.

Software development is often perceived as being primarily for users, but it also deeply satisfies the intellectual curiosity of developers, who enjoy the challenge of making complex systems respond to their commands. This intrinsic joy, characterized by the 'aha' moments when solving difficult problems, can be scaled through tools like OpenTelemetry. OpenTelemetry, which is gaining significant traction and adoption among organizations, including Honeycomb's clients, offers developers a comprehensive framework for making systems observable by providing critical telemetry data. Honeycomb, a leader in observability, integrates OpenTelemetry to enhance developer experience, aiming to make it intuitive and delightful while reshaping the approach to system monitoring and observability. As OpenTelemetry evolves, its integration into popular frameworks and platforms is anticipated within five years, emphasizing the importance of keeping developer experience at the forefront. Honeycomb's commitment to OpenTelemetry involves not just improving its utility but also making it accessible and enjoyable for developers, SREs, and DevOps teams, thus fostering a shift in how systems are built and understood.

Honeycomb, an observability platform, is experiencing significant growth and success, evidenced by a record number of bookings in Q3 2023 and a Net Revenue Retention Rate of 138% up to September 2023. Despite economic challenges, the company is expanding its leadership team to sustain this momentum, with key hires including CMO Reet Mand, SVP of Global Revenue Jason Robinson, Director of Open Source Austin Parker, and Senior Director of Product Marketing Josh Pederson. These strategic appointments aim to enhance Honeycomb's market presence and capitalize on its success in helping engineering teams efficiently manage complex tech environments. The platform is recognized by companies like Vanguard, Slack, and HelloFresh for its ability to maximize tech investments, resolve incidents swiftly, and improve customer satisfaction. Honeycomb's growth reflects an industry shift towards modern observability solutions, moving away from traditional APM tools.

Kubernetes has become a widely adopted platform for managing software deployments and scaling applications, celebrated for its ability to standardize and containerize services across diverse infrastructures. However, the initial excitement has given way to challenges in day-to-day management, revealing a new divide between application engineers and platform engineers, reminiscent of past silos where teams blamed each other for failures. While Kubernetes simplifies deployment, it does not inherently improve application observability or debugging, often making it difficult to identify whether issues stem from code or infrastructure. The shift to Kubernetes has also highlighted the need for high-cardinality data to effectively monitor the dynamic nature of pods, which traditional metrics systems struggle to handle. To overcome these challenges, engineering teams must foster collaboration, using tools like OpenTelemetry to bridge the gap between application-specific and infrastructure-specific metrics, ensuring a seamless and integrated approach to delivering optimal customer experiences. Ultimately, Kubernetes is not a panacea for software performance, and achieving high-performing, collaborative teams requires intentional effort and culture building.

Understanding production and leveraging observability tools like Honeycomb can significantly enhance the capabilities of customer-facing teams such as field engineers, customer success managers, and customer architects. By democratizing data and providing access to logs, traces, and metrics, support teams can effectively diagnose and resolve user issues without unnecessary escalation to engineering, thus avoiding frustration and inefficiencies. Honeycomb empowers these teams to quickly identify root causes, validate improvements, and tailor client alerting, while customer success managers use it to maintain long-term client relationships by ensuring successful observability journeys. Moreover, product training workshops help users learn through hands-on experiences, and customer architects employ Honeycomb to offer personalized recommendations for optimizing client instrumentation. This comprehensive approach enables teams to better understand complex systems, improve customer experiences, and create valuable technical content, ultimately fostering a more autonomous and efficient support environment.

Honeycomb's integration of Large Language Models (LLMs) through their Query Assistant has been a strategic move to enhance user engagement and streamline querying processes on their platform. Initially launched as an experimental feature, Query Assistant quickly evolved into a core product after iterative improvements informed by production data. Despite its success in boosting activation metrics and remaining cost-effective due to OpenAI's pricing, it did not meet all expectations, particularly in engaging free-tier users. The tool has proven effective in facilitating the learning curve for new users by converting natural language inputs into complex queries, thereby encouraging more manual and sophisticated querying over time. Operationally, the use of GPT-3.5-turbo keeps costs low while maintaining functionality, and the tool has been praised by customers and sales teams alike for its role in onboarding and engaging prospects. Although some initial limitations were noted, such as its discoverability and feature set, user feedback has been instrumental in refining the tool, making it a valuable addition to the Honeycomb platform.