December 2018 Summaries

The New Relic platform generates a large amount of data about applications and environments, which can be transformed into valuable insights to make decisions, stay competitive, and maintain peak performance. New Relic Insights is a key tool for unlocking this value, providing analytical tasks, visualizations, and intuitive tools. To get started with Insights, it's recommended to follow best practices guides, learn NRQL query language, and understand how the four basic building blocks - NRQL, widgets, dashboards, and data apps - work together. The Insights dashboard API provides programmatic control over dashboards, making them accessible across teams. Understanding attributes, using the data formatter, and creating alerts are also crucial for getting the most out of New Relic Insights. With its flexible architecture, Insights supports custom data attributes, allowing users to build tailored dashboards and take advantage of standard analysis capabilities. The community-generated resources on the New Relic Explorers Hub offer a wealth of knowledge and best practices for New Relic Insights.

New Relic Insights is a comprehensive platform designed to help organizations harness the vast amounts of data generated by their applications and environments, providing actionable insights through data visualizations and intuitive tools. The platform leverages the New Relic Query Language (NRQL), which enables users to execute SQL-like queries to extract valuable business insights. Insights offers features such as dashboards, widgets, and data apps that facilitate the creation of tailored views of application performance and user interactions. To enhance usability, the platform supports API integration, allowing for programmatic control and sharing of dashboards across teams, reducing duplication of efforts. Additionally, Insights accommodates custom data attributes, expanding its analytical capabilities. Educational resources, including tutorials and documentation, are available to guide users in mastering the platform's functionalities and to encourage best practices in data management and analysis.

Kafka's auto-commit mechanism can lead to data loss and duplication if a consumer experiences an out of memory (OOM) kill or other hard shutdown, as it may reprocess messages that were received but not yet committed. This occurs because the consumer has no knowledge of what happens to the message after it is pulled in, and auto-commits can occur at intervals controlled by `auto.commit.interval.ms`, which can lead to hundreds of messages being lost or duplicated. To mitigate this risk, users can consider using streaming solutions like Flink or Kafka Streams that offer exactly-once processing, implementing their own transactional semantics in consumers, or accepting the risk and building reliable services with stable alerts for service shutdowns.

New Relic's use of Apache Kafka has unveiled challenges, particularly with Kafka's consumer auto-commit configuration, which can lead to data loss or duplication during consumer service failures like out-of-memory (OOM) kills. Kafka's auto-commit setting automatically commits consumed messages at set intervals, simplifying offset management but risking message reprocessing if a consumer fails. While manual offset management offers precision, it adds complexity. Kafka version 0.11 introduced improvements for exactly-once processing, yet it imposes additional latency. Alternative solutions, such as using streaming frameworks like Flink or Kafka Streams, offer exactly-once processing, although some users may accept minor data inconsistencies due to the engineering complexities involved in completely resolving the issue. New Relic emphasizes building stable services and considering other streaming systems to mitigate these risks while acknowledging the trade-offs between engineering cost and data reliability.

Gareth Rushgrove's DevOps Weekly newsletter has been a must-read for almost 30,000 subscribers over the past nine years, curating an eclectic mix of DevOps stories and resources that keep readers informed and up-to-date. The evolution of DevOps since its early days is broader now, with more people involved in discussions and a greater range of topics being covered. While DevOps has become a marketing success story, it's still largely community-driven and focused on collaboration between developers and operations teams. Gareth sees DevOps as an implementation of specific practices, rather than the same thing as site reliability engineering (SRE), which has also evolved but is distinct from DevOps. The biggest drivers for DevOps adoption are speed, security, and compliance, while barriers include people, process, and cultural issues, as well as procurement and budgeting challenges. To get started with DevOps, Gareth recommends listening to people outside your organization, attending local DevOpsDays sessions, and diving into a wealth of resources such as books and online communities. Comprehensive monitoring and continuous integration and deployment (CI/CD) are critical components of the DevOps mindset, which has evolved over time but remains focused on collaboration, automation, and continuous improvement.

Gareth Rushgrove's DevOps Weekly newsletter, curated from Cambridge, UK, has been a vital resource for nearly a decade, informing its 30,000 subscribers about the evolving DevOps landscape. While Gareth, a product manager at Docker, gathers content for the newsletter, he has witnessed the broadening scope of DevOps from a niche interest into a diverse community-oriented conversation, influenced by both development and operations perspectives. DevOps has become a marketing success, uniting professionals under a common banner while the introduction of practices like Site Reliability Engineering (SRE) from Google and the rise of serverless computing continue to evolve the field. Key drivers for DevOps adoption include the need for speed and competitive advantage, despite cultural and organizational barriers. Monitoring and continuous integration/deployment remain crucial components of the DevOps framework, facilitating rapid and secure deployments. Gareth emphasizes the importance of engaging with external communities and resources to inspire enthusiasm and drive organizational change, while acknowledging that DevOps, despite its marketing-driven visibility, was never about being "cool" but rather about inclusivity and collaboration in the tech world.

Edge computing is about putting time-sensitive operations closer to where they will have an impact, typically in a location that minimizes latency and maximizes efficiency. This approach contrasts with cloud computing, which collects and analyzes data in a centralized location for greater control and scalability. The choice between edge and cloud computing depends on the specific use case, considering factors such as performance requirements, data processing needs, and system management complexity. Autonomous vehicles, like driverless cars, often require edge computing due to the need for real-time processing of sensor data to ensure safety and responsiveness. Edge computing offers advantages like faster response times, increased reliability, and dedicated processing for specific tasks, but also presents challenges such as managing deployments, monitoring usage, and troubleshooting issues in a distributed environment. Successful implementation of edge computing requires careful consideration of use cases, DevOps principles, deployment automation, versioning, scaling, and monitoring to ensure efficient management of modern applications and their components.

Edge computing, as highlighted in this text, involves decentralizing certain computational tasks to locations closer to the data source or end-user, enhancing the efficiency and speed of time-sensitive operations. This approach contrasts with centralized cloud computing, where data is processed in a central location, offering more control but potentially introducing latency. Edge computing is particularly crucial in scenarios like autonomous vehicles, where immediate data processing is vital for safety and functionality. While edge computing provides advantages such as responsiveness and reliability in connectivity-limited situations, it also presents challenges, including managing deployments across numerous nodes and handling variable provisioning issues. The text also emphasizes the importance of using DevOps principles in edge computing to ensure efficient management and suggests best practices like automating deployments and understanding the scaling differences between cloud and edge environments. Ultimately, successful integration of edge computing involves balancing the unique advantages and challenges of both edge and cloud systems.

Werner Vogels, AWS CTO, delivered a keynote at re:Invent 2018, sharing his experiences and insights on building reliable, scaleable, and secure cloud systems. He discussed various topics including databases, serverless computing, development tools, and architecture best practices. Werner highlighted the importance of reliability, scalability, and performance in minimizing the impact of failures and emphasized that cloud-native databases are the foundation of innovation. He also introduced new features such as AWS Lambda Layers and the Firecracker virtualization platform, which offer better isolation and efficiency for serverless environments. Additionally, Werner announced several toolkits and services to support modern development, including IDEs, languages, and workflows. The keynote concluded with the introduction of Now Go Build, a video series featuring startups that use cutting-edge software powered by AWS.

AWS re:Invent 2018 featured a keynote by AWS CTO Werner Vogels, who reflected on AWS's journey and key innovations like AWS Lambda and serverless technologies. He recounted his worst day at Amazon in 2004 when a bug disrupted the customer database, influencing AWS's focus on reliability and scalability. Vogels highlighted the shift from traditional enterprise databases to cloud-native solutions, exemplified by Amazon's migration from Oracle to DynamoDB and Redshift. The event showcased AWS's extensive use in diverse applications, including Fender's music tutorials leveraging AWS Lambda and SageMaker. Vogels emphasized the growing enterprise adoption of serverless computing and introduced new tools for developers, such as AWS Toolkits for IDEs and custom runtimes for AWS Lambda. He also introduced the AWS Well-Architected Tool and announced "Now Go Build," a video series exploring innovative startups using AWS technology.