November 2016 Summaries

A study by Datadog shows that Amazon's EC2 Container Service (ECS) is steadily gaining popularity for Docker orchestration, with adoption among its customers rising from zero to 15 percent in the year since it was first released. ECS usage correlates with larger and more complex Docker infrastructure, as well as an increase in running containers and Docker hosts when companies start using ECS. The study also notes that while Kubernetes has a larger buzz and community, ECS is gaining significant real-world use.

Docker usage is increasing, with a significant number of organizations adopting and sticking to the technology. Amazon's EC2 Container Service (ECS) is gaining traction as a container orchestration tool, steadily increasing in adoption among Docker users, particularly those with larger and more complex infrastructure. As organizations grow their Docker infrastructure, they are also increasingly using ECS to manage their containers, often accompanied by an increase in running containers and Docker hosts. This trend suggests that orchestration is becoming more prevalent among organizations as they adopt containerization.

Datadog's Zendesk integration allows organizations to scale their support teams by automatically tracking customer experience and creating tickets based on telemetry data. The integration provides a clear picture of customer experience, enabling support teams to quickly troubleshoot issues. With the integration, customers' feedback is parsed more efficiently, reducing time spent investigating issues. By leveraging this integration, organizations can expedite support and improve overall customer satisfaction.

Today, we are open-sourcing Redux-Doghouse, a library that enables developers to create Scoped Actions and corresponding Scoped Reducers in Redux applications. This pattern is useful for creating reusable UI components with Redux logic that can be used multiple times in different contexts without conflicting with each other. The Doghouse library helps maintain the state of complex, interdependent parts of a UI across an application while allowing developers to compose unique scopes onto each instance of a component's action creators and reducers. This approach makes it easier to reuse code in various contexts within large applications, improving development efficiency and speed.

Redux-Doghouse is an open-sourced library that allows developers to create Scoped Actions and corresponding Scoped Reducers, enabling the creation of reusable UI components with Redux logic. This pattern helps keep complicated pieces of UI code separate from one another, making it easier to split reducers and actions into modules by component rather than view and model code. The library provides a way to compose unique scopes onto each instance of a component's action creators and reducers, allowing instances to behave independently while still responding to actions at a higher level. It can be used in scenarios where multiple instances of the same Redux-enabled component need to coexist without conflicting with each other.

Kubernetes is an open-source container orchestration system that automates the scheduling, scaling, and maintenance of containers in any infrastructure environment. It provides a conductor-like function to manage containers, ensuring they start, stop, create, and destroy automatically as needed. Kubernetes can orchestrate containers on multiple clouds, facilitating multi-cloud deployments and migrations between platforms. The key components of a Kubernetes architecture include pods, which are deployable units that bundle containers together, controllers such as Deployments and ReplicaSets, nodes, clusters, namespaces, and services. These components work together to provide high availability, scalability, and automation in containerized environments. Monitoring in Kubernetes requires a rethink due to its dynamic nature, and tags and labels play a crucial role in providing visibility into the cluster and applications. With its complex architecture, Kubernetes introduces new monitoring challenges, but a Kubernetes-aware monitoring tool with service discovery can help overcome these hurdles.

Monitoring a Kubernetes environment demands a distinct strategy compared to traditional VM-based or unorchestrated container approaches, due to its dynamic nature and reliance on abstractions like Deployments and DaemonSets. Kubernetes offers extensive APIs for automation and cluster management, facilitating performance data collection that falls into categories like cluster state metrics, resource metrics, and work metrics from the Control Plane. Key components include worker nodes that run containerized workloads and Control Plane nodes that manage the cluster, with tools like Metrics Server and kube-state-metrics enhancing data aggregation and accessibility. Monitoring memory, CPU, and disk usage, alongside Kubernetes events, is crucial for understanding resource utilization and maintaining cluster performance. The Control Plane's various services, such as the API server and etcd data stores, also emit important metrics that help track cluster health, while keeping an eye on specific metrics like memory limits, CPU requests, and disk utilization can prevent resource allocation issues. Collecting Kubernetes events complements this by offering insights into pod lifecycle impacts, and Part 3 of the series promises to delve further into leveraging Kubernetes APIs for comprehensive metric collection.

Datadog has integrated with Application Performance Monitoring (APM) service provider New Relic to provide full-stack observability. This integration allows users to monitor app performance and end-user satisfaction in context with the rest of their infrastructure. Datadog collects New Relic metrics such as Apdex score, request throughput, and average response time for monitoring app health and performance alongside other application and server metrics. Additionally, Datadog captures metrics from New Relic Synthetics to provide insights on end-user experience via availability and scriptable transaction checks from browser engines deployed worldwide. With over 650 integrations, users can easily correlate metrics and events between systems, allowing for quick identification of performance problems' root causes.

Datadog has unveiled its Application Performance Monitoring (APM) platform to complement its existing infrastructure monitoring capabilities, providing full-stack observability. This integration allows users to monitor app performance and end-user satisfaction alongside their infrastructure metrics, with the ability to integrate with APM service provider New Relic for seamless visibility into app-level metrics. The new platform features Apdex score, a metric that estimates user satisfaction level on an application's response time, as well as Synthetics, which monitors app availability and load time. With Datadog's integration capabilities, users can easily correlate metrics and events between systems, collaborate with teams to resolve issues, and share insights across the organization. This solution is designed to help development and operations teams improve observability of all system components and their interactions, enabling them to identify and address performance problems more efficiently.

When shards are lingering in an "unassigned" state in Elasticsearch, it can indicate various issues with the cluster's configuration or data. To identify the cause, Elasticsearch provides tools such as the cat shards API and the cluster allocation explain API. The primary shard may be unassigned due to delayed shard allocation, not enough nodes, re-enabled shard allocation, missing shard data, low disk watermark, or multiple Elasticsearch versions. In some cases, forcing allocation of a primary shard can solve the issue, but this should be done with caution as it may result in losing data on the original shard. To resolve unassigned shards, it's essential to address these potential causes and consider monitoring for such issues using tools like Datadog.