April 2022 Summaries

Acceldata Pulse 2.1 has been released, introducing new features such as support for Action on Kerberos Enabled Environments and a playbook to kill long-running non LLAP Tez queries. Other enhancements include the addition of multiple email addresses via CLI, Jira integrations, and UI improvements for YARN capacity. The update also includes various alerts like HBase Master Failover Alert, Interactive Endpoint Check Alert, and Interactive JVM Heap Usage Alert. Additionally, users can now access help guides and API documentation from the Pulse UI, and search options have been added to Nodes and Tez Queries pages.

Data marketplaces are emerging platforms that enable companies to sell their internally-generated datasets to other businesses, offering a great opportunity for any organization to maximize the value of its fastest-growing asset. Leading cloud infrastructure and database providers have created data marketplaces such as AWS Data Exchange, Snowflake Data Marketplace, Oracle Data Marketplace, SAP Data Warehouse Cloud Data Marketplace, Nokia Data Marketplace, Informatica Cloud Data Marketplace, and Databricks Delta Sharing. These platforms differ from their predecessors in terms of the buyers, sellers, technology, and economic opportunity they offer. While data marketplaces make it incredibly easy to get started selling data, companies need to follow best practices to ensure success. A multi-dimensional data observability platform can help data sellers overcome common issues such as poor data discoverability, untrusted data, managing schema changes, and building knowledge graphs of data producers, curators, and consumers.

Rohit Choudhary, CEO of Acceldata, discussed understanding consumer lag in Kafka clusters and its importance for real-time business insights. Consumer lag is a delay between a consumer's most recent committed offset and the producer's end offset in the log. It can be caused by complex logic, stuck consumers, slow message processing, or more messages produced than consumed. Rebalance events due to new consumer additions or crashed consumer processes also contribute to consumer lag. Acceldata provides a unified dashboard for Kafka monitoring and allows enterprises to scale technology adoption without operational blindness. The platform's Kafka dashboard displays metrics such as Kafka streaming details, producer, topic, and consumer information, as well as Kafka events and other charts to optimize Kafka analysis and reduce consumer lag.

Enterprises are collecting vast amounts of data, but its value depends on accuracy, consistency, and relevance to specific business issues. Poor data quality can lead to increased operational costs, security risks, and growth limitations. High-quality data must exhibit six characteristics: accuracy, completeness, consistency, freshness, validity, and uniqueness. Establishing effective data quality programs involves continuous validation, accurate reporting, and real-time alerting. Data observability solutions like Acceldata Torch can help improve data quality at scale.

Bad data can lead to poor decision-making, making data quality crucial for success in today's business landscape. To ensure high-quality data, modern companies often rely on data quality policies and rules. However, configuring and managing these policies and rules can be time-consuming and mentally exhausting without the right tools. Acceldata Torch leverages advanced machine learning and AI to accelerate data quality policy and rule creation. It starts by automatically profiling your data asset and providing an interactive statistical summary with intuitive charts and graphs for better understanding. This establishes a baseline for efficiently creating data quality policies and rules. After profiling the data, Acceldata Torch makes AI-powered recommendations to streamline the process of creating data quality policies and rules. It can recommend various rule definitions such as null values, enumerations, duplicates, and custom rules based on the data asset. This significantly reduces the time required for this task compared to manual efforts. Additionally, Acceldata Torch allows users to configure their data quality policy to run on a predefined schedule and provides options for receiving notifications about rule execution events via email or Slack. Users can also easily view, edit, or delete their data quality policies as needed. Overall, implementing Acceldata Torch can save time and effort in managing data quality policies and rules, allowing businesses to focus on making informed decisions based on high-quality data.

Netflix has one of the largest data infrastructures globally, with dozens of data platforms and hundreds of petabytes in its data warehouse alone. To manage costs effectively, Netflix built a custom "data efficiency" dashboard that provides comprehensive cost and performance transparency for all data users and teams. The dashboard offers real-time views depending on the user's role and helps identify bottlenecks and potential savings opportunities. This approach has led to significant reductions in data warehouse storage costs, proving effective in managing expenses within a massive data environment.

Acceldata CEO, Rohit Choudhary, was a speaker at the ScaleUp:AI Conference hosted by Insight Partners. He discussed how large enterprises are using AI to solve complex data-related business issues. PhonePe, a Walmart subsidiary with over 350 million customers in India for digital payments, embarked on a data infrastructure expansion project. By improving visibility into every aspect of the company's data operations through data observability and AI/ML, PhonePe achieved significant growth and improvements in availability, performance, and cost reduction.

The NFL Free Agency period is underway, with professional football players seeking better opportunities and higher salaries. In the corporate world, employee shuffling has taken on new dimensions due to the Great Resignation, particularly in data engineering fields. Companies are increasingly recognizing the importance of becoming data-driven enterprises, leading to a surge in demand for skilled data engineers who can manage and maintain data infrastructure. Data engineers have become more valuable than data scientists, as they play a crucial role in building and maintaining data pipelines, ensuring data quality, and monitoring overall data infrastructure. The field is highly competitive, with companies struggling to find qualified data engineers. To attract and retain top talent, businesses should invest in modern multidimensional data observability platforms that empower data engineers to become 10x engineers and superstars.

CDO Magazine has included Acceldata in its list of top 25 data startups, recognizing the company's rapid growth and product leadership in the data observability category. Acceldata aims to help enterprises transform their complex data environments into stable, agile, and cost-efficient systems by monitoring performance, usage, spend, and other factors. The company offers deep data observability, covering metrics, logs, and data quality, to improve reliability, accelerate scale, and lower costs for real-time AI and analytics workloads.

In an interview with InsideBIGDATA, Acceldata's VP of Growth, Loretta Jones, discusses the importance of modern enterprises understanding their data and creating effective, reliable data pipelines. She highlights the challenges faced by businesses in combining operational data from multiple sources and achieving complete data observability. Data observability allows companies to monitor critical processes for business continuity, identify problems early on, reduce downtime, and make informed decisions based on data. To address these complexities, a multidimensional data observability solution is recommended, which provides visibility into the state of data, supporting systems, and transforming systems across the entire data lifecycle. This helps businesses understand what's happening with their enterprise data, including processing status, data quality, pipeline interruptions, and economic factors.

Data observability is crucial for managing complex modern data environments, enabling control, optimization, and reliability of data operations. It provides benefits in five key areas: job latency, SQL analytics, Spark analytics, resource management, containerized deployment, data as a critical business asset, extensible platform, and automation. Acceldata's multidimensional data observability platform improves data reliability, reduces complexity, and scales data usage for enterprises to accelerate their digital transformation.