September 2021 Summaries

Acceldata has successfully closed its $35 million Series B funding round led by Insight Venture Partners. The company's innovative data observability platform has garnered interest from new investor March Capital and existing investors Lightspeed, Sorenson Ventures, and Emergent Ventures. This investment will be used to enhance the capabilities of Acceldata's Data Observability Cloud, expand sales and marketing efforts, and grow its team. The company aims to support data-driven transformation for enterprises by providing deep visibility into data interoperability, compute performance, and data pipelines.

Apache Spark is a leading tool for data engineers working with large datasets, offering efficient and easy-to-use solutions for managing and processing massive amounts of data from multiple sources. Launched in 2013, Spark has become an essential part of the data engineer's arsenal, particularly as enterprises face increasing challenges in data management and governance. Spark is a unified analytics engine designed to rapidly query, analyze, and transform large-scale data. It originated from the AMPLab at the University of California, Berkeley, before being donated to the Apache Software Foundation in 2013 and becoming a top-level project in 2014. The Apache Spark community is diverse and includes commercial providers such as Databricks, IBM, and Hadoop vendors. Data engineers use Spark for various tasks, including stream processing, machine learning, interactive analytics, and data cleansing. It can handle petabytes of data across thousands of servers and offers a core data processing engine with additional libraries for SQL, machine learning, graph computation, and stream processing. Spark's core engine is optimized to run in memory, enabling faster data processing compared to alternatives like Hadoop MapReduce. It can be used with various languages and storage systems and runs on different cluster managers, making it a versatile tool for managing complex data environments.

Data processing with Apache Spark can be optimized by monitoring its performance, especially when running on Kubernetes. Spark on Kubernetes became generally available in March 2021, and companies are adopting this approach to improve their infrastructure's efficiency. Monitoring tools like Pulse integration provide an overview of Spark jobs, job status, memory usage, and other metrics. Accessing Spark Metrics can be done through the Spark UI or REST API, which returns JSON data for easy visualization and monitoring tool integration. The Kubernetes Dashboard offers basic metrics but may not directly link them to specific Spark jobs. Acceldata's Pulse integration allows users to create custom dashboards with various metrics and visualizations, improving Spark observability on Kubernetes.

Modernization in the context of data management can involve various approaches such as replacing existing technology with new ones or moving to the cloud. The benefits of modernization include business outcome focus, increased scalability and elasticity, and accelerated speed of execution. However, wholesale changes like a complete "lift and shift" to the cloud may not always be effective. Some common pitfalls in modernization efforts include losing control, retaining redundant data, migration costs, tech sprawl, data migration risks, data swamp, uncontrolled resource usage, and cost vs benefit analysis. Data observability can help reduce risk in a modernization strategy by providing end-to-end visibility into data, processing, and pipelines.

Data Observability is an emerging concept that extends from Application Performance Monitoring (APM) and focuses on improving complex data environments by monitoring external outputs, analyzing them, and taking action based on the insights gained. It emphasizes three key elements: monitor, analyze, and act. The approach complements or exceeds existing technologies in Compute Performance Monitoring, Data Quality and Governance, and Data Pipeline Management. By observing everything and engineering only as needed, Data Observability can help manage complex data operations more effectively and efficiently.

Spark is popular for its ease-of-use, speed, and power in large-scale distributed data processing. However, it can face operational challenges due to misuse by users. Common issues include data skew, executor misconfiguration, join/shuffle operations, and memory problems. To address these issues, developers should ensure proper data partitioning, configure the right number of executors based on workload and data spread, optimize shuffle operations, and manage memory usage effectively. By addressing these common issues, Spark performance can be improved, and operational tasks can be made more efficient.

The text discusses the challenges faced by large enterprises in managing their data effectively due to insufficient discipline and lack of visibility. It highlights that over 70% of employees have access to unauthorized data, and 80% of analysts' time is spent on manual data analysis due to poor initial data quality. The text emphasizes the importance of data observability in monitoring and managing data at scale across hybrid data lakes and warehouses. It also distinguishes between data sources and their derivatives, and explains the significance of data quality for various stakeholders like data producers, consumers, and critical data elements. Furthermore, it discusses different types of data outages and testing methods to ensure data quality throughout the data lifecycle. The text concludes by stating that data observability can help solve data quality issues and suggests using Acceldata's platform for automating data quality and reliability at scale.