May 2022 Summaries

Many enterprises still rely on Hadoop clusters and other on-premises databases for mission-critical analytics, but with the imminent end of Cloudera support for Hortonworks Data Platform (HDP) and Cloudera Data Hub (CDH), organizations are looking for alternatives. The Acceldata platform and its team can provide a safer, more efficient, and less expensive way to continue running CDH, HDP, or the open-source version of Hadoop for years to come while preparing infrastructure for an eventual migration to a modern cloud-native database. Acceldata's six-step migration process includes Proof of Concept, Preparation, Data Migration, Consumption, Monitoring, and Optimization. The platform provides recommendations for Snowflake best practices, cost intelligence dashboards, data inventory, performance baselines, deep insights into the performance of Snowpipe and COPY commands, data reconciliation, automatic discovery and profiling of data assets, continuous data observability, and powerful Cost Optimization tools for Snowflake to aid in data value engineering initiatives.

The panel session during the CDO & Data Leaders’ Summit 2022 focused on the importance of developing effective data environments to drive growth. Led by Acceldata Co-founder and CEO, Rohit Choudhary, participants discussed the characteristics needed by today's data leaders to create a strategy that takes a holistic approach to computational and logical elements powering data capabilities. The discussion emphasized the role of data in organizations, how to create effective strategies for deploying data resources, and the impact of data quality and reliability on overall business success.

A startup accidentally processed test code more than a hundred billion times, resulting in a five-figure bill instead of the expected $7. Such mistakes are not the only reason for unexpected cloud infrastructure costs; suboptimal resource allocation and lack of effective management can also lead to significant expenses. Modern cloud computing services like Snowflake make it easy to develop and scale data-intensive applications, but proper management is crucial to avoid unwanted costs. A multidimensional data observability solution can help by wiring up the entire data environment for better examination, understanding operations that cause high spending, and optimizing data operations. Acceldata's cost intelligence dashboard automatically aggregates usage across all Snowflake services and assigns a dollar-cost value, offering granular breakups of cost trends across each service. It also allows users to filter their spend by each service and dig deeper into more granular costs. Automatic cost anomaly detection minimizes unwanted expenses by flagging workload spikes that fall beyond average usage boundaries. Capacity planning helps predict cloud resource consumption, identify under- or over-utilization trends, and optimize annual cloud contracts.

Torch offers a data reconciliation policy feature that ensures consistency in data quality across different assets. This helps establish, validate, and maintain the quality of data contained within various sources. The data reconciliation policy allows users to customize based on specific data they want to use for each source. It also enables data teams to join both data sources based on an ID column provided and evaluates conditions specified by the user. Acceldata provides a way to check the integrity of migrated data by comparing source and target datasets, as well as performing RCA on migrated workloads that are not functioning as expected. To configure a reconciliation policy in Torch, users can follow specific steps outlined in the text, including setting up Source & Sink Asset Info, defining rule definitions, scheduling execution, and configuring alert notifications.

Facebook has built a next-generation storage system called Tectonic to manage its massive data infrastructure. The company previously had three data storage infrastructures tailored for each use case, but these systems were complicated and time-consuming to manage, created operational complexity, and led to wasted resources. Tectonic was designed as a unified file system for all of Facebook's data, allowing it to avoid wasteful resource overprovisioning and fragmentation while making the system easier to manage. The design enables IOPS or storage to be shared among different workloads that would otherwise be wasted in smaller clusters. Tectonic was also built with scale in mind, as each cluster can be multiple exabytes in size, large enough to serve an entire Facebook-sized data center.

Hadoop is still a popular choice for big data analytics, and many Fortune 500 companies continue to use it on-premises. However, commercial support for Hadoop platforms like Cloudera Data Hub (CDH) and Hortonworks Data Platform (HDP) will end soon, leaving customers with the challenge of migrating their analytics stack off Hadoop or managing it in-house. A data observability platform can help companies optimize their Hadoop infrastructure and manage it more efficiently, potentially saving costs and improving performance. Acceldata is one such platform that has helped many Fortune 500 companies successfully manage their on-premises Hadoop clusters.

Spotify has an impressive infrastructure for handling its massive user base and streaming service, with 406 million active users and 180 million paying subscribers. The company uses machine learning (ML) projects to analyze user interaction data and system data, including listening history, likes, and page load times. To handle the scale of this data, Spotify has upgraded its Event Delivery Infrastructure (EDI) twice in the last six years, moving from an on-premises system to Google's Cloud Platform (GCP). The company also built a new data processing engine called Scio and switched to a more powerful data orchestration platform, Flyte. Spotify continues to innovate and improve its infrastructure to maintain its position as the dominant music streaming service.

The modern data stack is transforming the way companies collect, ingest, and analyze data, offering speed, agility, and cost savings compared to traditional on-premises systems. However, this comes with its own set of challenges such as creeping complexity, performance bottlenecks, data errors, and uncontrolled cloud costs. The modern data stack typically includes a cloud data warehouse, data integration service, ELT data transformation tool, BI visualization tool, and reverse ETL tool. Despite the benefits, it is crucial to monitor and manage this complex machine to prevent issues from arising. Data observability platforms can provide real-time insights into data performance, reliability, and costs, helping companies maintain a healthy and efficient modern data stack.