November 2021 Summaries

Healthcare industry is increasingly data-driven, with massive volumes and diverse types of data being generated by patients, physicians, institutions, and software providers. Managing this data effectively presents significant challenges that require the right technology. HealthEdge, a healthcare software company, transitioned from relational database management systems (RDBMS) to distributed data systems to keep pace with growing data volumes and develop innovative solutions for its customers. However, this introduced new levels of complexity in ensuring data reliability and recency. To overcome these challenges, HealthEdge implemented multi-dimensional data observability strategy using Acceldata Data Observability Cloud. This provided a 360-degree view into their data ecosystem, enabling efficient fine-tuning of data systems without digging through large amounts of log file data. The adoption of commercial Data Observability solution has expedited HealthEdge's data-driven transformation and provided a reliable partner for ongoing collaboration.

Snowflake and Databricks, two major players in the cloud data industry, are engaged in a battle for supremacy. Both companies have recently announced significant performance improvements over their rivals, leading to a war of words between them. However, these benchmark tests may not accurately represent real-world workloads, as they only capture a single scenario and do not account for the unique configurations and optimization requirements of individual businesses. To truly optimize data workload price-performance, companies should consider using multidimensional data observability platforms like Acceldata's Data Observability platform, which provides precise metrics and recommendations to improve infrastructure speed, reliability, and cost.

Traditional data catalogs are being phased out due to their limitations in handling modern data infrastructure. The rapid growth of unstructured and semi-structured data, increased demand for real-time analytics, and the constant transformation of data as it travels through pipelines have rendered traditional data catalogs ineffective. Data discovery is emerging as a solution that automates metadata harvesting, updates metadata in real time, and provides relevant results for users' data searches. Acceldata's Data Observability platform offers powerful data discovery capabilities for the modern data stack by constantly scanning, profiling, and tagging data throughout its lifecycle using machine learning.

Data observability is an approach that enables monitoring, detection, prediction, prevention, and resolution of problems across infrastructure, data, and application layers in real-time. Unlike APM tools, which mainly focus on the application layer, data observability platforms extend monitoring capabilities all the way down to the data and infrastructure layers. This helps improve control over data pipelines, create better SLAs, and provide insights for making better data-driven business decisions. Data observability solutions offer clear advantages over APM tools in terms of providing more control over data pipelines, improving infrastructure layer observability, and enabling automatic collection and correlation of pipeline events to identify anomalies or spikes.

Apache Spark is a popular open source distributed computing framework that enables data engineers to process large amounts of data across multiple machines. It is optimized for machine learning and AI, making it valuable in batch processing tasks. Traditionally, companies have used the Java Virtual Machine (JVM)-based Hadoop YARN to manage their Spark clusters. However, with the rise of Kubernetes and cloud-native computing, many organizations are moving away from YARN to Kubernetes for managing their Spark clusters. Kubernetes offers numerous potential benefits such as scalability, open source flexibility, and compatibility with various infrastructure types. The transition from YARN to Kubernetes can provide better dependency management, resource management, and access to a rich ecosystem of integrations. Key steps in this migration include determining the complexity of jobs, evaluating data connectivity needs, analyzing compute and storage latency, and auditing monitoring and security policies. Switching to Spark on Kubernetes can yield significant benefits for data engineers, including simpler dependency and resource management, value-added integrations, and cost savings opportunities.