May 2022 Summaries

Robert Cooke, CTO & Founder of 3Forge, emphasizes the importance of unified solutions that cut across siloed automations, as companies seek to balance increasing IT costs with reduced budgets. He notes that opportunities for development work now exist globally, particularly in Asian markets, and companies are embracing a "follow the sun" model to increase productivity. To achieve this, Robert advocates for a data-agnostic mentality, leveraging commonalities across problems to find platforms and solutions that can be reused across an organization. Instead of replacing legacy systems with new ones, he recommends building on top of existing systems or using tools like reconciliation to ensure seamless transitions, while aiming for hybrid solutions that align with the market's direction and incorporate cloud pathways.

ChaosSearch commissioned the 2022 Data Delivery and Consumption Patterns Survey to understand how modern enterprises are reimagining and reconfiguring their data environments for the future. The survey of 209 IT and data managers reveals significant findings, including that enterprises still rely on data warehouse solutions despite the challenges they pose, with 82% using them in business-critical applications and 36% relying exclusively on data warehouses. The survey also shows that data environments are moving to the cloud, with 73% of respondents running their data warehouses on on-premise systems but planning to move them to the cloud over the next year. Additionally, the survey highlights increasing data delivery issues across all types of data environments, with 65% of respondents reporting an increase in these issues over the past three years. The most significant challenge for management is data preparation, with 53% of respondents citing this as a major issue. Finally, the survey reveals that data proliferation is driving high costs and low data reliability, with 52% of respondents having multiple copies of the same data across their enterprise. Overall, the survey provides valuable insights into how enterprises are preparing for a data-driven future.

A supercloud is an architecture that taps into the underlying services and primitives of hyperscale clouds to deliver additional value above and beyond what's available from public cloud providers. It's a custom cloud built for specific use cases, such as managing data, IT automation, wireless networks, and more, often leveraging existing environments like Amazon Web Services or Google Cloud Platform. Superclouds can be built upon a single public cloud platform or within multi-cloud environments, and their components add value to the public cloud platform or make multiple clouds function better than any individual cloud platform could alone. By using a supercloud approach, organizations can experience benefits such as improved automation, application functionality, efficiency of certain workflows, reduced risk of failures or cybersecurity attacks, and more. Supercloud services like ChaosSearch unlock the value from existing cloud object storage resources, allowing analysts to get insights quickly and ask different questions about their data without relying on a data engineer to move the data or make copies for analysis. The future of superclouds is intertwined with other concepts such as data mesh and data lakehouse, which aim to prevent organizations from building data pipelines and slowing down analytical processes.

You can build all sorts of amazing solutions to capture data, but if you can't show business value, then it's all pointless.” Raheem Daya emphasizes the importance of setting a clear vision for your team and tying data to actual outcomes to become meaningful. A manager's number one job is to take care of their team members, and empowering engineers with knowledge and decision-making skills is crucial. When choosing technology, consider whether it solves a business problem and provides flexibility for future growth. The democratization of data is changing the user experience, with users wanting to own and understand their data, and businesses seeking access to meaningful insights in near-instantaneous speed. To succeed, focus on the end user and their demands in a remote-first world, ensuring that people are engaged and getting what they need when they need it.

StreamSets is a modern DataOps Platform that helps customers build resilient data pipelines to enrich enterprise logs and other data before ingesting it into their data lake(s). ChaosSearch, on the other hand, is a cloud data platform designed to solve data lake analytics at scale, leveraging public cloud architecture. The two tools complement each other by providing data enrichment and smart pipeline capabilities that enable customers to ingest and analyze data from virtually any source, centralize data in cost-effective Amazon S3 cloud object storage, and activate data for multi-model analytics at scale. StreamSets solves the problem of data drift detection and provides support for multiple data formats, making it an ideal complement to ChaosSearch. By combining the two tools, AWS customers can transform their data into actionable insights that inform business decision-making.

Many enterprises face challenges when building data pipelines in AWS, particularly around data ingestion. StreamSets and ChaosSearch can optimize your AWS data ingestion pipeline processes, offering a streamlined solution to handle both structured and unstructured data for efficient analytics. The complexity of AWS data pipeline architectures requires careful design to ensure scalability and efficiency. Tools like StreamSets and ChaosSearch introduce features to help enterprises maximize efficiency and gain deeper insights, such as data drift detection, data enrichment, and schema-on-read approach. Combining the strengths of these solutions gives teams an end-to-end solution for building and maintaining resilient data ingestion pipelines in AWS.

### CloudWatch is a valuable tool for monitoring and logging AWS workloads, but it has limitations such as cost, retention, and only supporting AWS services. To extend log analytics beyond CloudWatch, teams should centralize data across clouds, collect and analyze all log data, store data efficiently and cost-effectively, fine-tune alarms, and make log data actionable with more powerful tools like ChaosSearch. By doing so, they can achieve a complete log analytics and cloud monitoring strategy that offers features like multicloud support, sophisticated alert configuration, storage flexibility, and more.

In today's digital landscape, data leaders face significant challenges in managing modern-day data. According to Kevin Petrie, Vice President of Research at Eckerson Group, a common destination for cloud data platforms is the cloud, where silos can be broken down and standardized on one version of the truth. However, migration complexity is a major issue, with data gravity and sovereignty requirements posing significant challenges. To overcome these obstacles, organizations are adopting hybrid models, leveraging software as a service options, and utilizing cloud-native applications. The use of developer tools such as Apache Airflow, Fivetran, Jupyter, and PyTorch can help bring data together, but there is also a need to consolidate tools and reduce proliferation. As the pandemic continues to impact the tech world, organizations must navigate an enormous increase in digital signals and optimize their data usage to generate revenue and reduce costs, while avoiding the risk of data becoming a liability. Observability has become a core tenet of machine learning platforms, with five subdisciplines identified, including business monitoring, operations observability, data quality, pipeline health, and model observability. Ultimately, humans will remain in control of machines, but they need to work together to achieve productivity and success.