September 2022 Summaries

The PICERL model is an effective method for managing data quality incidents, consisting of six steps: Prepare, Identify, Contain, Eradicate, Recover, and Learn. Firstly, prepare by considering the bigger picture and minimizing data quality issues through people, processes, and technology. Next, identify the issue by gathering evidence, evaluating its impact, and determining its root cause. Then, contain the incident to prevent further escalation and notify stakeholders if necessary. After that, eradicate the problem by resolving it or delegating the task to someone else. Recover by testing your data pipeline to ensure it's functioning properly again. Finally, learn from the incident by hosting a retrospective with your team and tracking relevant metrics consistently.

The Data Observability Playbook provides best practices for data engineers to effectively utilize and grow with their data observability tools. Key considerations include identifying goals, needs, and use cases before purchasing a tool, configuring the tool according to company requirements, and establishing advanced tests on vital data assets during implementation. Post-implementation best practices involve putting processes in place, giving feedback to improve the model, and expanding use cases as business grows.

Metaplane has introduced the ability to add tests directly from table and column pages on their app. Users can search for tables or columns using ⌘+K (or Ctrl+K on Windows) and view available test types in the "Tests" section. By clicking "+ Add test," Metaplane will start training a new model for the selected table. This update enhances data observability and allows users to monitor their data more effectively.

Developers often debate whether to buy or build the software they need, and data engineers are no exception. There is no one-size-fits-all answer to this debate. Instead, you must consider all relevant factors when making a decision. This guide analyzes the pros and cons of building your own tool, leveraging an open-source tool, or buying an out-of-the-box tool. Factors considered include time and money, expertise and support, customization, and compliance. The recommended order of operations is to look into commercial options first, open-source options second, and in-house options third. Ultimately, the decision depends on your specific needs and resources.

Data observability is often misunderstood, but it's essential for companies that leverage their data for operational or decision-making purposes. Contrary to popular belief, data observability isn't only for big teams or big data; even small teams can benefit from it. Monitoring just one system isn't enough; you need end-to-end visibility into your entire data pipeline. Full test coverage from day one is unnecessary; start with the most important assets and expand over time. Additionally, not all data observability platforms are expensive or require a long implementation period.

The Metaplane app has been redesigned with a new homepage that features a prominent search bar, allowing users to quickly access the data they need. Additionally, the homepage provides an overview of failing tests and integration statuses. Users can also add Metaplane to various platforms like Alfred, Chrome, or Raycast by using its custom search URL.

Data observability tools are becoming increasingly popular among businesses that rely heavily on data for decision-making and operations. However, not every company needs such a tool. Here are six signs you might need one: 1. Your data directly impacts the company's bottom line. The more your business relies on data to drive operations, the more valuable a data observability platform becomes. 2. Stakeholders frequently flag issues with data quality. A data observability tool can notify you instantly of any anomalies and help you investigate their root cause and impact. 3. Your team often operates in the dark when it comes to understanding your data infrastructure. Data observability platforms centralize metadata, providing actionable insights that improve efficiency and effectiveness. 4. You rely on intuition rather than data-driven decisions for setting priorities. A data observability tool can provide the necessary metadata to prioritize work and allocate resources with confidence. 5. Your team is growing rapidly. Data observability tools can help new employees understand your data infrastructure, improve their experience, and reduce turnover. 6. Your team isn't growing fast enough. Even if you're not expanding your team, a data observability tool can still maximize the leverage of your existing team members by taking menial work off their plate and enabling them to act more quickly.