May 2022 Summaries

The modern data platform and its use cases are complex, leading to broken dashboards and "data downtime" - a period of time when data is partial, erroneous, missing, or inaccurate. Data observability, an organization's ability to understand the health of its data across the entire system, can reduce data downtime by providing automated monitoring, alerting, and triaging to identify and evaluate data quality and discoverability issues. By connecting to existing stacks without modifying pipelines or writing new code, data observability solutions monitor data at rest with minimal configuration and reduce time to detection and resolution. Data health analytics also play a crucial role in measuring performance, setting SLAs, and optimizing resources. However, data observability is different from testing or monitoring, as it provides end-to-end coverage, scalability, and lineage for impact analysis. When combined with reverse ETL, data observability can make data more actionable, reducing the likelihood of severe events like data downtime and fostering a culture of innovation and data-driven excellence.

The Summer of Data` is a promotional campaign by Census, a data integration platform, aimed at helping users stand out in their data careers through events, community engagement, and exclusive summer swag. The company has introduced new features such as warehouse-centric observability capabilities to ensure data quality for reverse ETL, role-based access controls, and expanded connectors with popular business tools like Braze and HubSpot. To celebrate the launch, Census is hosting various live sessions, including a guide to operationalizing customer churn prediction and a showcase of how payments platform Spreedly uses Census with Looker for marketing personalization. Additionally, users can leave reviews on G2 in exchange for a $50 gift card, and the company is hiring new talent to join its team.

Census has introduced role-based access controls, allowing data teams to securely share sensitive business data with end users while maintaining control over data quality and security. This feature enables admins to assign user roles, empowering better collaboration between data and operations teams, and reducing the risk of data breaches or unauthorized changes. With this release, Census provides a more flexible and secure way for businesses to manage their data warehouse, improving compliance, operational efficiency, and cost-effectiveness.

You can rely on forecasted time series data in various aspects of life, including daily routines and business operations. Forecasting is crucial for businesses to predict demand, optimize inventory, and minimize losses due to unsold merchandise or missed profits. Exponential smoothing is a widely used time series forecasting method that estimates future values based on past observations, with a critical difference from simple moving averages in assigning exponentially decreasing weights over time. There are three types of exponential smoothing models: single (simple) exponential smoothing, double exponential smoothing, and triple exponential smoothing, each addressing different scenarios such as no seasonality or trend, no seasonality but with a trend, and both trend and seasonal patterns respectively. Simple exponential smoothing is used for data with no seasonality or trend, requiring a single smoothing parameter to control the rate of influence from historical observations. The equation for simple exponential smoothing predicts future values based on the most recent actual value and predicted value, using a smoothing factor (alpha) that controls the level of smoothing between 0 and 1. In Python, simple exponential smoothing can be implemented using the `SimpleExpSmoothing` class from the `statsmodels.tsa.api` module, with steps including importing necessary packages, creating an instance of the class, setting the smoothing factor, fitting the model to the data, and forecasting future values.

Census has announced a set of warehouse-centric observability capabilities that enable data teams to track and fix issues in their reverse ETL pipelines with confidence. The platform provides advanced tools such as Sync Logs, API Inspector, Sync Dry Runs, Custom Alerting, and Invalid/Rejected Records to help data teams understand the health of their pipelines and identify potential issues before they become major problems. With Census's observability suite, data teams can sync data into downstream business tools with confidence, without having to spend time debugging or troubleshooting, allowing them to focus on delivering high-quality data that drives business insights and action.

Facebook offers a large audience for promoting products, but advertisers need to target the right people with the right message. To do this, businesses can sync data about potential customers between their Snowflake data warehouse and Facebook Ads using various methods. The manual method involves creating an audience from scratch, while the automated method uses Census reverse ETL to optimize and trust the automatic updates. Alternatively, businesses can use Facebook's Python API to sync data directly. To set up this process, advertisers need to create a Facebook Developer Account, app, and access token, and then configure settings such as audience name and retention period. The automated method with Census offers convenience and simplicity, but requires verification that the information is updated correctly without maintenance required.

Bulk updates are essential for maintaining accurate business data, but manual methods can be time-consuming and prevent data teams from focusing on more critical tasks. Bulk update functionality, such as HubSpot's built-in features and third-party tools like Census, can help speed up these processes, allowing data teams to focus on revenue-generating work. Census is a reliable way to import customer data from multiple sources into HubSpot, addressing the pain point of collecting data from various tools and databases. By using Census and following an eight-step process, data teams can implement bulk updates and radically change their data approach, eliminating manual work and improving efficiency.

DevOps and DataOps are two frameworks that aim to improve the speed and reliability of software development and data management processes, respectively. Both share core principles such as continuous improvement, customer-first mentality, identification and elimination of waste, and process-focused simplification and standardization. However, they differ in their focus and approach, with DevOps handling code and DataOps handling data. DevOps has a more established pipeline that focuses on automation, while DataOps is still evolving and requires careful planning and execution. To succeed in DataOps, teams must adopt an automated mindset, stay up-to-date with new tools, prioritize innovation, and focus on delivering value to end-users. By understanding the similarities and differences between DevOps and DataOps, organizations can leverage these frameworks to streamline their processes, improve efficiency, and drive business growth.

The private sector has an abundance of data, but public sector organizations often start with a blank slate, collecting data only after identifying a specific question to answer. This approach allows for more streamlined decision-making and resource allocation, as they don't have the luxury of having pre-existing data infrastructure. Businesses can learn from this approach by focusing on big-picture thinking, selecting key performance indicators (KPIs) carefully, and being judicious with resources. By treating fast data as a privilege and embracing imperfection, companies like ClickUp are finding ways to balance speed and efficiency in their data operations, ensuring quality over quantity of data is prioritized. Regular communication and careful gatekeeping of analytics requests also play crucial roles in optimizing data resource allocation.