Debugging bad data in Elasticsearch and Kibana – a guide

Snowplow distinguishes itself by reporting not only successful data but also bad data that fails to process, allowing users to quickly identify and address data tracking issues. Recently, Snowplow enhanced its capabilities by enabling the loading of bad data into Elasticsearch, facilitating the use of Kibana for monitoring and diagnosing data issues. The Kibana Discover interface provides tools for viewing and filtering bad rows, helping to differentiate between genuine data tracking errors and irrelevant data such as those generated by malicious bots or necessary browser processes like OPTIONS requests. By filtering out these unimportant errors, users can focus on genuine tracking issues, diagnose their causes, and implement solutions, such as schema updates, to resolve them. This proactive approach to managing bad data ensures a high degree of confidence in the data trends, reflecting true business trends rather than data anomalies.