We've recently completed a large beta test for our new product, a test management tool, and learned important lessons about running successful beta tests with error reporting, specifically using Sentry. During the test, we received mostly user feedback and questions about missing features, but few actual problems and bugs were reported. We implemented Sentry error reporting throughout our stack to receive additional useful metrics, including chat notifications in Slack when issues arose. This allowed us to identify and fix issues quickly, even when users didn't report them directly. By leveraging Sentry's breadcrumb feature and detailed event trails, we were able to diagnose complex problems like a rare data query deadlock. The experience showed that with robust error reporting in place, beta tests can provide valuable feedback beyond user reports, helping ensure software quality and customer loyalty before launch.