September 2022 Summaries

AllTrails has adopted Hex for exploratory analysis, data validation, and web apps to improve their workflow. The company uses a blend of SQL and JavaScript in Google Dataform to build dynamic data pipelines on the Google Cloud Platform. Analysts at AllTrails leverage tools like Amplitude, Hex, Preset, and Looker to deliver compelling stories to stakeholders across the organization. Hex has enabled robust exploratory analyses by allowing analysts to write SQL queries directly and store them in hosted notebooks for easy access and visualization of data from their warehouse (BigQuery). The platform also supports dynamic markdown variables, enabling updating values in summaries and comments.

Data notebooks are a powerful tool for data analysis and exploration. They allow users to run code, examine the output, and iterate on their work in a single document. Notebooks have become an integral part of data workflows at companies like Netflix, Lyft, PayPal, and Bloomberg. The key features of notebooks include: 1. Running code: Users can write and execute Python code within the notebook interface, using all the same libraries as in a regular Python environment. 2. Exploring data: Notebooks allow users to run lines of code in isolated cells, making it easier to see the output of just those lines and iterate on their analysis. 3. Presenting results: Users can incorporate visuals from the data outputs directly into the notebook document, alongside text and other images, to present a full story. Notebooks are designed for an iterative process of running code, exploring data, and presenting results. They enable users to take everything they do in a data exploratory pipeline and bundle it into a single document: the notebook. Notebooks can be used with various programming languages by changing the kernel, making them versatile tools for data analysis.

Hex has launched two new capabilities: first-class integrations with Airflow and Dagster, and a public API for programmatically running Hex projects from other systems. These features allow users to customize their integration of Hex into larger stacks, workflows, and systems. The orchestration support includes the ability to trigger project runs within DAGs (Directed Acyclic Graphs) in Airflow and Dagster. Additionally, a new public API provides four endpoints for running projects programmatically, including RunProject, GetRunStatus, CancelRun, and GetProjectRuns. These features are available on Hex Teams plans and above.

Lead scoring is a methodology used by companies to prioritize leads based on their likelihood of converting into customers. It involves using data and predictive models to rank leads, allowing sales teams to focus on the most promising ones. To build an effective lead scoring model, it's crucial to understand the business context, stakeholders' needs, and available data. Key steps include collecting relevant data, training and iterating on predictive models, assessing performance using appropriate metrics, collaborating with stakeholders, and deploying the final model into production. Regular monitoring and iteration are necessary to ensure the model remains effective over time.

Hex is a platform that aims to streamline fragmented analytics workflows by making them collaborative and easy to share with stakeholders, earning it the comparison to Figma for data. Similar to how Adobe's local-file-based tools were not ideal for collaboration in design teams, powerful but individualistic tools like Notebooks, SQL IDEs, and scripts have made collaboration difficult among Data Scientists and Analysts. Hex addresses these issues by being web-based and collaborative by default, allowing users to work together, peer review data projects, and share their work with other stakeholders in an organization.

This post discusses three methods for querying data in a Pandas DataFrame using SQL, with the assumption that the reader already has a DataFrame ready to go. The first method involves using the pandasql package, which is a simple wrapper that converts DataFrames into SQLite and allows them to be queried with SQL. Results are automatically returned as a Pandas DataFrame. The second method uses DuckDB, an in-process OLAP system designed for analytics workloads. It works natively with Pandas DataFrames and is optimized for speed when compared to SQLite. The main difference between pandasql and DuckDB is that the latter requires explicit conversion of results to a DataFrame. The third method involves using the .query() function in Pandas, which is not exactly SQL but can make some basic queries easier. It's a simple WHERE or .filter() equivalent. The query syntax is modified Python without many specifics, and it can be used for more detailed information on Pandas parsers and numexpr.