July 2024 Summaries

The cryptocurrency market has seen significant growth, with over 22,000 active cryptocurrencies and a total market capitalization exceeding $2.5 trillion USD. Stablecoins have also experienced massive growth, with transfer volume peaking at over $150B a day in May 2024. However, accessing this data in a consumable fashion is complex due to the lack of public repositories for fundamental metrics like those found in traditional financial markets. Blockchain nodes are responsible for validating transactions and maintaining the blockchain ledger, making it easier to participate in the ecosystem, but financial reporting remains complex. An ecosystem of reporting firms has emerged to provide real-time crypto data, with Artemis being a notable provider, offering cleansed, structured, aggregated data to various consumers. Artemis aggregates data from multiple blockchains and provides it through its Artemis Terminal, API, and data shares on Snowflake, making crypto data instantly consumable for institutions, research analysts, and investors. The company leverages an MDS tech stack, favoring composable, open-source technologies like Dagster+, which has streamlined their data operation and provided a more intuitive framework for pipelines. Artemis has also taken the bold step of publishing its open-source dbt models to address analyst concerns and align with the culture of decentralized digital currency. The company's focus on growth and openness positions them well to support the evolving cryptocurrency market.

Petal, a FinTech firm, adopted Dagster as a data orchestrator to incrementally improve their data operations. Initially, they retained parts of their existing pipeline and adapted them to work with Dagster. The team is building out the constructs and plans to adopt more capabilities over time. Petal also uses dbt Core for data transformation and has integrated Dagster with dbt for scheduling and collaboration. By adopting Dagster, Petal aims to streamline their pipeline, derive value from event-based triggering, and create a dependency graph. The company's goal is to provide a modern data-centric tool for more context-aware orchestration.

Eunice Ho, the Operations Lead at Dagster Labs, shares her experiences and insights about the company culture. She emphasizes the importance of celebrating successes together, fostering a high-trust culture by being transparent and open in decision-making, and maintaining core values that guide the team's decision-making process. The company prioritizes continuous growth, investing in employees' skills through training and mentorship programs. Dagster Labs is committed to building an inclusive and supportive environment where everyone feels welcome, with a focus on remote work flexibility, allowing teams to collaborate efficiently across different time zones.

Dagster and Great Expectations are used to improve the reliability of data pipelines by integrating data quality checks. This allows for the detection of erroneous or malformed data before it is propagated downstream, ensuring that stakeholder-facing products such as dashboards and reports receive accurate and reliable data. By combining Dagster's Asset Checks with Great Expectations' extensive suite of data quality tests, developers can simplify their testing logic without sacrificing the reliability of their data pipelines. The integration enables users to define expectations for their data assets using a more straightforward approach, reducing the amount of custom code required for testing and maintaining data quality.

Zippi, a Brazilian fintech firm, adopted Dagster to build a reliable and adaptable data platform. The company, which provides financial products tailored to micro-entrepreneurs and small businesses, initially focused on operationalizing its machine learning model for loan underwriting analysis. With Dagster, Zippi was able to deliver faster insights with less downtime, reduce maintenance costs, and gain greater visibility into its data assets. The team implemented a centralized platform, fostering collaboration and autonomy among data practitioners and stakeholders. By running its ETL service on Dagster, Zippi unified its data stack, eliminating local scheduling tasks and one-off Python scripts. Today, the team has full observability across the data platform, monitoring, and alerting in Slack, making it easier to diagnose and troubleshoot issues. The company plans to continue building out more integrations, automating steps of the process, and remodeling its pipelines following a reorganization at Zippi.