March 2023 Summaries

Python projects can be challenging to manage as they grow in complexity. One way to overcome this is by structuring your code and project directories to make it easier to navigate and maintain the project. This includes organizing your code into modules, using consistent naming conventions, and utilizing version control systems like Git. Additionally, using tools like pip for dependency management, virtual environments for isolating dependencies, commenting code for readability, testing with automated tests, linting and styling with Ruff or Flake8, packaging to share with others, and structuring your project into a neat folder structure with key files such as `README.md`, `setup.py` or `pyproject.toml`, `tests/`, and `src/`. By following these best practices, you can make your Python projects more manageable, maintainable, and scalable.

A partitioned data asset is a way of modeling data that lies between a single monolithic data asset and a set of distinct data assets. Partitioning helps data engineers and ML engineers organize data and computations, making data pipelines more performant and cost-efficient by operating on subsets of data instead of all of it at once. Data orchestrators need to understand partitions to effectively run data pipelines, as they help answer questions like "what needs to be done?" and "what did I do?". Without an understanding of partitions, data orchestration can lead to uncertainty, less trustworthy data, and painful debugging. Dagster is a data orchestrator that strives to fully model the relationship between computation and data, including partitioned data assets and pipelines, making it well-suited for modeling partitions.

In this blog post, the authors explore the issue of fake GitHub stars and how to identify them. They use a combination of machine learning and heuristics to detect suspicious accounts and star patterns on GitHub repositories. The authors purchased fake GitHub stars from various vendors to gather data for their analysis and developed a simple heuristic to identify obvious fake accounts based on limited activity. However, they found that sophisticated fake accounts were more challenging to identify using this approach alone. To improve the detection, they used unsupervised clustering techniques to group accounts with similar behavior and identified suspicious repositories. The authors share their findings and provide an open-source solution in Python and dbt for others to analyze GitHub repositories and detect fake stars. They also discuss the importance of detecting fake accounts to maintain trust on GitHub and highlight that while building models for 100% accuracy is hard, techniques with high precision and recall can be developed, and simple heuristics can still provide valuable insights.

This latest major release of Dagster, version 1.2: Formation, introduces several significant enhancements to its partitioned data asset support and config and resources features, aiming to improve the overall developer experience. The new release includes dynamic asset partitions, updated UI display for partitioned assets, and improved backfill capabilities. It also introduces Pythonic config and resources, providing a more streamlined and reliable configuration API. Additionally, the release highlights updates to existing integrations with data warehouses such as Snowflake, DuckDB, and BigQuery, as well as new guides and tutorials to help users navigate these changes.

Dagster Cloud, a serverless development platform, aims to improve the deploy speed of Dagster code from over 3 minutes to around 40 seconds. To achieve this, they implemented a system that uses Python Executable (PEX) files, which bundle Python packages and some bootstrap code into executable files. This allows for faster builds and deploys by avoiding the need to build and launch Docker images. The PEX tool provides features such as isolation, determinism, composition, and cross-platform builds. By using PEX in combination with S3 storage, Dagster Cloud can significantly improve deploy speed without sacrificing repeatability or consistency. However, there are trade-offs, including potential memory issues if multiple environments are run on the same container, and limitations when building for non-Linux platforms. The implementation also replaces traditional Docker-based workflows with PEX-based ones, eliminating the need to download and launch Docker images.

Python packages are discussed in this article, focusing on managing dependencies for data people. Dependency management is crucial as it keeps track of other packages required to work correctly. The Python Package Index (PyPI) is a central repository of open-source Python packages that can be used to search and make packages available. Two methods of managing dependencies are explored: the old way using `setup.py` files, which were previously recommended, and the new way using `pyproject.toml` files, introduced as part of PEP 518 and PEP 621. The latter is simpler and easier to read than `setup.py`. Extras can be specified in both methods, allowing for optional features that require additional dependencies. Alternative tools like Poetry provide a more user-friendly approach with version constraint resolution and automatic virtual environment management. Virtual environments create isolated Python environments to allow different versions of Python and libraries to be used without interfering with each other. They are created using the `python -m venv` command, activated using the `source` command, and deactivated by typing `deactivate`. It's best practice to keep virtual environments in the same directory as the project and add them to `.gitignore` if they're in a Git repository.

Python packages are collections of files and directories that contain code, documentation, and other necessary files. They provide a way to share and reuse code in the Python community, making it easier to organize and maintain complex projects. A package is simply a directory with an `__init__.py` file that tells Python to treat it as a package. The `__init__.py` file can be used to initialize the package or set up its components. Python packages are managed using pip, which allows users to install and manage packages from the Python Package Index (PyPI) and other package indexes. Relative imports allow you to import modules relative to the current module, while absolute imports use the full path of the module or package being imported. Understanding how to manage dependencies effectively is crucial for writing robust and maintainable code in Python.