February 2024 Summaries

The most popular database is MySQL according to Stack Overflow's survey in 2021. People at smaller companies may prefer to work with React.js, Vue.js, Express, Angular, and jQuery for a variety of reasons such as wide recognition and support, streamlining the development process, and familiarity being an asset for developers. However, the choice of technology may also be influenced by project-specific requirements and limitations.

The February ClickHouse newsletter highlights the latest developments in real-time data warehouses. Key points include a deep dive into ClickHouse's architecture, the release of Grafana v4.0 plugin for improved SQL observability, and the 24.1 release featuring an experimental Variant type. Additionally, Benjamin Wooton is featured as the community member of the month, and upcoming events such as meetups and training sessions are announced.

The StreamKap connector for ClickHouse is a powerful tool that enables real-time analytics on streaming data. It provides efficient ingestion of CDC events, supports transformations in the pipeline, and handles schema evolution automatically. This connector can significantly enhance your ability to analyze and gain insights from large volumes of semi-structured or structured data in real time. The performance tests show that the StreamKap ClickHouse destination connector scales well with increasing numbers of tasks, making it suitable for both backfill operations and streaming changes. The ability to handle schema evolution automatically is a significant advantage, as it saves developers from having to manually manage these changes. Overall, the StreamKap connector for ClickHouse offers an efficient and flexible solution for real-time analytics on streaming data. It can help businesses make more informed decisions by providing near-real-time insights into their data.

Grupo Masmovil, a leading Spanish telecommunications operator, has successfully implemented ClickHouse to improve monitoring solutions for Radio Access Networks (RAN). The company migrated from Telecom-specific tools to open source technologies like ClickHouse, which enabled them to achieve significant improvements in data processing times, storage efficiency, and cost savings. By using ClickHouse's powerful SQL capabilities and real-time query performance, Masmovil has transformed its approach to monitoring RAN networks, resulting in more reliable solutions, faster results, and increased data retention.

The article discusses the challenges of managing data in machine learning, including the proliferation of specialized tools that can lead to increased architectural complexity and data costs. It highlights the potential benefits of using a single database or data warehouse like ClickHouse as the central datastore for the machine learning data layer. The author explains how ClickHouse can simplify infrastructure and enhance developer efficiency by handling tasks such as data exploration, preparation, feature extraction, training and evaluation, inference, vector store management, and observability. By using a single system like ClickHouse, users can avoid the need for multiple specialized tools and reduce maintenance costs, architectural complexity, and data duplication expenses.

ClickHouse version 24.1 introduces 26 new features, 22 performance optimizations, and 47 bug fixes. The release includes the ability to generate shingles, functions for Punycode, quantile sketches from Datadog, compression control when writing files, speeds up for HTTP Output and parallel replicas and memory optimizations for keeper and merges. Additionally, ClickHouse's popularity is growing due to contributions from the community. The Variant type has been introduced as an experimental feature, allowing users to work with maps more easily. String similarity functions have also been added, including Levenshtein, Damerau Levenshtein, Jaro Similarity, and Jaro Winkler. These algorithms can be used for spell checking, plagiarism detection, and other applications in natural language processing, computational linguistics, and bioinformatics. The vertical algorithm for FINAL with ReplacingMergeTree has been optimized to reduce cache trashing and improve performance.

This post is about a new file format called "Iceberg" which provides for efficient storage and retrieval of data in columnar databases such as ClickHouse. The author explores how to use SQL queries with this new file format, and demonstrates some examples using real-world datasets like the Ookla internet speed dataset. The post begins by explaining what Iceberg is and why it's useful for storing large amounts of data in a columnar database. It then goes on to describe how one can use SQL queries with this new file format, including details about importing and querying data stored in an Iceberg file. Next, the author delves into some more advanced topics related to working with geospatial data using UDFs (User Defined Functions) in ClickHouse. They show how to create a view that simplifies SQL queries for visualizing internet speeds across different regions worldwide. This involves computing centroid coordinates and generating colors based on download speed values. In conclusion, the post provides an overview of the Iceberg file format and demonstrates its utility for working with large datasets in ClickHouse. It also showcases some practical applications of using SQL queries to analyze geospatial data using UDFs.

The new ClickHouse Grafana plugin is designed for use in SQL-based Observability workflows. It includes a number of new features that significantly simplify the querying of logs and traces, while retaining the flexibility to query other data types such as time series. These features include auto-generated SQL queries based on user filters, drill down into spans associated with traces, and integration with AI services for natural language query assistance. The plugin is currently in Beta, allowing users to provide feedback and suggestions for future enhancements.