January 2022 Summaries

The article explores the distinctions between batch processing and stream processing in handling data, highlighting their respective use cases and advantages. Batch processing involves the sequential and simultaneous handling of large volumes of data, typically executed at the end of a business cycle, such as payroll processing, data cleansing, and ETL operations, often taking hours or days to complete. Conversely, stream processing analyzes and manages data in real-time, allowing for immediate evaluation as events occur, which is ideal for applications like real-time anomaly detection, IoT event processing, and personalized user experiences in connected devices. While batch processing is characterized by handling finite and static datasets at specified intervals, stream processing deals with continuous, dynamic, and infinite data flows, offering faster insights and responses. Transforming batch data into stream data requires continuous data packet transmission and real-time analysis, facilitated by platforms like Kafka and Pulsar. Ultimately, the choice between batch and stream processing depends on the need for speed and real-time interaction versus the security and completeness of data processing.

GQLAlchemy 1.1 introduces several new features to enhance its functionality as a Python Object Graph Mapper (OGM), similar to SQLAlchemy for relational databases. This update simplifies the process of working with graph databases by allowing users to store Python objects directly into a graph database without using Cypher, and it includes automatic schema validation, serialization, and deserialization. It also offers on-disk storage solutions for large properties, enabling the storage of extensive data without managing separate storage solutions. Additionally, GQLAlchemy now supports integration with streaming and triggers, allowing users to handle data streams and database triggers programmatically in Python. Future plans for GQLAlchemy include supporting the execution of Python functions within a graph database, which will enable the setup of custom triggers and modules.

Katarina Supe's tutorial on building a Twitch analytics app using GQLAlchemy demonstrates how to efficiently manage graph-based data and applications. By employing GQLAlchemy, an object graph mapper (OGM), developers can create data models and manage interactions with the Memgraph database, ensuring data accuracy through schema validation. The guide outlines the process of loading Twitch data from a CSV file, using GQLAlchemy to define data classes and load data into the database. It also covers creating an API endpoint to determine the most popular games by counting the number of streamers who play them. The tutorial emphasizes the use of GQLAlchemy's query builder to simplify database queries and prevent errors, showcasing a method to return results in descending order of popularity. Additionally, the tutorial includes a brief overview of a frontend implementation using Semantic UI React, providing an example of how to visualize the data by fetching it through the componentDidMount method and rendering it using the games and players state variables. The tutorial concludes by highlighting the benefits of using GQLAlchemy to interact with graph databases and encourages users to engage with the community for support and inspiration.

Memgraph, an in-memory graph database, has introduced a solution for handling large properties that aren't used in graph computations through its GQLAlchemy library, which provides on-disk storage support. This is achieved by using the SQLitePropertyDatabase to store extensive metadata separately from the graph database to enhance efficiency and manageability. The GQLAlchemy library, built on Pydantic, functions as an Object Graph Mapper (OGM) that connects graph database objects with Python objects, allowing users to define which properties are stored on disk or in memory via model definitions. This approach alleviates the burden of manually implementing separate storage solutions for large properties, thereby improving performance and simplifying project maintenance. Through the use of the on_disk argument, users can automate the storage process, ensuring that large properties are efficiently managed without consuming excessive RAM, thus speeding up graph algorithms.

Real-time analytics is a powerful tool that enables organizations to process and analyze incoming data instantly, providing immediate insights to drive timely decision-making. This capability is particularly beneficial for businesses dealing with large volumes of data, such as eCommerce platforms, financial services, and customer relationship management systems, allowing them to personalize user experiences, detect fraud, and optimize operations. Implementing real-time analytics requires a clear understanding of business needs, precise identification of data sources, a robust data infrastructure, and the selection of an appropriate analytics tool. By integrating real-time analytics with big data, companies can efficiently process vast data sets, yielding actionable insights that enhance decision-making processes. As the number of IoT-connected devices rises, the demand for real-time analytics is expected to grow, making it an essential component for businesses aiming to remain competitive and responsive in an increasingly data-driven world.

GQLAlchemy enhances the functionality of Memgraph by allowing Python developers to programmatically manage data streams and database triggers, eliminating the need to directly use Cypher query language for these tasks. This integration facilitates connecting to data streams from platforms like Kafka, Pulsar, and Redpanda, and running graph analytics on the data, while also simplifying the creation and management of database triggers for operations such as CREATE, UPDATE, and DELETE. Triggers enable custom notifications and the execution of graph algorithms post-data updates, which can be extended with Python-written query modules for further data analysis or integration with external systems. This approach streamlines workflows for Python developers in the graph database environment, offering a more accessible and flexible way to handle dynamic data interactions and analytics through GQLAlchemy's interface.

Memgraph offers a comprehensive benefits package designed to enhance employee well-being and productivity, including private health insurance to facilitate regular checkups, a gym allowance, and increased vacation time. Flexible working hours allow employees to tailor their schedules for optimal productivity, while the company also supports remote work with a stipend for home office setup. Team bonding is encouraged through virtual or in-person gatherings, and employees can participate in a share options scheme as part of their compensation. Performance bonuses and a refer-a-friend program further incentivize and reward staff, reflecting Memgraph's commitment to a supportive and engaging work environment.

The blog post by Katarina Supe details the creation of a React app integrated with WebSocket to visualize real-time streaming data using D3.js, focusing on Twitter users who retweeted with the hashtag #christmas. The application is part of a dockerized system consisting of five services, including a Python script for collecting retweets, a Kafka cluster for data streaming, and a Memgraph platform for analytics. The frontend, built using create-react-app, employs D3.js to draw and update graphs based on dynamic community detection and PageRank algorithms. The process involves setting up WebSocket connections within React component lifecycle methods to handle real-time data updates and using D3.js for rendering and updating SVG elements. The blog emphasizes the learning experience gained from developing this application and acknowledges resources such as a Reddit network explorer for additional insights into real-time visualization techniques.

Mutimir 2021, organized by the Croatian nonprofit NGO Penkala, is an annual scientific conference aimed at promoting academic activities and fostering interactions between young researchers and private industry representatives. Attended by representatives from Memgraph, the conference provided a platform for Croatia's emerging academic talents to present diverse research topics, such as the ecological impacts of fire and innovative engineering concepts, while also highlighting the role of private companies like Gideon Brothers, Visage Technologies, PhishAR, and Omnisearch in advancing applied science. The event emphasized the importance of strengthening the collaboration between academia and industry, with attendees expressing a commitment to participating in future editions to support the ongoing development of this vital relationship.