September 2024 Summaries

The text discusses the integration of Neo4j Graph Database and Analytics with a recommendation engine called Neo4j Keymaker to boost sales in retail industries. The solution architecture involves gathering data from various sources, developing a graph data model, populating interconnected data in a graph, building recommendations using graph analytics algorithms, and consuming recommendations through an API. The text also provides details on how the Neo4j Keymaker engine works, including its phases, scoring model, and weight allocation. Additionally, it describes a Salesforce integration that leverages Apex methods to call the Neo4j Keymaker engine's GraphQL API and display recommendations in a Lightning Web Components (LWC) template. The text concludes by highlighting the benefits of using Neo4j Graph Database and Analytics for sales optimization and customer satisfaction.

This blog post describes how to build a GraphRAG agent using the Neo4j graph database and Milvus vector database. The agent combines the power of graph databases and vector search to provide accurate and relevant answers to user queries. It uses a routing mechanism to decide whether to use the vector database or the knowledge graph, and includes fallback mechanisms in case the initial retrieval is insufficient. The agent also evaluates its own answers for relevance and accuracy, and can refine its search or attempt to correct errors if necessary. By combining the strengths of graph databases and vector search, this agent provides a more nuanced understanding of the information and leads to more accurate and nuanced answers.

Text embeddings excel at encoding unstructured text but struggle with structured data operations like filtering, sorting, and aggregating. To overcome these limitations, knowledge graphs and structured tools can be used to provide precision and flexibility in RAG applications. The proposed solution involves using tools designed for structured data, such as Cypher queries, to address complex user queries that require metadata filtering, sorting, and aggregation. By combining structured data approaches with unstructured text search techniques, more accurate and relevant responses can be delivered, enhancing the user experience in RAG applications.

The author used Neo4j, a graph database, to model and analyze data from the Titanic's passenger manifest. They started by creating nodes for each passenger, relationship connections between them, and properties of relationships such as ticket type and embarkation point. However, they soon realized that categorizing the data differently was necessary to simplify queries and better understand the social implications of the data. The author restructured their graph to make more data into properties of nodes, allowing for easier querying and analysis. They extracted specific data from the CSV, such as cabin and boat side information, and used it to answer questions about survivorship based on factors like gender, age, and side of the ship assigned a cabin or lifeboat. The analysis revealed that females had a higher survival rate than males, children had a very high survival rate regardless of their gender, and there was little difference in survivorship related to which side of the ship each passenger was assigned a cabin. Additionally, the data suggested that adult males who tried to escape from the port side did so in boats with limited capacity. The author emphasizes the importance of continually investigating and augmenting data to gain deeper insights into complex social issues.

The Titanic passenger manifest records a wealth of information about passengers, including their relationships with each other. By leveraging graph database technology and Cypher queries, researchers can extract complex familial relationships from the data, asking questions such as "What proportion of surviving women were single or married?" or "Did those who survived have siblings aboard?" The graph allows for flexible querying, enabling the analysis of multiple properties of individuals, including their age, ticket number, and relationship categories. These queries reveal insights into survivorship rates, with children being more likely to survive than adults, and men having lower survival chances than women without children. The study highlights the power of graph databases in extracting complex relationships from relational data, enabling new lines of inquiry into historical events.

The HybridCypherRetriever enhances hybrid search by adding a graph traversal step. It combines vector search with full-text search, then uses a specified Cypher query to retrieve additional information from the graph for each node retrieved through hybrid search. This retriever is particularly useful for answering complex and nuanced questions that require both contextualized information and precise matching of exact words or phrases. By combining vector search and graph traversal techniques, the HybridCypherRetriever provides a more comprehensive solution than either method alone, enabling users to retrieve valuable information from their Neo4j databases. The retriever can be used to answer questions like "What are the names of the actors in the movie set in 1375 in Imperial China?" and returns accurate results by combining vector search with full-text search and graph traversal.

Neo4j has introduced significant changes to its platform, including the decoupling of server versions from Cypher query language versions, allowing users to choose between different versions independently. This change enables flexibility for database administrators, application developers, and security professionals by providing a seamless transition process. The new approach also introduces calendar-based versioning, where releases are versioned with a format like YYYY.MM.Patch, enabling regular access to new features, simplified upgrade paths, and optional long-term support. Additionally, Neo4j has implemented a thoughtful deprecation policy, prioritizing customer needs and minimizing disruption, ensuring a smooth transition for users as the platform evolves.

Neo4j's graph technology integrates with AWS to tackle fraud with advanced pattern recognition, reducing false positives and transforming financial security. Financial institutions struggle with identifying and thwarting fraud due to high false positive rates relying on traditional rule-based approaches and relational technologies. Graph technology excels in revealing hidden patterns within data, effortlessly uncovering complex fraud patterns. Neo4j's graph capabilities shine in detecting fraud by uncovering connections that link individual data points. Advanced pattern matching using Neo4j Cypher query language helps track down entire complex and deep money trails, detect circular money flow, and identify suspicious patterns. Graph Data Science algorithms provide powerful tools for analyzing graph data efficiently, uncovering hidden patterns, and making informed decisions. Feature engineering transforms raw graph data into meaningful inputs for ML models, while data visualization enables exploration and investigation of data to drive meaningful outcomes within organizations. Effective data loading into Neo4j is crucial for optimal performance and efficient querying, with techniques including base nodes first, keeping it simple, and efficient initialization.

Generative artificial intelligence (GenAI) is transforming various industries with its ability to analyze large data sets, inform strategic decisions, and streamline processes. However, executives, data scientists, and developers face ethical concerns, such as hallucination, bias, and lack of transparency, which can lead to inaccurate or misleading information. To address these challenges, companies are adopting guidelines and regulations, including the European Union's Artificial Intelligence Act, which distinguishes AI from traditional software and defines it as a machine-based system that may exhibit adaptiveness after deployment. Companies can apply principles such as "Do No Harm," "Be Fair," "Ensure Data Privacy," "Honor Human Autonomy," "Be Accurate," "Be Transparent," and "Be Accountable" to ensure the ethical use of GenAI technology. By incorporating these principles into their development process, companies can build trust with their customers and stakeholders while also fostering innovation and success.

GraphSummit Europe 2024 is an upcoming conference in London on October 16-17, focusing on graph technology, connected data, and GenAI. The event will feature presentations from industry leaders such as BMW Group and the London Stock Exchange Group, who will share insights into their use of graph technology to solve complex data challenges. Attendees can expect hands-on learning opportunities through technical workshops and interactive training sessions, allowing them to gain practical skills in implementing graph technology and connected data solutions in their own organizations. This conference is designed for data professionals looking to explore the future of data-driven decision-making and stay ahead of the curve, with space limited and demand high, so registration is recommended early to secure a spot.

Join us in London on October 16 and 17 for GraphSummit Europe 2024, a conference focused on graph technology, connected data, and GenAI. The event will feature presentations from industry leaders such as BMW Group and the London Stock Exchange Group, who will share insights into their use of graph technology to overcome complex data problems and achieve remarkable results. You'll have opportunities to learn through technical workshops, interactive training sessions, and hands-on learning experiences, and can explore how GenAI and graph technology are reshaping entire industries. This conference is designed for data professionals looking to stay ahead of the curve in data-driven decision-making and will provide a great opportunity to network with leading organizations and data practitioners.

Knowledge graphs can be a powerful tool for transforming invoice data and lowering costs in business operations. They organize information in a web of relationships between entities, allowing for the visualization and analysis of how they interact. By applying knowledge graphs to invoice data, companies can spot trends, detect anomalies, and understand patterns in their procurement and supply chain. This enables them to negotiate better prices with suppliers, lower supply-chain risks, and identify high-centrality suppliers. For example, a company found that purchasing 33,000 mayonnaise containers from a single supplier resulted in savings of around $410,000 if purchased at the lowest price. By leveraging knowledge graphs, businesses can make data-driven decisions to optimize their relationships with suppliers and reduce costs.

The GraphRAG patterns described in this text are a set of retrieval strategies for advanced RAG systems that leverage graph structures for more effective retrieval. The most basic pattern, the Basic Retriever, uses vector similarity search on chunk embeddings to retrieve relevant chunks. Intermediate patterns like Parent-Child Retriever and Hypothetical Question Retriever build upon this by incorporating additional context or relationships within the data. Advanced patterns like Graph-Enhanced Vector Search and Global Community Summary Retriever use graph structures to provide more comprehensive context for answering questions. Each pattern has its own set of required pre-processing steps, graph patterns, and retrieval queries, making it essential to experiment with different patterns to find the most suitable one for a specific application. The journey to discovering ideal GraphRAG patterns is ongoing, filled with trial, error, and innovation.

This blog post details how to build a movie recommendation system using Neo4j. The focus is on recommending movies to users based on their reading histories and ratings. It covers the setup of Neo4j, mapping data into Java with Neo4j Object Graph Mapper (Neo4j-OGM), and crafting Cypher queries for recommendations. Additionally, it includes guidance on setting up and using Neo4j Aura, Neo4j’s cloud-hosted database service. The post provides a step-by-step guide to creating a scalable recommendation engine that adapts to users' preferences and behavior. It emphasizes the use of Neo4j's graph database capabilities with Neo4j-OGM in Java to deliver personalized suggestions by leveraging inherent relationships in data. By integrating Neo4j Aura for cloud hosting, the system ensures robustness and scalability, allowing it to manage growing datasets effortlessly.

Fine-tuning` is an approach that involves supervised training of a large language model (LLM) to optimize its performance on a specific task or domain. This method adapts the internal knowledge of the LLM for a particular task, reducing hallucinations and providing more accurate outputs within its domain. However, it requires retraining the model for different tasks and domains, making it less flexible than `retrieval-augmented generation` (RAG). RAG uses an external information retrieval system to access up-to-date information from various sources like databases or APIs, allowing LLMs to tap into a vast pool of real-time data. This approach is more accurate, especially when combined with graph databases, as it provides better adaptability to new domains and evolving knowledge. Moreover, RAG offers several advantages over fine-tuning, including access to the latest information, insights into relationships and data points used by the model, and a level of explainability that fine-tuning lacks.

Mix and Batch is a technique used in Neo4j for faster data loading, reducing the time it takes to load large datasets. The use of Knowledge Graphs has been shown to improve the performance of Large Language Models (LLMs) by providing up-to-date information and reducing AI hallucinations. Graph analysis was applied to the 2024 Olympics, revealing hidden patterns and connections between nations and sports. A CSV file can be converted into a Neo4j graph database using various methods, including the use of RAG system. The power of graph databases is being showcased in various projects and applications, such as data loading, knowledge graphs, and analysis of complex networks like the Olympics.

The text discusses the challenges of making sense of complex, interconnected information in various fields. It introduces a novel method that uses metadata-driven ontologies for more consistent graph construction, demonstrated through an exploration of Supreme Court case data. The approach aims to address limitations of earlier AI systems, such as Retrieval-Augmented Generation (RAG), by combining structured knowledge retrieval with the generative power of large language models (LLMs). GraphRAG is a step forward in this direction, representing knowledge as a graph structure and enabling multi-hop reasoning and contextually relevant results. However, it relies on LLMs for graph construction, which introduces limitations such as inconsistencies in the graph structure and propagation of LLM biases or errors. The text proposes an alternative approach that uses metadata-driven ontologies to guide the construction of a knowledge graph, allowing for structured exploration of data without relying on LLMs. This approach enables focused relevance, enhanced coherence, and efficient exploration, making it a promising path towards agentic graph exploration.

The Neo4j GraphRAG Python package is used to build a GraphRAG application that leverages both vector search and full-text indexes. The HybridRetriever class combines the strengths of both indexing methods, using vector search for semantic similarity matching and full-text indexes for lexical similarity matching, such as dates and names. This approach enables accurate retrieval of relevant information even when user queries include specific strings or have different meanings in a wider context. By combining the capabilities of vector search and full-text indexes, the HybridRetriever class provides a more comprehensive solution for GraphRAG applications than using vector search alone. The package code is open source, and users can find it on GitHub, where they are invited to share their insights via comments or on the GraphRAG Discord channel.

Neo4j has introduced AuraDB Business Critical, a cost-effective service tier for mission-critical applications that require advanced security, scalability, and support. This new service offers enterprise-grade features at over 20% less than Neo4j AuraDB Virtual Dedicated Cloud. It provides a unified data management hub with Aura Console, which administers, models, and visualizes data across Neo4j tools, along with a GenAI copilot to enhance productivity. Additionally, Neo4j has introduced NeoDash, an interactive low-code dashboard builder that creates maps, charts, tables, and other visualizations of graph data. The company also offers expanded vertical scaling with up to 512GB AuraDB instances for large datasets and applications, as well as the ability to process up to 15x more data per cluster with read-only secondaries, enabling horizontal scaling without compromising latency or performance.

The Neo4j Python Driver has been improved with Rust extensions, offering significant performance improvements. The new driver is up to 10 times faster than the previous version and is available on GitHub and PyPI. The performance boost was achieved by rewriting a hot path in the driver code that involved encoding and decoding between Python types and the binary protocol called PackStream. Benchmarking tests were conducted using various workloads, including sending much data and receiving little data, as well as sending little data and receiving much data. The results show a speedup of anywhere between 1.16 and 4.26, with more data transmission resulting in larger speedups. The new driver is designed to be flexible and can be used for various use cases, including number-crunching tasks. However, it's essential to note that the border between Rust and Python lies in the implementation detail, which may affect users relying on driver internals.

The Query API is now available on Aura across all tiers and cloud providers for production workloads, allowing users to query Neo4j databases via HTTPS. The new feature uses secure HTTP(S) communications to make Cypher requests with a JSON document response, eliminating the need for drivers or custom libraries. Users can submit simple queries, such as retrieving data from an instance using CURL, and parameterized queries to load bulk data in a single request. The Query API also supports advanced features like transaction management, secure JWT authentication, and performance improvements. With its ease of use, security, and efficiency, the Query API is designed to be an accessible method for working with Neo4j Aura databases.