November 2023 Summaries

The latest version of Neo4j 5 Enterprise Edition (v.5.14) introduces a new storage engine called Block format, which significantly increases the efficiency of graph-native storage. The new format lays out data on disk in a structure that reduces fragmentation and improves performance by up to 40% compared to the existing Record format. With better memory usage, processing queries requires less memory, and storage is more compact, allowing for faster query execution. The Block format beta version is now available, with the general availability (GA) release expected early in 2024. Users can migrate or copy their databases to the new format using various commands, and performance tests show significant improvements over Record format across different scenarios. Despite some limitations, such as offline conversion for large databases and lack of incremental bulk import, the Block format is a major leap forward in graph-native database technology.

NODES 2023 was a significant event that brought together thousands of graphistas worldwide, with over 90 graph experts sharing their knowledge and experiences. The conference accumulated an impressive 492 days of watch time across its 20-hour duration. The event featured various sessions, including presentations on creating graph dashboards, improving data quality, unleashing the magic of user-defined procedures, and more. Additionally, Sudhir Hasbe, CPO of Neo4j, introduced the Neo4j Product Vision and Roadmap, announcing new product capabilities. All sessions are now available to watch, and a curated playlist with over 100 videos is also available for those who missed some sessions or want to revisit their favorites.

The text covers various topics related to Neo4j graph databases. Py2Neo, a Pythonic interface for Neo4j, has reached its end-of-life and is no longer maintained. The official Neo4j driver has taken over its responsibilities. Meanwhile, developers can explore new features such as visualizing DateTime data with Neo4j, clustering graph data using the K-medoids algorithm, and automating development workflows with GitHub Actions. Additionally, there are updates on upcoming events, including NODES 2023, GraphSummit Series, and Connections conferences in different regions. The text also highlights a speaker, Chris Zirkel, who will showcase how to streamline development with GitHub Actions at the NODES conference.

Neo4j and Amazon Web Services (AWS) have formed a multi-year collaboration agreement to accelerate enterprise AI development by addressing key challenges such as LLM hallucinations. The partnership integrates Neo4j's graph database with AWS Bedrock, enabling the creation of knowledge graphs that capture the contextual depth of enterprise data. This integration helps reduce hallucinations, creates personalized experiences, and provides complete answers during real-time search. The combination also unlocks unstructured data, providing accurate and contextual responses grounded in facts rather than LLM guesses. As a result, organizations can tailor GenAI solutions to their specific data domains while making AI results more complete, accurate, and transparent. Additionally, Neo4j AuraDB Pro, a fully managed graph database offering, is now available on the AWS Marketplace, simplifying the process of building knowledge graphs and enriching GenAI projects.

This week's edition of This Week in Neo4j covers various topics including advanced ingestion strategies for RAG applications, new ways of handling constraints with Cypher, a podcast summarizing highlights from NODES 2023, and an open-source Pipeline Security Tool built with Neo4j. The featured speaker for the upcoming GraphSummit Series is Oleg Šelajev, who will discuss better integration tests for Neo4j applications using Testcontainers. Additionally, there are summaries of blog posts on advanced RAG strategies and new property type constraints in Cypher. The podcast also provides an overview of NODES 2023, while a new open-source tool called Raven scans GitHub workflows for vulnerabilities in CI/CD pipelines and stores them in a Neo4j database.

The Neo4j Needle StarterKit is a template designed to accelerate development of applications using Neo4j databases. It provides a responsive design, theme flexibility, and modular architecture to reduce development time and improve Time To Value (TTV). The starter kit includes key features such as responsive design, dark/light mode theme, Neo4j integration, auto-login, side navigation, and modular approach. The architecture is modular, with each component serving a specific purpose, including the App, QuickStarter, Header, PageLayout, Content, ConnectionModal, and Driver. The starter kit can be used to build applications with Neo4j by following a simple installation process and utilizing code snippets provided. With its responsive design, theme flexibility, and modular architecture, the Neo4j Needle StarterKit is an excellent starting point for developers looking to build applications with Neo4j.

The k-medoids algorithm is a clustering technique that can be used on graph data. It offers advantages over traditional k-means for explaining cluster differences and can be applied directly to graph data by calculating the shortest paths between nodes. The key difference between k-medoids and k-means lies in how centers are defined, with medoids being actual data points rather than centroids. K-medoids is particularly useful for graph data because it allows for easy interpretation of clusters through their central reference points. However, it can be computationally expensive due to the need to calculate all pairwise distances between nodes. The Neo4j library provides a implementation of k-medoids that can be used on graph data, including a distance array generation function and a function to write results to Neo4j. The algorithm's effectiveness depends on the value of k, which should be chosen based on the silhouette score of each node in the cluster. A high silhouette score indicates a well-defined cluster with a dense central region and a gradual decrease in similarity towards the periphery. The algorithm can also be used on other types of data, such as genetic data, but may require adjustments due to differences in connectivity and data distribution. Overall, k-medoids is a valuable clustering technique that offers advantages over traditional methods for explaining cluster differences and can be applied to a variety of graph data sets.

This approach uses Large Language Models (LLMs) in conjunction with Graph technology to analyze annual reports, a task that was accomplished previously using Natural Language Processing (NLP) APIs. The new method allows for more detailed breakdowns of what the model should focus on, framed by business context and without delving into document complexities, making it easier and more agile to extract valuable insights aligned with specific business goals. By designing graph models tailored to the desired information extraction, the approach enables a more dynamic and responsive data extraction process, allowing for swift refinement of the model through prompt revision. The results significantly exceeded initial expectations, providing a deeper understanding of the information contained in annual reports, thereby enhancing the quality and depth of insights derived from the data.

Neo4j has made significant advancements in its capabilities with the introduction of new features such as Vector Search, LangChain integration, and Embeddings for improved Image Retrieval. These innovations enable more efficient semantic search over unstructured data, revolutionizing image retrieval by leveraging embeddings and cosine similarity. Additionally, Neo4j is expanding into new areas, including GIS Plugin Building and Document QA with VertexAI, while also hosting various events and conferences, such as NODES2023 and GraphSummit Series. The company is seeking input from its users through the Neo4j Research panel, offering a chance to connect directly with product development teams and influence future products, and has featured a speaker, Sharmistha Chatterjee, who discussed graph algorithms for privacy and fairness in the healthcare industry. Furthermore, Neo4j's Vector Index implementation is being explored further with customization options available through LangChain, and users are encouraged to share their experiences and participate in various activities, including a tweet of the week highlighting knowledge graphs as a potential future direction for AI knowledge.

The new features introduced in Neo4j 5 allow for more precise type checking and enforcement of data quality. Users can now restrict types allowed to be stored under a property using type constraints, which provide better control over data consistency. The `valueType` function has also been added, allowing users to get the most precise type assigned to a value. Additionally, Neo4j's type normalization process has been improved, making it easier to compare and analyze types. These new features can help improve data quality by enforcing constraints on specific property types, such as integers or strings. Users can try out these features in their own databases using the provided examples and query scripts.

The creator of py2neo, Nigel Small, has announced that the project is end-of-life. The Neo4j community is grateful for his work and legacy. Py2neo elegantly bridged graph thinking with Pythonic principles, but its maintenance has ceased. For ongoing work with Neo4j in Python, it is recommended to migrate to the official driver or neomodel. Neomodel is an open-source Python OGM (Object Graph Model) for Neo4j that ensures a good level of code quality and maintenance. The guide outlines the steps to migrate from py2neo to neomodel, including installing neomodel, updating imports, connecting to Neo4j, performing Cypher querying, defining objects and relationships, indexing, testing, and creating objects. Some features of py2neo are not covered by neomodel yet, such as converting results to pandas DataFrame/Series and numpy ndarray objects. The guide concludes with a call for feedback and support from the Neo4j community forum or GitHub issues.

Rdflib-Neo4j is an open-source Python library that enables data ingestion of RDF formatted data into Neo4j, overcoming the limitation of Neosemantics being unavailable on cloud-based Neo4j deployments. The library introduces a client-side evolution of Neosemantics, empowering data ingestion in both on-premise and Aura ecosystems. It currently has limited capabilities compared to Neosemantics but is in early access with import functionality. To use the library, users need to configure their Neo4j graph database and Python code, specifying connection credentials, custom mappings, and RDF data source. The library is an open-source project as part of the Neo4j Labs incubator, welcoming contributions and commercial agreements for additional support.