October 2018 Summaries

Graph databases, particularly those built on Neo4j, are being utilized to tackle pressing issues such as managing critical infrastructure and detecting cyber threats. By incorporating space and time dimensions into graph visualizations, users can gain actionable insights from data. Visualizing data in both space and time allows for the identification of patterns and relationships that may not be immediately apparent when viewing data on its own. This approach enables organizations to extract valuable information from complex networks and make informed decisions. The use of tools like KeyLines, which is compatible with Neo4j, facilitates the creation of visualizations that bridge the gap between an organization's data and end-users who require actionable insights.

When trying to understand Neo4j, the author was out of their skill set and couldn't find anyone willing to learn it. However, after four years of attending meetups and a career change, they met Jason Cox at a civic hackathon and together they started learning about Neo4j. The author then met Karin Wolok, Neo4j's Community Development Manager, who inspired them to start the first Philadelphia Neo4j and GraphDB Meetup. With Karin's guidance, the meetup group was formed and their first event was successful, with 30+ attendees. Since then, they've had 16 more meetups, refining their skills in communicating interests and building a community of graph database enthusiasts. They now host regular meetups with live demos using Neo4j's online sandbox or desktop app, aiming to make the technology accessible to everyone.

Graph analytics has a long history dating back to Leonhard Euler's solution of the "Seven Bridges of Königsberg" problem in 1736. However, it wasn't until recent years that graph technologies have seen an explosion of interest and usage. This growth is driven by several forces, including real-world applications of graph analytics, digitization, and advances in computing power. The convergence of analytics with transactions, also known as "translytics," enables organizations to better understand real-world networks and forecast their behaviors. Graph analytics are particularly useful for handling connected data and responsive to dynamic changes, which is why businesses are turning to them to improve predictive capabilities and decision-making frameworks for artificial intelligence. The integration of graph technologies with traditional OLAP and OLTP systems, such as HTAP, enables continual analysis to become ingrained in regular operations, leading to new forms of real-time business-driven decision-making processes.

This week in Neo4j has seen the release of several new features and applications, including DeepGL's use for extracting features from peer-to-peer networks, APOC's import functionality for relational data, and Agent Smith 2.0, a "top" application that now watches transactions for stability. Graphs have also been explored in relation to AI, with Morgan Vawter writing an article on the topic and Bajal showing how to visualize a Kubernetes cluster using Neo4j. Additionally, there's been news from GraphConnect NYC 2018, including an experience report by Arina Igumenshcheva and a post by Igor Bobriakov on integrating Spark, GraphX, and Neo4j. The community has also featured Devansh Trivedi, who is using Neo4j for his "100 Days of Machine Learning" challenge, creating word-pair frequency graphs and content-based recommendation engines.

CALIBRE, a company that works with large government customers, has replaced traditional SQL queries with Neo4j to improve data management and analysis. They have found that Neo4j allows them to easily train analysts and developers alike on the graph database, making it accessible to a wide range of users. By using Neo4j, CALIBRE is able to analyze complex relationships between parts and equipment, such as tracking 10 million parts in a single tank, and build more efficient chairs by identifying interchangeable parts. The company has also seen benefits in terms of speed and ease of use, with Neo4j allowing them to traverse relationships and query the database without writing recursive SQL or dynamic SQL. With Neo4j, CALIBRE is now able to integrate data science and analytics processes, using Python to connect directly to the database and perform tasks such as data retrieval and analysis. Overall, Neo4j has been a game-changer for CALIBRE, enabling them to work more efficiently and effectively with complex data sets.

NoSQL databases are a spectrum of data storage technologies that are more different than they are alike, designed to overcome the challenges of high volume, velocity, variety and valence in today's data landscape. Relational databases can no longer handle these challenges, which include rapid changes in data structure, large datasets becoming unwieldy, and dense or sparse, connected or disconnected data. NoSQL databases address these challenges by optimizing for high write loads and having more flexible data models, enabling them to handle a wide diversity of data and flexibly adapt to future data needs. Understanding how NoSQL databases overcome these challenges is crucial in finding the right database for an enterprise use case.

Caterpillar, a 90-year-old company, is exploring natural language processing (NLP) for purposes such as vehicle maintenance and supply chain management. NLP enables computers to understand human language, which can be represented naturally by graphs due to their flexibility and structure. The company has been experimenting with various use cases, including a dialog system to interact with machines through queries, and reading warranty documents at scale to extract meaning from large text datasets. Graph databases are being used to model and query these data structures, allowing for the creation of complex relationships and ontologies that can be instantiated in real-world applications. The ultimate goal is to enable computers to understand human language and extract meaning at a large scale, with potential applications in areas such as supply chain management and customer service.

Gartner's hype cycle suggests that graph technology is at a peak of inflated expectations, but this article aims to cut through the noise by highlighting five noteworthy use cases for graph technology and graph analytics. Graph technology stores data and relationships in mathematical graph theory, emphasizing connections between data points. This approach allows for faster query performance and better analysis of complex data relationships. Five proven use cases highlighted include machine learning, fraud detection, regulatory compliance, identity and access management, and supply chain transparency. These use cases demonstrate the benefits of graph databases in various industries, such as powering recommendation engines, detecting fraudulent social media accounts, and tracing sensitive data for compliance purposes. By modeling complex relationships and providing scalability and agility, graph technology can bring greater transparency and efficiency to business operations.

Graph algorithms, particularly those used in Neo4j Graph Analytics, are essential for analyzing complex and ever-changing data, uncovering key patterns and trends, and unlocking numerous opportunities. Networks are a representation of complex systems, and graph models provide the mathematical tools to analyze these structures. By understanding networks and their connections, organizations can make new discoveries, develop intelligent solutions, and gain insights into propagation pathways, flow capacity, and system robustness. Graph algorithms, such as PageRank, have multiple applications across various domains and use cases, making them a powerful tool for exploring connected data.

This week in Neo4j highlights various developments and tutorials across different topics, including community detection using Louvain algorithm, creating a schema.org linked data endpoint on Neo4j, and implementing authorization in GraphQL. Featured community member Will Lyon is showcased for his work on the GRANDstack and graph databases. Various blog posts and talks are covered, such as learning taxonomies from user-tagged data, running decision trees in Neo4j, and using Neo4j to depict English law cases. The summary also includes upcoming events, including an Open Data Journalism Workshop, and a tweet of the week highlighting issues with real-time data access for public transportation.

IBM uses Neo4j to design its next-generation Power chips, leveraging the graph database's capabilities to accelerate graph algorithms. The company has seen significant benefits from using Neo4j, including improved performance and reduced latency. IBM sees a strong demand for graph technology across various industries, with applications in AI, cybersecurity, master data management, and dynamic pricing. The partnership between IBM and Neo4j is considered strong, with both parties collaborating on research and development to accelerate the adoption of graph technology.

You can use a graph database like Neo4j without directly touching your product, and it's suitable for managing company knowledge as well as improving software development processes. The architecture of most companies is inherently complex networks, mirroring their organizational structure, which are difficult to navigate even with microservices. Many third-party software components, especially open-source ones, cause delivery dependencies that can lead to chaos if not managed properly. Graph databases help track and analyze these dependencies, providing valuable insights into progress, bugs, and critical areas. By leveraging graph databases, companies can turn their accumulated knowledge into wisdom, gaining a competitive advantage in the tech industry.

Building a graph database model from the highest possible vantage point using natural language and domain-specific language helps develop a model that truly stands the test of time. Most basic texts on graphs start with vertices and edges, but modeling them is where the fun part begins. A graph can be viewed as a way of modeling the world using interconnected triples in the format of noun-verb-noun, which can be applied to any language regardless of the order of subject, verb, and object. The model should capture the nouns of that world, form sentences with verbs, labels, and relationships, resulting in a graph model that is easy to reason about. When building this model, consider questions you'll ask of your model, such as "How long did I spend giving talks?" or "Will I be out of talk A in time for talk B?", which may require different data structures like durations or end times. The model should also account for nuances like multiple roles at different companies or the semantics of relationships. Once the model is built, it needs to be converted into a graph, where considerations include viewing the world as instances of those nouns and ensuring unambiguous routes or paths for traversal decisions. The final step involves diving into Cypher query language, using property nodes and relationships to define a schema on the graph, storing data under that schema, and defining aliases and primary languages for more complex concepts like ticketing systems.

Graph analytics is essential for analyzing today's connected data, as simple statistical analysis alone fails to capture behaviors within complex systems. The concept of "valence" in big data refers to the tendency of individual data to connect and form new aggregations, with higher valence indicating more connections within a dataset. Valence increases over time but not uniformly, leading to power-law distributions and scale-free networks with hub-and-spoke structures. These dynamics complicate traditional analytics approaches, requiring more sophisticated methods to model scenarios such as network evolution and emergent properties. Graph algorithms can reveal the workings of intricate systems and networks at massive scales, empowering organizations to understand their data in new ways and uncover patterns that are undiscoverable using traditional methods.

This week in Neo4j saw significant developments across various areas of the graph database ecosystem. The release of version 3.4.8.0 of the Neo4j Graph Algorithms library, which now supports weighted PageRank, was a notable highlight. This feature was showcased in blog posts by Tomaz Bratanic and Plushcap (me!), where we demonstrated its effectiveness in finding influential IP addresses on an AT&T Network telecommunications dataset from Kaggle. In another area, Joe Depeau presented a webinar on modeling financial risk using Neo4j, focusing on FRTB compliance, while also demonstrating how to model investment risk at the trading desk level as a graph. Additionally, Alex Tavgen shared an article about using Neo4j to store the storylines of an interactive theatre production, where audience participation influenced the next scene. The Graph Gallery was also launched, providing single-page applications that take advantage of Neo4j Desktop's management services for Neo4j databases. Other notable topics included extensibility for Java developers, Kubernetes backups, and next-generation chatbots with NLP services and graphs.

IT operations is complex, with thousands of systems and connections, making graph technology a suitable fit. Clayton Ching views a graph database as a requirement for Ignio, Digitate's flagship product for cognitive automation of complex IT infrastructure. Ignio is designed to understand context across IT enterprise applications and infrastructure, including dependencies and relationships between systems. Neo4j provides the necessary graph data to create, update, and understand these relationships. The company chose Neo4j due to its leading position in the market and impressed with its customer base and individual interactions. Using Neo4j has enabled visualization and understanding of complex IT environments, making it easier to accomplish tasks that were previously challenging. If started over, Clayton Ching believes Ignio should have begun with a graph database for an easier implementation and better ability to understand relationships. The future of graph technology in the IT operations space is headed towards becoming an essential part of products, reducing complexity and providing context understanding capabilities.

Algorithms were named after Al-Khwarizmi, a mathematician who lived 1200 years ago and made significant contributions to graph theory. Graph search algorithms are used to traverse graphs in the most efficient way possible, with two basic types: depth-first and breadth-first searches. Dijkstra's algorithm is a type of breadth-first search that finds the shortest path between two nodes in a graph by exploring all possible paths level by level. The A* algorithm improves upon Dijkstra's by combining elements of both breadth-first and best-first searches, using a heuristic to guide its search and choosing the node with the lowest overall cost at each level. Graph search algorithms are essential for solving real-world problems, but the choice of algorithm depends on the type of results desired. Understanding the basics of graph technology is crucial for making informed decisions about which algorithm to use in different situations.

Knowledge graphs are key to delivering relevant search results, meeting the four criteria for relevance: query, context, user, and business goal. Knowledge graphs provide an entity-centric view of linked data, enabling growth, enhancement, and continuous updates. They store multiple views in Elasticsearch for fast response and serve as a pattern for others to use. The rise of knowledge graphs has been observed across industries, including ecommerce, finance, health, and criminal investigation. Data sparsity is a challenge that can be addressed through collaborative filtering, tagging-based systems, or trust networks. Knowledge graphs are being used for uncovering patterns in disease progression, causal relations involving disease, symptoms, and treatment paths. They provide traceability from the knowledge graph back to its source of information, maintaining data relevance and up-to-date status. Elasticsearch is used as a cache and powerful search engine on top of Neo4j, providing fast, reliable, and easy-to-tune textual search capabilities. A signal is any component of a relevance-scoring calculation corresponding to meaningful and measurable information, which can be controlled through signal modeling or ranking functions. Personalizing search involves including users as a new source of information for customized result sets. Concept search moves from searching for strings to searching for things, using data enrichment techniques such as adding synonyms or machine learning tools. Combining Neo4j and Elasticsearch offers two approaches: one with documents in Elasticsearch and the graph database, and another with tighter connection between Elasticsearch and the knowledge graph. The latter approach appears more performant by narrowing down the operation based on context or user.

Tracing data lineage across all investment data silos is next to impossible without a graph database, which can effectively manage risk in compliance applications. A proper FRTB risk assessment requires line of sight into the entire dataset, but current systems are unable to provide this level of transparency and control.

This week in Neo4j has been about supercharging developer productivity with the latest release of neo4j-graphql.js, which now allows developers to spin up a GraphQL API backed by a graph database with just type definitions. The new features include auto-generating query/mutation types and resolvers, augmenting a GraphQL schema with pagination, ordering, and _id fields, flexible handling of relationship types, and middleware support for authentication/authorization. Additionally, there have been releases of the Kettle plugins for Neo4j, which now add metadata injection support to handle more complex scenarios. The community has also seen blog posts on using the Cosine similarity algorithm to find similar Game of Thrones episodes based on character appearances, a word-pair frequency graph in Neo4j, and building graphs based on the Medium blogging platform. A video tutorial has been created showing how to get up and running with the Neo4j Sandbox, which creates a temporary Neo4j instance in the cloud for learning about Neo4j graphs. The community has also seen an experience report from the GraphConnect conference, interviews on the podcast, and a tweet of the week that showcases the power of Neo4j queries.

In this interview, Eric Spiegelberg from GraphAware discusses the company's work on cybersecurity and its natural language processing framework for Neo4j. He believes that graph technology will become omnipresent and has already changed the world with the release of the Panama Papers. Spiegelberg shares his experience working with Neo4j, highlighting its flexibility and power in solving complex problems. He also talks about the potential applications of graph technology in cybersecurity, where it can be used to analyze high-volume, unstructured data. The interview showcases GraphAware's efforts to leverage Neo4j for cybersecurity research and development, and Spiegelberg expresses his enthusiasm for the future of graph technology in this field.

The Vanguard Group, a financial services company, has been working on refactoring its monolithic applications into microservices. The team faced challenges in managing code and dependencies, particularly with large amounts of code (3-4 million lines) in their Java archives. They initially used visualizing tools like Structure101 to understand jar dependencies but found it difficult to manage complex relationships between services and jars. The team then moved to using Neo4j, a graph database, to model their services and modules as nodes and edges, respectively. This approach allowed them to visualize relationships between services and jars, enforce best practices, and track metrics such as acyclic dependency principles and instability. With Neo4j, the team built tools like Modularity Assessment Tool Suite (MATS) and Enterprise Service Catalog to manage jar dependencies, track metrics, and provide visualizations of service dependencies. The team plans to use Neo4j to automate static analysis, identify dead code, and improve data management and governance in their architecture.

Neo4j is a graph database platform that provides a foundation for building compliance solutions, particularly in the context of Fundamental Review of the Trading Book (FRTB) regulations. It enables banks to effectively capture investment data lineage across internal and external applications and data sources, tracing risk factors back to their original data sources and visualizing the lineage of risk factors to create compliance models. Neo4j's native graph technology provides agility in assessing risk factors and computing capital requirements in real-time, allowing trading desks to take full advantage of market opportunities while remaining compliant with evolving regulations. The platform also supports a range of innovative uses beyond risk modeling, including credit risk analysis and value-at-risk calculations. By using Neo4j, organizations can streamline their internal systems, build a firm foundation for future compliance applications, and maximize available capital and drive investment profits.