April 2016 Summaries

Neo4j is being used by TruSTAR, a cybersecurity company, to enhance their incident response system. The company's data model consists of highly interconnected pieces of data that are best represented in a graph database. Neo4j was chosen due to its ease of setup and community presence, which were crucial factors for the small startup. While there haven't been any major "ah-ha" moments, the transition to a graph database has been largely frictionless, with TruSTAR's users benefiting from enhanced data visualization and spatial awareness. The company is now live in production, offering a solution to shorten cybersecurity incident response times.

The Cosmic Web paper by the Barabasi Lab explores the concept of a graph database representing the universe, with galaxies as nodes and relationships between them. The authors describe three models for correlating galaxies: Fixed-Length Model, Varying-Length Model, and Nearest Neighbors Model, which provide varying degrees of accuracy in representing real-world constellations. The raw data is imported into Neo4j 3.0 using the LOAD CSV mechanism, allowing for visualization of the graphs in the browser. A test script demonstrates the performance of the Bolt binary protocol, retrieving all neighbourhood relationships in a single go. Additionally, the use of ngraph to load and layout 200k relationships without preparation results in a nice two-dimensional graph rendering of the cosmos.

The newly released Neo4j 3.0 offers a redesigned architecture and aims to provide the world's most scalable graph database, increased developer productivity, and various deployment choices. This significant update can convey a substantial amount of information through a short five-minute video, equivalent to approximately 8.99 million words. Interested parties can learn more by reading the announcement blog post or subscribing to the Neo4j YouTube Channel for updates on the latest videos. Users can also download Neo4j 3.0 to explore its capabilities and leverage massive graphs in their applications or projects.

Neo4j 3.0 is the first release in the 3.x series, featuring a completely redesigned architecture and offering greater scalability, developer productivity, and deployment choices. The new data store expands Neo4j's address space, allowing for graphs of any size. Performance and scale are maintained through dynamic pointer compression and index-free adjacency. The cost-based query optimizer has been improved with added support for write queries and parallel indexes capability. Official language drivers backed by the Bolt binary protocol enable full-stack developers to build powerful applications. Java Stored Procedures provide direct access to the graph, giving developers a way to run imperative code when needed. Neo4j Browser Sync allows users to synchronize saved scripts and graph style sheets across connections. The new release also streamlines operations with a single config file structure and aggregation of log files.

As a software developer and author, the writer emphasizes that technical documentation should cater to different user groups with varying needs, focusing on explaining how the software works best for each individual to achieve their goals and make them happy. The progress of insight from exploring new things is non-linear, making traditional linear documentation structures less effective. Writing technical documentation requires reduced language, comprehensiveness, bias-free content, and expert knowledge. A graph-based structure can support a non-linear learning path by offering multiple connections between doc items, allowing users to find their individual path through the documentation. The Structr Knowledge Graph is an example of this approach, providing a comprehensive and intuitive documentation experience for users.

We were involved in a huge project at Veneto Banca to innovate the bank's IT system from scratch, adopting the microservices paradigm. We realized we needed a tool to govern our services and developed a "service catalogue". Initially, we used an RDBMS-based approach but faced difficulties with relationships and queries. After discovering Neo4j's power in treating relationships as first-class citizens, we refactored our components to build a graph-based data model for our services. This choice provided intuitive, powerful, easy-to-integrate features that helped us implement an effective IT government tool. Implementing Neo4j was the right choice due to its ability to manage relationships strictly, be whiteboard-friendly, and offer a powerful yet easy query language.

The project at Schleich, a German toy manufacturer, involved migrating a central application with gigabytes of production data to a new technology and system architecture, while cleaning up the data model and developing new applications with new functionality and data interfaces. The challenges included dealing with complex models and many connected entities, as well as involving power users in the development process to ensure that the software models were correct. A flexible software platform like Structr was essential for this project, allowing for easy modification of the data model and user interface, as well as handling changes effectively. With a flexible system, users can see new features and get bugs fixed quickly, leading to increased user acceptance. The project's success relied on an agile approach and a technology stack designed for flexibility, which is reflected in the use case of semantic product management.

This graph and data visualization application, built with Tom Sawyer Perspectives, presents the flow of commodities within the United States in various views, allowing for exploration of commodity routes. The application displays information from a subset of data from the United States Census Bureau imported into a Neo4j graph database. It utilizes orthogonal layout, which is suitable for flow-oriented information, and allows nodes to be collapsed for an aggregated visualization of relationships between states and commodities. The application offers multiple view types, including Commodity Flow Query, Map, and Chart views, providing users with the ability to tailor their queries, filter data, and gain analytical insight into commodity routes.

Graphs are being used in a variety of applications beyond their traditional use cases, including rules engines that can interpret complex sets of rules to determine outcomes. These rules engines can be easily updated by business users without requiring technical expertise, allowing for rapid changes to business logic. Graphs are also being used in intelligence and law enforcement to uncover and investigate malicious activity, such as terrorist networks, by analyzing connected data quickly and discovering unknown relationships. Additionally, graphs are being applied to other domains, including education, finance, and entertainment, to analyze complex systems and generate insights.

MITRE Corporation is a federally-funded, non-profit company that manages seven national research and development laboratories to address issues of cybersecurity. Analysts need to track large amounts of detailed information to be successful in cybersecurity, including network and endpoint vulnerabilities, firewall configurations, and intrusion detection events. To determine the appropriate response to an alert, analysts must answer questions such as whether a threat is legitimate, what it means if an alert is true, and whether it's related to a system that needs protection. A graph model can help analyze and relate these pieces of information. MITRE has built a tool called Cauldron to analyze data in a way that prevents cyber attacks, which takes into account network segmentation, firewalls, and vulnerabilities. However, this tool was expensive to develop and required custom code. To address this, MITRE developed CyGraph, a small research project that uses Neo4j graph databases to build a generic data-driven architecture for analyzing cybersecurity threats. This allows for flexibility in extending the data model and morphing the analytics based on new information. Graph visualization is also an important aspect of this approach, as it enables analysts to explore and understand complex relationships between different pieces of information. CyGraph can be used to analyze multiple threat alerts, determine attack response, and provide a domain-specific language for automating queries. The technology has been applied in various real-world scenarios, including customer dependency graphs, packet capture data analysis, and process modeling and simulation.

Tom Sawyer Software has developed a new technology called Geographic Drawing Views, which combines the power of rule-based graph drawing and Open Layers map library to provide a way to visualize spatial data. This new technology allows users to use custom and information-rich maps with logical graph visualizations, introducing features that were missing in each technology separately. With geographic drawing views, users can define visual objects on the map including nodes, edges, and connectors, and configure which objects should be positioned by their geographical locations or automatically arranged by a graph layout engine. The technology also allows for an unlimited number of nesting levels with independently-configurable mixture of geographical and logical views, saving screen real estate and allowing users to see multiple levels of detail simultaneously.

Neo4j 3.0 is coming soon and is now available for review as its first release candidate, featuring new features such as the "Bolt" binary protocol with official drivers for multiple languages, Java stored procedures, Neo4j Sync, operability changes, and a documentation library including developer and operations manuals. This release includes welcome bug fixes and performance improvements, but also has some known issues, notably that the Neo4j Browser only works with authenticated Bolt connections. The team invites users to provide feedback via public Discord or email to help test the software thoroughly and potentially win a rare blue Neo4j water bottle by sharing their thoughts on Twitter using hashtags #neo4j and #feedback.

The digital universe is doubling in size every two years, with an expected 44 zettabytes of data by 2020. To discover and use patterns and connections in data for business purposes, graph databases are increasingly used. The authors have been dealing with census data for over 15 years and were initially hesitant to adopt a graph database concept due to complexity concerns. However, they found that representing citizen data as a graph database was the perfect solution for scenarios such as providing comprehensive and online real-time ancestral trees, determining heirs and calculating heritage shares, identifying old/helpless citizens without any relatives living nearby, and constructing domestic/international migration routes, investigating causes and discussing consequences. The authors faced challenges in data modeling, export, and import, including mutual relations, birthdate info representation, and special characters. They also discussed the importance of change data capture mechanisms to keep RDBMS and graph database synchronized. To address these issues, they developed a generic ETL solution that transforms and continuously synchronizes RDBMS data to a graph database, consisting of a visual design tool, an ETL execution engine, and APIs for development.

The text discusses the versatility and benefits of using graph databases, specifically Neo4j, in various industries such as fraud detection, master data management, logistics and delivery. It highlights how graph databases can be used to identify patterns and relationships that were previously unknown or difficult to detect, leading to improved efficiency, productivity and profit. The growing demand for Neo4j expertise is also mentioned, with companies paying above-average salaries and daily rates for graph database knowledge, making it an attractive option for organizations looking to implement a graph database into their system.

Larus Business Automation, a Neo4j Italian partner since 2013, has a long history of working with the graph database, starting with its first major projects in winter 2010. They have implemented various proof-of-concept projects for companies such as Barclays and Veneto Bank, showcasing the capabilities of Neo4j in fraud detection and complex communication channel analysis. The company has also delivered international consulting services to validate graph models and tune Cypher queries. Recently, they introduced two new integration projects: a bidirectional integration API between Neo4j and Couchbase, and a brand-new Neo4j 3.x JDBC Driver with Bolt protocol support. The driver aims to provide efficient performance, ACID compliance, and support for Neo4j 3.0 features like procedures. It is designed to be easy to use and integrate with popular suites such as Pentaho and Jasper Studio.

The text discusses the importance of relevance in content marketing, particularly in digital publishing. It highlights how having a visitor on-site, it's challenging to keep them engaged and moving towards a recurring relationship with the business. The key to solving this challenge is providing relevance to visitors through various means such as recommending similar pieces of content, using metadata to boost engagement, and organizing content in categorical and hierarchical manner. The author shares their experience of implementing a graph database solution using Neo4j to solve this problem, leveraging the Stack Overflow tags corpus to generate metadata for content tagging, and building an ontology to classify latent connections in the network. The approach allows for dynamic categorization, content filtering, and collaborative filtering, enabling flexible and scalable solutions for customer experience.

The text discusses the analysis of leaked financial documents by journalists using various tools and techniques. The ICIJ (International Consortium of Investigative Journalists) used a graph data model to represent relationships between entities such as companies, officers, and clients. They analyzed metadata from documents using Apache Solr and Tika, and connected it to create a graph in Neo4j. The text highlights the importance of extracting named entities, determining relationships, and analyzing data using graph queries and visualizations. It also discusses potential issues with the ICIJ data model, such as duplicates and incomplete information. The authors provide examples of how to extend the basic graph model used by the ICIJ, including introducing new entities, relationships, and activities. They demonstrate how to create a graph from leaked documents using Cypher in Neo4j and show interesting queries that can be run on the graph.

The International Consortium of Investigative Journalists (ICIJ) conducted a massive financial data leak known as the Panama Papers, which exposed offshore tax structures used by high-profile individuals and celebrities over the past 40 years. The leak consisted of 2.6 Terabytes and 11.5 million documents, making it one of the largest data leaks in history. To analyze this vast dataset, ICIJ worked with graph technology, specifically tools like Linkurious and Neo4j, which enabled them to process complex connections quickly and efficiently, without requiring specialized technical expertise. This approach allowed investigative journalists to make sense of the massive dataset, highlighting the importance of democratizing scalable data analysis technologies for a free and open society.

The Panama Papers data leak is a massive financial dataset that exposed offshore tax structures created for high public officials and celebrities by Panamanian law firm Mossack Fonseca over the past 40 years, at a size comparable to every data leak in the last decade. The International Consortium of Investigative Journalists (ICIJ) analyzed this data using graph technology, specifically Neo4j and Linkurious, which enabled them to process large volumes of connections quickly and efficiently. This approach allowed the ICIJ team to work with complex and interconnected datasets without requiring technical expertise. The use of graph technology in big data analysis is no longer exclusive to the ivory tower, as scalable solutions are now available to everyone, including startups and investigative journalists, making it possible for anyone to make sense of massive datasets at scale.

The text discusses the integration of graph databases with the Internet of Things (IoT) to create smart services that can handle large amounts of data and produce actionable outcomes. The author highlights the limitations of traditional relational databases in handling IoT data, which requires complex event processing and un-foreseen matches to deliver business value. Examples are provided, such as Uber's use of graph databases to optimize taxi service, and a building management system that uses graph databases to predict maintenance needs. The text also touches on the challenges of storing and processing large volumes of IoT data, including the need for flexible data stores like graph databases. As the enterprise focus shifts towards investing in new digital business offerings, the author predicts that IoT-supported smart services will become the norm, leading to increased use of graph data.

Neo Technology humorously announced a fictional product, "Neo 4 Java," on April Fool's Day, claiming it enhances database development speed and scalability, while playfully attributing its origins to a coffee-stained napkin sketch by Emil Eifrem. The announcement details the supposed performance benefits of "Neo 4 Java," such as improved read and write scalability, native storage and processing, and an easy learning curve, emphasized with exaggerated claims and satirical comments about its addictive nature and transformative effects on users. Despite the playful narrative, the announcement serves as an April Fool's joke, blending whimsy with elements of graph database technology.