August 2023 Summaries

The Aerospike Database is a high-performance, globally distributed NoSQL database designed for real-time, mission-critical applications. It can be integrated with Snowflake to reduce latency and increase the availability of data needed for real-time decisioning. Aerospike's low latency, high availability, infrastructure efficiency, and scalability make it a powerful companion to Snowflake. By combining these two technologies, businesses can enhance their data caching capabilities, improving application performance, user engagement, and customer satisfaction.

Snowflake is a leading cloud-based data warehousing platform that enables businesses to store, process, and analyze vast amounts of data, but it can benefit from the integration of Aerospike Database, which is designed for real-time, mission-critical applications. By combining Snowflake with Aerospike, organizations can reduce latency and increase availability of data needed for real-time decisioning, leveraging Aerospike's high-performance, globally distributed, multi-model NoSQL database capabilities. Caching with Aerospike "supercharges" Snowflake by providing near-instantaneous access to petabytes of data, while its low latency plus high availability make it a game-changer for operational applications. Additionally, Aerospike's infrastructure efficiency keeps costs low and scales from gigabytes to petabytes without performance degradation, making it an ideal companion to Snowflake, with numerous businesses already successfully leveraging their power in real-world use cases.

The user profile store is a crucial component of web-facing applications with large user bases, as it enables personalization and effective consumer engagement. These stores consist of current, historical, and derived user/consumer data combined with behavioral attributes. Industries such as banking, telecommunications, e-commerce, and retail rely on fast access to user profile stores. The requirements for a scalable User Profile Store include real-time operations on large datasets, 24x7x365 availability, easy integration with business systems, and the ability to handle thousands of cascading database interactions simultaneously. Ad Tech use cases require discovering everything possible about a user and applying algorithmic rules in just milliseconds. As companies track users or customers in new ways, identity graphs are becoming popular for identity resolution and targeting. Aerospike is a preferred choice for many multinational firms in the Ad Tech space due to its ability to handle large volumes of user profile, fraud detection, and identity resolution operations at millions per second or billions per day scale.

The Aerospike database offers two options for building a global cluster: strong consistency (synchronous) and high availability (asynchronous). While synchronous clusters ensure data is always consistent, they tend to run slower than asynchronous ones. High availability technology prioritizes speed over accuracy, allowing some data to be stale or even non-existent. This approach works well for recommendation engines, AdTech, and fraud detection systems that emphasize quick user experiences. In contrast, strong consistency use cases require accurate and up-to-date data at all times, making them suitable for money-related actions, inventory management, and social graphs. The choice between these two options depends on the specific use case and its requirements.

The Aerospike database technology offers two primary options for building a global cluster: Strong Consistency and High Availability. Strong Consistency ensures data is always current and accurate, with low latency reads, high availability, and no lost writes. However, this approach prioritizes accuracy over speed, making it suitable for money-related actions, inventory management, and social graphs where data integrity is paramount. In contrast, High Availability prioritizes speed and performance, allowing for some data staleness or loss, but ensuring the application provides a good user experience. This approach is commonly used in AdTech and fraud detection systems, where quick reads are necessary to serve ads or detect fraudulent transactions. The choice between Strong Consistency and High Availability depends on the specific use case, with Aerospike supporting both options for various applications.

Aerospike has announced a new graph database called Aerospike Graph, designed to simplify building and deploying applications for highly connected enterprise-scale datasets. The database combines the unlimited scale, high throughput, and low latency of the Aerospike Database with an independently scalable graph compute engine built on Apache Tinkerpop. Aerospike Graph features independent scaling of compute and storage resources, enabling optimal resource utilization, consistently low-latency queries, and extreme throughput for complex multi-hop graph queries at scale. The database can be deployed on Google Cloud, providing a future-proof path for IT and development organizations to follow. Users can interact with the database using the Gremlin Console or client applications, and can load data into it using the Graph bulk loader.

The article discusses the challenges faced by a European Ad Technology customer when scaling their system, which initially used MongoDB and AWS for data storage and processing. As traffic increased to 700 million requests per day, infrastructure costs for the MongoDB cluster exceeded $30K per month, slowing down report generation and click processing. Asteriosoft conducted an in-depth analysis and proposed a solution by migrating MongoDB workload to Aerospike, reducing infrastructure costs to $5K per month and improving query processing throughput. The example highlights the importance of utilizing specialized tools instead of relying on generic solutions and emphasizes the need for architectural analysis to address accumulated problems in large-scale systems.

The text discusses the challenges faced by a European Ad Technology company when scaling its system, which led to increased infrastructure costs and decreased performance. The initial approach of using PHP and MongoDB on AWS resulted in high costs due to MongoDB's inability to handle large volumes of data and queries efficiently. After attempting to use Redis for indexing document searches, the developers suggested scaling MongoDB further, but this was deemed too expensive by the business owner. An architectural analysis conducted by Asteriosoft proposed a solution by migrating the workload to Aerospike, a specialized tool that provides faster response times for key-value storage and statistical data processing. This resulted in significant cost savings and improved system performance, allowing the company to increase traffic without additional infrastructure investments. The example highlights the importance of utilizing specialized tools and conducting deep architectural analysis to resolve accumulated problems and reduce costs.

Aerospike's configuration format can be difficult to learn and lacks the ecosystem of more standard formats, but using the asconfig tool in Aerospike tools 8.3.0 allows developers to write configurations in a standard YAML format with schema validation and IDE suggestions. By setting up an Integrated Development Environment (IDE) for Aerospike configuration suggestions and installing the Red Hat YAML VS Code extension, developers can use code completion and error highlighting features to write Aerospike configurations more efficiently. The asconfig tool also enables converting existing Aerospike configurations to YAML format, making it easier to adopt this standard format. Additionally, launching an Aerospike cluster in Docker using a YAML configuration is straightforward with the asconfig tool, allowing developers to easily deploy and manage their Aerospike nodes.

The Aerospike Observability Stack provides essential tools for developers and database operators to monitor their systems architecture and establish operational workflows. Key components of the stack include Aerospike Instrumentation & Metric Exporter, Prometheus, Grafana, and Alertmanager. The metric exporter makes Aerospike metrics available in a format that can be easily integrated with various monitoring solutions. Prometheus TSDB stores Aerospike metrics outside of the cluster, while Grafana allows users to visualize these metrics in dashboards. Alertmanager processes alerting rules and integrates with popular communication systems for notifications. The Aerospike Observability Stack also includes pre-built dashboards for specific use cases. Examples provided demonstrate how to set up and explore the system using single or multi-node configurations.

The Aerospike Observability Stack is a comprehensive set of tools for monitoring and managing Aerospike databases, allowing developers and database operators to monitor latency objectives, understand infrastructure utilization, configure alerting, and more. The stack consists of primary components including instrumentation, metric exporters, Prometheus, Grafana, and Alertmanager, which provide storage, visualization, and alerting capabilities. With the Observability Stack, users can deploy a single node or multi-node Aerospike cluster, and customize dashboards and alerts to suit their needs. By leveraging the stack, organizations can gain insights into system health, quickly identify issues, and optimize performance.

In the latest version of Aerospike Database, version 6.4, significant improvements have been made to query and data distribution capabilities. The release introduces secondary index on flash storage, allowing for faster restarts and reduced RAM consumption. Additionally, the new Secret Agent feature enables integration with external secrets management services such as AWS Secrets Manager, Google Cloud Secret Manager, Azure Key Vault, and Hashicorp Vault. Furthermore, Cross-Datacenter Replication (XDR) throughput has been optimized by distributing ship requests across service threads using partition affinity. Lastly, support for Amazon Linux 2023 and Debian 12 has been added, while single-bin namespaces are no longer supported.

Aerospike Database 6.4 introduces several new features, including secondary index on flash, which allows for persistent storage of secondary indexes without consuming RAM, enabling warm restarts and improved performance in mixed workloads. The new feature also comes with a trade-off in terms of device IO cost. Additionally, Aerospike now supports integrating with secrets management services through the Secret Agent, which can fetch secrets from external services like AWS Secrets Manager. The release also includes improvements to Cross-Datacenter Replication (XDR) throughput and support for newer operating system distributions such as Amazon Linux 2023 and Debian 12. Furthermore, Aerospike has removed support for single-bin namespaces, a feature that provided savings in storage space, but is no longer necessary due to advancements in flash storage technology.