December 2023 Summaries

Adobe's VP of Engineering, Sandeep Nawathe, presented the technical approach to segmentation and personalization for its Adobe Experience Platform (AEP) at Aerospike Summit 2023. The platform handles data from over 12,000 customers and performs identity resolutions and generates audiences for various destinations. To address the complexities of managing customer experiences at an extreme scale, Adobe leveraged Aerospike as a managed service on Azure to tackle the issue efficiently. This solution allows for streaming segmentation with Aerospike, enabling real-time personalization at scale by enriching streaming data and taking an innovative approach to storing incremental state.

Adobe Experience Platform handles up to 30 trillion queries per day, providing real-time customer profiles and advanced analytics for personalized experiences across channels. The platform integrates various data sources, enabling organizations to centralize and standardize customer data from any system. Adobe's Real-Time CDP handles data from over 12,000 customers, performing identity resolutions and generating audiences for destinations such as email service providers and customer support. To tackle the complexity of real-time personalization at scale, Adobe uses Aerospike, a managed service on Azure, to store incremental state and enrich streaming data with its profile store. This approach enables efficient handling of extreme scale and complexity in managing customer experiences.

The carbon footprint of information technology is becoming a significant concern due to its substantial energy demand and associated emissions. Data centers account for 1-1.5% of global energy consumption and contribute to greenhouse gas emissions. As data volumes continue to surge, it's crucial to consider both scaling and efficiency in software development. More efficient software can handle higher workloads on the same hardware, reducing carbon emissions and total cost of ownership. Companies should develop IT sustainability strategies that focus on understanding sources of carbon emissions and implementing sustainable practices. A standardized approach for measuring and displaying carbon emissions from software technologies is recommended to help buyers make more informed decisions.

The modern organization prioritizes sustainable IT infrastructure to mitigate its environmental impact, which is substantial due to the energy demand of software systems and hardware infrastructure. Proper tracking and reporting of emissions are essential to understand the environmental impact of technology. Data centers, end-user devices, and software development are significant sources of carbon emissions, with AI posing a challenge as it demands significant energy resources. Efficient software can play a pivotal role in mitigating these emissions, with examples like Aerospike reducing carbon emissions by 80% compared to legacy solutions. A holistic approach to sustainability strategy involves understanding the sources of carbon emissions and adopting standardized approaches for measuring and displaying emissions. Ultimately, it requires a collective commitment from industry leaders, organizations, and consumers to embrace sustainable practices and technologies.

Real-time databases like Aerospike are essential components of modern data environments, working in tandem with streaming technologies to enable more real-time use cases in production. The growth in streaming architectures has expanded the market for real-time databases, making them a differentiator for various industries. While some people may wonder if the proliferation of streaming technologies obviates the need for databases, the truth is that high-performance, distributed databases have been a real-time component of companies' data environments even before the widespread adoption of distributed streaming platforms like Kafka. Real-time databases deliver workloads with over 1 million transactions per second at under 100-, 10-, or even single-millisecond throughput at 99th percentiles, making them crucial for applications that require low-latency access to specific data points. Additionally, real-time databases can help narrow the operational-analytical gap by providing native secondary index capabilities and integrating tightly with Trino and Presto. Overall, streaming technologies and real-time databases are essential tools that work best in tandem and are rarely alternatives to each other.

A real-time database is essential for a data streaming environment as it enables organizations to thrive in production with real-time use cases. Streaming technologies have exploded in scale and throughput, making real-time databases like Aerospike a differentiator for various industries. Real-time databases can handle high-performance distributed workloads with over 1 million transactions per second at under 100-millisecond throughput at 99th percentiles. However, streaming systems introduce latency, which may not be significant if the application pipeline has minimal latencies. Financial services and AdTech firms often connect their real-time databases directly to data sources and destinations to minimize hops and reduce latency. Streaming platforms typically need a database for high-performance distributed workloads and for managing operational-analytical gaps. Real-time databases are critical in narrowing the gap between operational and analytical systems, with modern databases like Snowflake offering improved analytics tools that enable real-time operational workloads. Aerospike provides seamless integration with streaming platforms, its robust real-time database architecture, and unique capabilities that elevate data architectures with streaming.

The blog post offers a comprehensive guide on setting up a real-time data streaming pipeline from an Aerospike Database to Amazon Managed Streaming for Apache Kafka (Amazon MSK) using AWS Identity and Access Management (IAM) for secure authentication. The process begins with the creation of an Aerospike Database cluster and the insertion of sample messages, which are streamed to Amazon MSK via Aerospike's Kafka Source Connector. The guide includes detailed instructions on setting up a three-node Kafka cluster within AWS MSK, configuring IAM policies and roles to authenticate database interactions, and establishing a Kafka client machine. It further elaborates on creating Kafka topics, testing message production and consumption, and using the Aerospike Kafka Source Connector for outbound data transfer. The tutorial underscores the ease of constructing a real-time streaming pipeline by integrating Aerospike with AWS MSK, leveraging IAM credentials for secure data transmission.

The Aerospike Database provides various deployment strategies for building geographically distributed applications. These include intra-regional deployments as well as clustering options between regions within the same continent or even different continents. Key terms to understand are "region," "availability zone," and "site" / "multi-site." The Aerospike Database supports strong consistency models, such as active-active single-region multi-site synchronous clustering and active-active multi-regional multi-site synchronous clustering. Additionally, it offers high availability models like disaster recovery, single-region multi-zone highly available single cluster, active-active cross-datacenter replication, active-passive cross-datacenter replication – cascading updates, and active-passive cross-datacenter replication – hub and spoke updates. These deployment options enable applications to achieve the desired resiliency, availability, and accuracy for their specific needs.

The Aerospike Database offers multiple deployment options for building geographically distributed applications, including intra-regional, single-region, multi-region, disaster recovery, high-availability, and cross-datacenter replication models. These deployment models provide varying levels of resiliency, availability, and consistency, depending on the specific use case requirements. The database supports strong consistency models, linearizable consistency, session consistency, and eventual consistency, allowing developers to choose the best model for their application's needs. The deployment options also consider factors such as latency, reads, writes, and cascading updates, enabling developers to select the optimal configuration for their geographically distributed applications. By providing a range of deployment options, Aerospike enables developers to build highly available, scalable, and resilient applications that can handle diverse use cases, including adtech, fraud detection, recommendation engines, social feeds, and more.

In a distributed database system, achieving optimal performance requires careful management of various factors such as horizontal and vertical scaling, strong consistency, and geo-distribution for active-active systems. Aerospike's CTO and Founder, Srini Srinivasan, outlines four essential principles for modern distributed database management: scaling out (horizontal scaling), scaling up (vertical scaling), strong consistency and high availability, and geo-distribution for active-active systems. By focusing on these areas, organizations can ensure that their distributed databases operate at peak efficiency while meeting critical service level agreements (SLAs).

The text discusses the key principles of managing distributed database systems, focusing on high-throughput and low-latency performance, as outlined by Srini Srinivasan, Aerospike's CTO and Founder. It highlights the importance of scaling out, which involves adding extra nodes to a cluster to prevent single points of failure, and scaling up, which leverages solid-state drives for improved data access speed and capacity. The text emphasizes strong consistency and high availability, ensuring all nodes reflect up-to-date information, and details Aerospike's unique roster-based approach to maintain these properties efficiently. Furthermore, it addresses the concept of geo-distribution for active-active systems, allowing globally distributed databases to handle transactions consistently and without data loss, even if certain parts of the system go offline. The summary concludes by stressing the importance of balancing consistency, availability, and partition tolerance to avoid negative impacts on application performance and service level agreements, recommending Aerospike's observability and management tools for optimizing distributed database management.

The Aerospike APJ Summit 2023 brought together technologists from around the world to discuss sustainable data practices and their impact on reducing CO2 emissions and costs without compromising overall efficiency. Five keynote sessions highlighted valuable insights for organizations looking to modernize their data infrastructure, including scaling Aerospike with Kubernetes, exploring how technology can save humanity, a deep dive into Aerospike Database 7, an overview of massively scalable high-performance Aerospike Graph, and managing observability stacks. The summit aimed to address the state of real-time data in a world increasingly concerned with sustainability and provided environmentally conscious solutions for working with real-time data.

Aerospike APJ Summit 2023 explored sustainable data practices and offered environmentally conscious solutions for working with real-time data, reducing CO2 emissions and costs without compromising efficiency. Keynote speakers shared their experiences in leveraging Aerospike to transform their technology stacks and address sustainability goals. Sessions covered scaling Aerospike with Kubernetes, the role of technology in environmental challenges, Aerospike server deep dive, Aerospike Graph for uncovering relationships across diverse data sets, and an observability stack for managing what matters. These sessions provided valuable insights and actionable solutions for organizations looking to modernize their data infrastructure and make it more sustainable.

Vectors are transforming AI applications by enabling language models, image recognition systems, and generative AI applications. However, challenges remain in scaling these applications due to high costs, limited throughput, and the need for more than just vector search and model execution. Open source models like Hugging Face have accelerated AI adoption, but enterprises must address scale, throughput, and cost efficiency issues when deploying ML models into production. Innovation will likely bring down infrastructure costs and make new application patterns commonplace in the future.

Vectors are transforming the AI landscape by enabling new use cases while overcoming long-standing barriers. Machine learning teams struggle to deploy models in production due to high costs, throughput limitations, and the need for database considerations beyond vector search. However, the availability of open-source models and sample applications has accelerated innovation in this space. With strong coding skills no longer a prerequisite, enterprises can now build with these available models, unlocking new opportunities and value creation. Despite challenges, innovation will bring down infrastructure costs and introduce new application patterns, ultimately making it possible for enterprises to deploy AI applications at scale.

Naren Narendran, Aerospike's Chief Scientist, shares insights on how AI and machine learning are shaping modern technology landscapes. He highlights the importance of vector databases in improving large language models like ChatGPT and predicts that advancements in AI will continue to evolve over the next decade or two. Narendran emphasizes Aerospike's role in providing real-time data for AI applications, as well as the potential of vectors in enhancing semantic search capabilities across various media types.

Naren Narendran is Aerospike's Chief Scientist with a strong interest in adapting novel technological and scientific concepts to practical applications in scalable and performant hardware and software. He has spent three decades in various activities, including fundamental research at Bell Labs, leading teams and launching new products at Google and Amazon. Narendran views the current state of AI as having reached a significant scale, with advancements in neural networks and large language models like ChatGPT bringing about increased public attention. However, he notes that there are still deficiencies to be addressed, such as understanding context behind generated content. Real-time data plays a critical role in AI applications, and Aerospike's low latency and high-scale capabilities can provide the necessary confidence in results. Vector search has significant potential, capturing semantics in a uniform way across media types and providing a universal tool for analysis and generation of language. Looking ahead, Narendran expects to see continued advancements in AI and ML, with the current notion of neural networks potentially being replaced or boosted by a technology that takes us to the next level of generalized intelligence.