December 2024 Summaries

Harness, a company focused on enhancing performance and user experience, faced challenges with its pipeline execution search system due to MongoDB's limitations in handling large volumes of data and complex queries. As the scale of operations grew, MongoDB struggled with indexing inefficiencies, complex aggregations, and slow query times, prompting the need for a more efficient search solution. The company transitioned to Elasticsearch, which offered improved full-text and fuzzy search capabilities, better aggregation performance, and effective index lifecycle management. This migration involved adopting a CQRS architecture, where writes were managed by MongoDB and reads were handled by Elasticsearch, optimizing performance and scalability. The new system allowed for faster search results, reduced query latency, and improved resource efficiency, making the infrastructure more robust for future growth.

AI-powered test automation is increasingly becoming the standard for validating enterprise applications, as demonstrated in a comprehensive evaluation of leading AI agents like OpenAI Operator and Anthropic Computer Use against Harness AI Test Automation. The study tested 22 real-world enterprise tasks, including complex banking workflows, audit trail navigation, and messaging application interactions, revealing significant performance differences. Harness AI Test Automation achieved an 86% success rate, outperforming others due to its specialized architecture and enterprise focus, which includes handling intricate interfaces, security requirements, and API-driven validation. While other solutions like browser-use, OpenAI Operator, and Anthropic Computer Use showed varying degrees of success, their performance declined with task complexity. These findings highlight the critical role of tailored AI solutions in enterprise settings, encouraging ongoing enhancements to address evolving challenges and improve software development processes.

Gaurav Nanda discusses the use of Goroutines in Golang to enhance the efficiency of downloading features in devcontainers within Gitspaces, part of Harness' Cloud Development Environment. Features, which add functionality without manual installation, traditionally take a lot of time due to individual processing and dependency resolution. By leveraging Golang's concurrency with Goroutines and channels, this process becomes parallelized, significantly reducing the time spent on feature builds. This is achieved by creating a system that manages a queue of features, processes dependencies, and updates a counter to monitor download progress. Although the current implementation assumes fewer than 100 features, further optimizations such as throttling can improve scalability. This approach not only speeds up the setup of Gitspaces but also enhances developer productivity by allowing more focus on development tasks.

Harness AI Test Automation, a tool designed for software testing, was compared to a general web browser-use solution in a study evaluating their performance on diverse web tasks. The study found that Harness AI Test Automation had a higher success rate, completing 89.5% of tasks compared to browser-use's 68%, particularly excelling in complex scenarios such as enterprise applications and specialized interfaces. While both tools managed simple tasks well, Harness AI Test Automation demonstrated superior capabilities in handling multi-criteria tasks, complex data tables, and specialized web interfaces, proving its design intent for nuanced software testing interactions. The evaluation highlighted the importance of task complexity and the need for specialized tools in handling intricate web tasks, with Harness AI Test Automation providing more adaptive and efficient test definitions through its multi-agent architecture. Future evaluations will expand on these findings, comparing Harness's tool to other emerging technologies, as the company remains focused on enhancing software engineering through AI-powered solutions.

Harness AIDA Support is an AI-driven assistant designed to provide instant and accurate answers to user queries about Harness products, enhancing the efficiency of accessing information in its extensive documentation and resources. By integrating AI technology, AIDA Support converts user questions into numerical representations called embeddings, which are then matched with a vector database to find the most relevant content. This process involves using Google's text-embedding model and Weaviate as the vector database for rapid retrieval of information. Once relevant data is identified, AIDA Support utilizes the OpenAI's GPT-3.5 model to formulate a contextually accurate response, enabling users to receive immediate assistance without relying on traditional support tickets. This innovation not only improves customer support but also ensures continuous access to information, making the user experience with Harness more seamless and efficient.

Databases serve as the crucial backbone for web applications, ensuring the efficient management and storage of user data and application-specific information. This article explores how feature flags can be used to conduct performance testing between different databases, specifically MariaDB and PostgreSQL, by distributing users between them and analyzing query execution times. A simulated environment is set up using Docker and Python, where sample data is loaded into both databases, and a series of queries is executed to compare performance. The experiment uses the Harness FME platform to implement a feature flag that determines which database a user interacts with, and tracks query execution events. The findings reveal that PostgreSQL resulted in a significant 84% reduction in query time overall, with specific queries involving grouping and limits performing much faster, while others like self-joins and SQL date functions were slower. This approach allows for a granular analysis of database performance, aiding in informed decision-making regarding database engine selection and infrastructure optimization for improved user experience.

The blog post by Joy Ebertz delves into the complexities of authentication and identity management systems, highlighting the often confusing landscape filled with acronyms and overlapping terms. Ebertz distinguishes between authentication and authorization through relatable examples, before exploring historical and modern solutions for authentication, such as same-sign on, Kerberos, SAML, and OpenID Connect (OIDC). The article explains how these systems evolved to meet the demands of both on-premise and cloud-based applications, with a focus on federated identity management and the role of Identity Providers (IdPs) and Service Providers (SPs). Ebertz also discusses the convergence of traditional authentication authorities and cloud-based Identity as a Service (IDaaS) providers as they expand their capabilities to include user management and direct login options. This convergence reflects a trend toward integrated Identity and Access Management (IAM) solutions that cater to complex enterprise needs, driven by historical developments and ongoing technological advancements.

Harness utilizes Google BigQuery ML (BQML) to detect cloud cost anomalies through time-series forecasting models, specifically using the ARIMA_PLUS model, which is highly effective for cloud cost data due to its ability to capture seasonal patterns, trends, and irregular spikes. BQML enables in-database machine learning, eliminating the need for complex data transfers and external ML platforms, thus streamlining the process of anomaly detection in cloud cost management. The approach involves preparing cloud cost data in BigQuery, training the ARIMA_PLUS model, and using BQML's ML.DETECT_ANOMALIES to identify suspicious cost spikes. Additionally, BQML supports cost forecasting and requires periodic retraining to stay updated with the latest trends. With its SQL-based machine learning capabilities, BQML offers a scalable and efficient solution for handling large datasets and automating the anomaly detection process, significantly reducing false positives by incorporating seasonality detection. The system is cost-effective, with BQML training costs clearly labeled within Google Cloud Billing for easy tracking and analysis.

Harness, a company focused on enhancing developer productivity, identified slow build times in its harness-core repository due to the use of large Bazel modules and began transitioning to smaller build targets as recommended by Bazel. This shift led to several technical challenges, including a "Class not found" error due to classpath limits, which was temporarily addressed by increasing the CLASSPATH_LIMIT. However, this fix led to an "Argument list too long" issue, eventually resolved by upgrading Bazel to version 5.0.0, which handled long classpath arguments more efficiently. Developers also encountered a "Too many open files" error on macOS, resulting from exceeding the OS's file descriptor limit during builds. Attempts to raise these limits using macOS tools were unsuccessful, so the team extended the default Java toolchain to bypass JVM-imposed limits, thereby stabilizing the build process. These solutions illustrate the complexities of scaling build systems and the iterative process of debugging and optimizing development workflows.

Harness's Commitment Orchestrator is an advanced tool designed to optimize AWS cloud expenditures by providing organizations with enhanced visibility and intelligent management of Reserved Instances and Savings Plans. As cloud operations grow, managing these commitments across various regions and accounts becomes increasingly complex, often leading to inefficient use of resources. Commitment Orchestrator addresses these challenges by offering deep insights into cloud commitments, enabling strategic decision-making, and automating the management of new and existing commitments. It provides comprehensive visibility into commitment-related metrics, such as coverage, utilization, and spending, while offering intelligent orchestration through a recommendation engine that balances savings with flexibility and minimizes risks associated with over-commitment. The tool also facilitates the continuous optimization of existing commitments, ensuring organizations achieve maximum savings and operational efficiency. Ultimately, Commitment Orchestrator transforms cloud commitment management into a competitive advantage by simplifying complex decisions and improving financial accuracy, collaboration, and time savings.

Harness's Policy-Based Cloud Cost Optimization Recommendations Engine is a customizable and scalable solution designed to enhance cloud cost efficiency across AWS, Azure, and Google Cloud environments. Leveraging YAML-based policies powered by Cloud Custodian, the system allows organizations to define and execute cost-saving rules tailored to their unique requirements. It provides a unified approach to cost management by automatically applying policies that detect and address cost inefficiencies without relying solely on vendor-specific recommendations, thus avoiding vendor lock-in. This engine continuously analyzes cloud resources, estimates potential savings, and delivers actionable recommendations, making the cost-saving logic transparent and adaptable to new resources and accounts. By integrating detailed billing and usage reports from cloud providers, the system calculates cost savings and generates data-driven insights to optimize cloud spending. Developed by Anmol Maheshwari at Harness, this tool aims to help organizations reduce unnecessary cloud expenses, enabling them to reinvest savings into growth and innovation while maintaining transparent and automated cost management.

Harness AI Test Automation is a tool designed to enhance the testing process for web applications by allowing tests to be defined using natural language, which simplifies the creation and maintenance of tests without requiring coding expertise. This tool can execute a wide range of web tasks such as information retrieval, e-commerce transactions, enterprise application workflows, travel booking, and interaction with software documentation sites, demonstrating adaptability across different domains. It navigates web pages, evaluates assertions, and self-heals in response to changes, reducing maintenance and increasing reliability. The AI Test Automation framework supports a multiagent approach to handle complex tasks, enabling the automation of user interactions traditionally requiring detailed scripting. However, it still faces challenges with complex data tables and some task instructions. Future work will involve benchmarking this tool against other AI technologies to highlight its performance and areas for improvement.

In 2025, software engineers commonly use AI coding assistants, and the author shares his experiences working on a project to transition API endpoints from Split to Harness, following Split's acquisition by Harness. The project involved updating an API client to accommodate new Harness endpoints while maintaining functionality for existing Split endpoints, which proved challenging due to differences in API structures. The author describes leveraging AI tools like Claude AI within the Windsurf IDE to assist with code generation and problem-solving, emphasizing the need for clear design goals and iterative testing. Despite initial difficulties, such as AI-generated "AI Slop" and compatibility issues with the OpenAPI generator, the author ultimately found AI tools to be significant time savers when used judiciously. The experience highlighted the importance of maintaining a strong understanding of the desired outcomes and actively managing AI-generated code to ensure it meets project requirements. The author concludes with lessons learned about using AI assistants effectively, drawing a parallel to the legend of John Henry to illustrate the potential efficiency gains AI can offer in software development.

Flaky tests, which unpredictably fail and succeed on subsequent runs, are a common issue in the software industry, significantly affecting productivity and costing companies over time. Notably, a 2022 survey reported that companies like Google and Microsoft experience a high percentage of flaky tests, which often lead to manual restarts of failed jobs in continuous integration (CI) environments. To address this problem, Hemanth Sridhar suggests utilizing TestNG's failed tests feature and Harness's Retry failure strategy to create a pipeline for managing flaky tests. This involves running an initial suite of tests, identifying failures with a testng-failed.xml file, and employing a retry mechanism with specific parameters such as retry count and intervals to attempt passing the failed tests again, marking the overall pipeline status based on the final outcomes. Sridhar's approach emphasizes structured retries to improve CI efficiency and reduce the impact of test flakiness on software releases.

APIs can undergo changes that are classified as breaking or non-breaking, with breaking changes potentially disrupting client integrations. While non-breaking changes, often referred to as additive changes, are generally safe, the definition can vary based on client implementation. Common breaking changes include adding stricter server-side validation, introducing new required fields, removing existing response body fields or headers, altering permissions, eliminating an authentication type, removing query parameters, changing field types, removing enum options, significantly altering behavior, or modifying return types and response codes. Exceptions exist, such as more informative error code changes, but success code alterations often lead to issues. These nuances highlight the complexity of maintaining API compatibility and the importance of careful change management.

In an airgapped AWS environment, data synchronization from Google BigQuery to ClickHouse is achieved through a proxy-based networking solution that circumvents strict outbound policies. This setup involves deploying ClickHouse in a Kubernetes cluster using Helm charts, with a corporate proxy server facilitating controlled external communication. Data is initially exported from BigQuery to a Google Cloud Storage (GCS) bucket, from where ClickHouse retrieves it using its GCS function. The proxy server routes ClickHouse's outbound requests, enabling secure data ingestion and analytics within ClickHouse. This configuration, implemented via a ConfigMap-based proxy setup, ensures scalability, flexibility, and security, allowing cross-cloud data workflows despite network isolation constraints. The approach demonstrates how ClickHouse's configuration system and Kubernetes can be utilized to maintain secure and efficient data operations between isolated cloud infrastructures.

Kubernetes faces challenges with efficient node autoscaling due to fluctuating workload demands, leading to either capacity shortages or cost inefficiencies. The Cluster Autoscaler offers a native solution, automatically adjusting the number of nodes based on demand, but it has limitations such as reactive scaling and lack of multi-cluster awareness. Karpenter improves upon this by providing faster, cloud API-driven provisioning without the need for static node groups, but it still lacks advanced spot orchestration and bin-packing capabilities. Harness CCM Cluster Orchestrator builds on Karpenter, offering advanced spot orchestration, dynamic workload splitting, and enhanced bin-packing, while also integrating with Harness’s Commitment Orchestration for cost optimization. This makes Cluster Orchestrator a more efficient and cost-effective solution by proactively managing resource utilization and providing out-of-the-box spot orchestration without additional configuration.

Harness has developed a cloud savings strategy that leverages a sigmoid-based approach to optimize the purchasing of AWS Reserved Instances (RIs). By automating the buying process, the aim is to achieve significant cost savings without overcommitting resources. The previous system triggered abrupt changes in purchasing behavior, resulting in inefficiencies. To address this, Harness implemented a sigmoid function to adjust purchase targets dynamically, ensuring a smooth transition as the target coverage is approached. This method enables aggressive purchasing at lower coverage levels and gradual reduction as the target nears, preventing overspending and enhancing predictability. The adoption of this approach has led to a 5-10% increase in cloud savings and more stable purchasing patterns. Future enhancements may include dynamic smoothing factors and customer-specific tuning to further optimize cost management.

Harness employs a meticulous and structured software release process to ensure high-quality, reliable products for its users, integrating quality checks at every development stage. The process begins with developers writing and testing code locally, followed by deploying it to a personal space using the Harness CI/CD tools. It includes multiple validation stages such as sanity testing, pull request checks, and snapshot builds, which are automated to catch issues early. These stages feature a range of checks, including build number verification, feature flag validation, static code analysis, and security compliance, among others. The release process culminates in a structured rollout across multiple production zones, starting with internal testing in PROD-0 and gradually expanding to broader environments, ensuring that each version is rigorously tested in real-world scenarios before full deployment. The release cycle is repeated weekly, with a strong emphasis on continuous integration, testing, and improvement to maintain high software standards.

Harness' Event-Relay Webhook Trigger is a transformative feature designed to enhance pipeline automation by enabling real-time responses to external events, which significantly improves operational efficiency, reduces server loads, and streamlines development workflows. This solution acts as a central hub for automation, providing immediate pipeline initiation upon event occurrence, eliminating the need for polling-based systems, and offering flexibility to handle multiple event types through a single webhook. The integration process involves creating a Generic Webhook in Harness, configuring it with third-party artifact repositories, and testing it to ensure seamless operation. Additionally, Harness supports various other webhooks, such as Git and Slack, to further automate and adapt pipelines across different platforms, while incorporating HMAC authentication to ensure data security. This event-driven, scalable architecture ensures rapid and reliable processing of events, allowing teams to efficiently manage and respond to changes within their environment.

Software failures, increasingly frequent and disruptive, have become a significant concern, as highlighted in Harness' 2024 Software Failure Sentiment Report, which reveals widespread consumer frustration and a demand for greater accountability. The report shows that over half of U.S. consumers have been impacted by software outages, which are likened to public health crises due to their severe impact on essential services like banking, healthcare, and transportation. This erosion of trust leads to decreased brand loyalty, with a significant portion of consumers calling for enforceable standards and accountability in software development. Businesses are urged to adopt strategies such as canary deployments and feature flags to enhance software resilience and reduce the risk of outages. As demand for regulatory oversight grows, particularly in vital sectors, companies must prioritize reliable software practices to restore trust and maintain their competitive edge in a digital-first world.

Jenkins, a stalwart in continuous integration for over two decades, is increasingly seen as outdated in the face of modern cloud-native and scalable software delivery demands. Originally designed for simpler workflows and a virtual machine-dominated environment, Jenkins now struggles with the complexities of modern applications, plugin chaos, pipeline sprawl, and governance gaps. Migrating off Jenkins is challenging due to its deep plugin dependencies, unique pipelines, and cultural resistance, but offers an opportunity to modernize CI/CD strategies with platforms like Harness. Harness simplifies the transition with features like automated pipeline conversion, a Kubernetes-native architecture, built-in governance, and AI capabilities for continuous improvement. The case of Ancestry.com illustrates the benefits of such a migration, including reduced maintenance effort, increased deployment frequency, and decreased downtime. Embracing newer platforms not only addresses the limitations of Jenkins but also positions organizations for future success in software delivery.

In 2025, the DevOps landscape is expected to undergo significant changes, driven by the rise of agentic AI, which will surpass generative AI in its impact on software delivery processes. AI-generated code will increase security risks and developer workload, necessitating the implementation of robust security measures to streamline quality and security testing. The emergence of cross-functional engineering teams will replace traditional standalone DevOps teams, fostering innovation by integrating various disciplines into cohesive units. Additionally, the reduction of onboarding barriers through technological advancements will facilitate outsourcing, enabling companies to address skill shortages efficiently. Human connection and empathy will become crucial differentiators as AI takes on more technical tasks, emphasizing the importance of aligning AI-driven operations with real-world challenges. The growing role of AI will also lead to increased specialization, allowing professionals to focus on areas of expertise and enhancing job satisfaction. These trends highlight the importance of preparing for transformative shifts in DevOps, emphasizing collaboration, automation, and continuous improvement to maintain a competitive edge.

In the context of modern engineering practices, such as microservices and DevOps, tracking service health and engineering initiatives has become increasingly complex, highlighting the limitations of traditional spreadsheets. Scorecards in Internal Developer Portals (IDPs) offer a more efficient alternative by providing centralized visibility, real-time updates, and automation that reduces human error. Unlike spreadsheets, which require manual updates and can become siloed, IDP scorecards integrate with existing tools and allow for customizable metrics and dashboards, enhancing collaboration and contextual insights. They also offer scalable solutions that can seamlessly handle the growing number of services and metrics within an organization, while providing actionable insights through trend analysis. Moreover, scorecards improve developer experience by consolidating metrics and insights in a single platform, thereby enhancing efficiency and reducing workflow disruptions. As organizations grow and systems become more complex, adopting IDP scorecards represents a strategic move towards maintaining resilient systems and aligning on strategic initiatives.

Infrastructure automation has significantly evolved, transitioning from manual configurations to sophisticated self-service platforms, enhancing consistency, scalability, and governance in deployments. Tools like OpenTofu, which treat infrastructure as code, have revolutionized this space by providing transparency, adaptability, and reduced vendor lock-in. The evolution of infrastructure management is marked by four stages: initial automation through version control, collaborative workflows, specialized infrastructure pipelines, and advanced orchestration. The final stage involves fully self-service models with robust governance and compliance mechanisms, where security and compliance can be automated. OpenTofu, as a community-driven alternative to Terraform, offers advantages such as greater transparency and faster innovation. Infrastructure automation is now a strategic necessity, allowing organizations to turn infrastructure from a cost center into a competitive advantage by integrating with CI/CD pipelines and other tools.

Internal Developer Portals (IDPs) have become essential tools in modern software development, providing centralized access to tools, services, documentation, and processes to enhance productivity, standardization, and development velocity. The text outlines a comprehensive guide for evaluating IDPs, emphasizing the need for a strategic approach that involves defining organizational goals, assessing usability and developer experience, and ensuring effective integration with existing tools. Key considerations include support for self-service infrastructure, search and discoverability, observability, scalability, performance, and security. The guide stresses the importance of continuous improvement based on developer feedback and usage data to ensure that the IDP remains an effective tool for enhancing developer experience and streamlining workflows.