September 2026 Summaries

Playwright’s command-line tools are presented as effective for executing, parallelizing, and reporting browser tests, while common testing bottlenecks instead involve authoring coverage and maintaining tests after interface changes. The comparison distinguishes conventional framework CLIs, including Cypress, WebdriverIO, TestCafe, and Maestro, from agentic tools that use natural-language, visual, Gherkin, or codebase-driven approaches to create or repair tests. Cypress is positioned as a close JavaScript-focused alternative with an interactive runner, WebdriverIO and Maestro extend coverage to native mobile, and TestCafe avoids separate WebDriver setup. Among agentic options, Kane CLI uses plain-English objectives and emits structured NDJSON results, Shortest embeds natural-language steps in Playwright TypeScript tests, Midscene relies on screenshots for selector-poor interfaces, Hercules interprets Gherkin without step definitions, and Autonoma generates tests from a repository and runs them on pull-request previews. The article notes that agentic tools typically require language-model access and can create ongoing usage costs, while portability, licensing, CI output, browser or device coverage, and maintenance behavior should be evaluated against a team’s actual constraint; it also discloses that TestMu AI, maker of Kane CLI, has an interest in one of the products reviewed.

Browser MCP alternatives differ mainly by where automation runs and how much work is delegated to the agent versus the tool. Chrome DevTools MCP is presented as the strongest local option for live Chrome control, network inspection, and unique performance tracing, with configurable tool categories and compressed screenshot formats to reduce context use. Selenium MCP supports Chrome, Firefox, Edge, and Safari through WebDriver, making it suitable for existing Selenium environments and cross-browser needs, while Kane CLI accepts plain-language objectives and returns verified pass-or-fail results with evidence, exports to Playwright, and incurs usage-credit costs. Mobile MCP extends automation to iOS and Android simulators, emulators, and real devices using accessibility-tree-based interaction, and ExecuteAutomation Playwright MCP retains Playwright compatibility while adding test-code generation and API testing. The comparison cautions that large tool sets consume agent context and notes that Puppeteer MCP was archived in May 2025 and Midscene discontinued its MCP packages, emphasizing the need to verify project maintenance before adoption.

At Testμ Conf 2026, Salesforce engineer Monika Sharma explained how DeepEval, an open-source Python framework often described as “pytest for LLMs,” evaluates non-deterministic LLM responses through semantic scoring by a judge model rather than exact string matching. DeepEval provides RAG metrics including faithfulness, answer relevancy, contextual relevancy, precision, and recall, along with agent, safety, conversational, and customizable G-Eval metrics, producing scores from zero to one, pass/fail results, and written rationales. Sharma recommended starting thresholds around 0.5 and raising them as model behavior becomes stable, while noting that a “none” result generally signals an output-format parsing problem rather than poor quality. She positioned evals as either unit or integration tests depending on whether they assess direct agent APIs or UI-based experiences, and said they can be incorporated into standard CI/CD pipelines. However, she emphasized that human judgment remains necessary for multimodal outputs such as chatbot responses containing images, emoticons, and feedback controls, which text-focused evaluation may not understand. Production autonomy should be supported by many varied utterances and consistent results across repeated runs, not isolated passing tests, while agents should generally have read-only access to knowledge bases for security reasons.

At Testμ Conf 2026, Sanity CEO and co-founder Magnus K. Hillestad argued that AI agents are highly capable yet unreliable because current approaches treat them too much like digital employees rather than programmable, short-lived, parallelizable systems. Drawing parallels with alchemy and the Industrial Revolution, he suggested that today’s investment in models, chips, data centers, and networking may resemble the canal-building phase that enabled later breakthroughs, while the crucial emerging challenge is creating an organizational “operating system” for human and machine work. Sanity’s experiments illustrated both potential and limitations: its multi-agent Myriad system reportedly increased some engineers’ output more than tenfold but generated excessive coordination overhead and costs exceeding $10,000 on some days, while a later direct-delegation system called Persona reduced costs by over 90% with comparable or improved output, though agents could still fail to complete delegated tasks. Hillestad concluded that effective agent systems require programmable clarity, context, checkpoints, correction, verification, and higher-level ways to express intent, leaving important open questions about coordination, non-determinism, context management, and leadership in mixed human-AI organizations.

At Testμ Conf 2026, Paramount Quality Engineering Manager Partha Sarathi Samal presented “backwards scoring,” a method for evaluating automated tests by replaying every logged production incident against every candidate test rather than relying solely on code coverage. Tests receive one of three verdicts: proven if they uniquely catch a real incident, duplicate if they detect an issue already caught earlier by another test, or unproven if they have no historical catches. In an 18-month study of approximately 4,200 tests, 80% were proven, 8% duplicate, and 12% unproven, although no tests were automatically removed. Samal emphasized that the method measures only documented reactive catch history and can undervalue preventive tests that stop defects in CI before production incidents occur, while incomplete incident logging can make an otherwise useful suite appear unproven. He therefore recommends treating scores as evidence for owner review rather than deletion decisions, using them alongside coverage metrics and complementary approaches such as mutation testing, test impact analysis, chaos engineering, and observability.

TestMu AI engineering leaders Sai Krishna and Srinivasan Sekar presented an approach to testing customer-facing video AI agents by automating the human participant in live calls rather than relying on manual testers. Their system creates configurable “test candidates” with distinct personas, avatars, languages, dialects, behaviors, and memories, allowing repeated scenarios across diverse participants through joinable WebRTC links without SDKs, code changes, or test hooks. Because video agents must handle facial expressions, lip sync, audio, sentiment, turn-taking, and long-term context in addition to spoken language, evaluation extends beyond transcripts to assess full recordings, multi-turn conversational quality, goal completion, avatar presentation, and criteria-supported evidence. The presentation highlighted difficult technical issues including real-time latency, lip synchronization, distinguishing low audio levels from silence, interruptions and pauses, and memory-based red teaming for multi-round interviews. The platform generates scenarios from supplied context, supports user-defined validation criteria and personas, runs tests through HyperExecute, and is expanding toward native meeting apps, noisy environments, multi-person detection, and additional visual-attention metrics, while acknowledging that sign-language support remains a gap.

Piramal Finance’s Testμ Conf 2026 session described how its Build Your Own Tech (BYOT) platform expanded software delivery from nearly no non-engineering contributors two years ago to 947 builders, 1,306 projects, and 369 production applications. BYOT lets business teams create internal tools through a browser-based environment while engineering embeds deployment, authentication, databases, secrets, approved integrations, and access controls as governed guardrails. KriyaOS extends this approach across requirements, mockups, user acceptance testing, monitoring, and automated remediation, with shape notes requiring agreement before code generation to reduce iteration costs. Quality is mandatory rather than optional: integration, regression, and smoke tests are generated automatically, run as CI gates, and can trigger tickets and repair attempts after failures. Although many projects were abandoned or remain unfinished, the speakers argued that successful applications, including some that substantially reduce manual work, justify experimentation and enable smaller business improvements that conventional prioritization would overlook. They emphasized that engineering has shifted toward maintaining reusable standards, secure defaults, API catalogs, and platform infrastructure, while people retain responsibility for product decisions, compliance policies, architecture of core systems, and judgment about what should be built or released.

Nilesh Dalvi of Glean argues that AI coding tools can significantly accelerate code creation but have limited effect on overall delivery velocity because coding represents only about 20 percent of engineering work; making it five times faster reduces a 100-hour cycle to roughly 84 hours. The larger opportunity lies in the surrounding work of deciding what to build, verifying changes, deploying safely, monitoring production, and learning from outcomes, which requires AI to operate as a “coworker” rather than merely an assistant. This shift depends on making organizational context accessible across systems, exposing tools through programmatic interfaces, triggering workflows from events such as escalations or failed evaluations instead of manual prompts, and applying judgment about whether to act, ask for help, stop, escalate, or roll back. Dalvi emphasizes gradual, risk-based autonomy supported by clear permissions, evidence, metrics, guardrails, and rollback mechanisms, while treating human edits, rejected changes, and reversions as valuable feedback that improves the system over time.

Agentic AI testing tools are presented as platforms that can derive tests from stated intent, execute them in live browsers or devices, interpret failures, and repair tests when interfaces change, shifting human work toward setting objectives and reviewing outcomes. The comparison ranks 15 products using autonomy, test-code ownership, coding-agent integration, execution breadth, and reviewability, distinguishing tools that export maintainable code, such as TestMu AI, QA Wolf, and Checksum, from vendor-hosted platforms focused on no-code automation, managed services, visual validation, enterprise governance, or mobile testing. It highlights specialized strengths including Meticulous’s session-derived tests, Applitools’ visual AI, UiPath Test Cloud’s audit and compliance features, Tricentis Tosca’s packaged-enterprise support, and Revyl’s mobile-first approach. The account also separates testing conventional software with agents from evaluating AI agents themselves, which requires scoring qualities such as hallucination, bias, completeness, and context awareness rather than fixed assertions. Common constraints include CAPTCHAs, browser downloads, deeply nested frames, canvas or WebGL content, long animations, and inadequate requirements documentation, while selection guidance emphasizes deciding whether tests remain portable after a contract ends, matching tooling to development-agent workflows and application surfaces, and trialing products on difficult real-world flows rather than simple login tests.

Microsoft Power Platform engineering described an agent-assisted test automation approach for a monorepo with more than 100 packages and 14,000 tests, where running every test on each pull request previously delayed feedback by three to four hours. Its architecture separates probabilistic LLM-based reasoning from deterministic Playwright execution, using specialized planner, generator, healer, and reporter agents while requiring human review for every generated test or repair. Risk-based test selection analyzes code changes through abstract syntax trees, dependency graphs, failure history, and business criticality to prioritize impacted tests, while full regression suites continue to run nightly and at release gates. The system also measures integration-test coverage by matching business-rule intent to test specifications, identifies gaps before pull requests merge, and uses evidence such as logs, DOM snapshots, screenshots, and execution history to classify failures as product, test, or infrastructure issues. Proposed self-healing fixes are repeatedly validated in isolated “gauntlet” runs before being submitted as draft pull requests. Reported pilot outcomes included 60% faster regression feedback, 40% less test repair effort, 30% less manual triage, and 90% accuracy in product-versus-test-bug classification, though these were presented as internal observations rather than general benchmarks. The presentation emphasized that successful large-scale automation depends on governance, auditability, data privacy, isolated environments, historical execution data, and phased adoption rather than unrestricted AI automation or replacing existing test suites outright.

Agentic QA tools are distinguished from conventional automation by their ability to create tests from stated intent, execute them on live applications, interpret outcomes, and repair tests after interface changes without manual selector updates. The comparison ranks ten products using autonomy, test ownership, coding-agent integration, execution breadth, and reviewability, highlighting TestMu AI for agent-invokable verification and large-scale device coverage, QA Wolf for exportable Playwright and Appium tests with managed triage, TestSprite for exploratory suite generation, and testRigor for broad plain-language testing across web, mobile, desktop, APIs, and mainframes. Other tools specialize in consolidated cross-surface testing, visual validation, enterprise packaged applications, governed and auditable testing, or mobile-first workflows. The discussion notes common limitations involving CAPTCHAs, downloads, nested iframes, canvas or WebGL content, long animations, and sparse requirements documentation. It also separates using agents to test conventional software from evaluating AI agents for qualities such as hallucination, bias, completeness, and context awareness, and recommends selecting tools based on ownership requirements, application surface, coding-agent compatibility, governance needs, and performance on a difficult real-world user flow.

AI-powered testing tools use machine learning and language models to generate tests, maintain them as interfaces change, interpret failures, assess visual output, and discover coverage opportunities, aiming to reduce the engineering effort required to sustain regression suites. The comparison evaluates 15 platforms based on verified vendor claims, meaningful AI involvement in at least two testing functions, and whether users retain their tests as portable code or depend on a vendor platform. TestMu AI is positioned as a broad lifecycle option with export to Selenium, Playwright, Cypress, and Appium; testRigor emphasizes plain-language, resilient test specifications; Applitools focuses on visual validation; Meticulous creates and prunes tests from recorded user sessions; UiPath emphasizes governed, audit-ready enterprise automation; and Tricentis targets model-based testing for major packaged software platforms. Other products specialize in areas such as managed code ownership, exploratory test generation, pull-request-based repairs, mobile testing, desktop coverage, localization, and AI-feature evaluation. Across the category, common technical limitations include CAPTCHAs, downloads, deeply nested frames, canvas or WebGL validation, long transitions, and sparse requirements documentation. Selection should begin with the primary bottleneck—such as authoring, maintenance, coverage discovery, visual correctness, governance, or platform support—then be validated through trials involving realistic, complex workflows and consideration of whether the resulting test suite remains accessible after a contract ends.

Mahesh Sathiamoorthy argued at Testμ Conf 2026 that AI agents often fail in production not because frontier models lack general capability, but because they were trained on task distributions unlike an organization’s specific workflows, making reliability a data and evaluation problem. He described agent data as reinforcement-learning environments—replicas of production systems containing tasks, available tools, and verifiers that determine whether goals were actually achieved—and identified these environments as the scarce, proprietary asset compared with widely available compute, models, and optimization libraries. Organizations can improve agents by post-training models, automatically evolving prompts through approaches such as GEPA, and adapting the agent harness and tools, while using production failures to create new test environments in a continuous data flywheel. Examples involving Snowflake data-engineering tasks and Credit Karma credit-card recommendations showed how curated environments can support systematic evaluation, reduce hallucinations and compliance risks, and enable smaller open models to lower cost and latency. Sathiamoorthy also emphasized testing representative high-impact failures before deployment, moving lengthy policy instructions from prompts into model weights through post-training, and managing the infrastructure costs of running and restoring large numbers of virtualized environments.

Gil Zilberfeld’s Testμ Conf 2026 session argues that testing LLM-based systems requires replacing fixed pass-or-fail assertions with a layered evaluation funnel because identical prompts can produce different responses across runs. He recommends thoroughly testing deterministic scaffolding, APIs, integrations, error handling, and guardrails with conventional tests, then applying low-cost sanity checks to model outputs for required sections, relevant entities, timing, and other baseline criteria before deeper evaluation. Semantic quality should be defined through small human-authored golden data sets that specify what acceptable responses contain, enabling scorecards that rate multiple criteria and track quality trends over time rather than relying on a single snapshot. The session emphasizes that prompt changes alone are not reliable bug fixes, illustrated by an example in which a request for concrete test data led to credit-card-like values; durable fixes require code-level enforcement, updated privacy requirements, and automated checks. Zilberfeld also highlights token costs, model drift, prompt injection, jailbreaks, safety, bias, and the need to make any unacceptable failure deterministic through logic outside the model or protective guards. He concludes that testers’ work is increasingly about quality architecture: understanding risks, defining “good enough,” and designing systems that can safely use non-deterministic AI outputs, especially when agents act on responses without a human reader’s judgment.

Agentic test management tools use AI to support test-planning activities such as generating and refining test cases, prioritising execution by risk, interpreting failure patterns, identifying coverage gaps, and maintaining traceability between requirements, tests, runs, and defects; unlike execution tools, they primarily manage the repository and reporting process. The comparison evaluates ten platforms based on confirmed vendor documentation and distinguishes them by the extent to which their AI supports ongoing repository maintenance rather than only initial case creation. Test Manager by TestMu AI is positioned for large, aging repositories through case generation, refinement, and end-to-end traceability; Testomat.io focuses on autonomous application exploration and automated-failure clustering; TestRail and aqua cloud emphasize auditability, compliance, and data-control options; Xray and Zephyr Scale provide Jira-native workflows; PractiTest centers on risk-based prioritisation and release-readiness decisions; QMetry supports broad toolchain integration and flaky-test detection; Tricentis qTest serves enterprise programs using a wider automation and analytics stack; and Testmo combines manual, exploratory, and automated testing in one workspace. The text notes that AI-generated tests remain dependent on requirement quality, generated volume does not ensure meaningful coverage, these management systems still require separate execution tools, and traceability depends on disciplined, current source requirements, recommending that teams select tools by their most costly planning problem and trial them against real legacy repositories rather than demonstration projects.

A Testμ Conf 2026 panel examined how agentic engineering is reshaping quality engineering organizations, with speakers describing a shift from embedded QA roles and test-case volume metrics toward pooled, risk-based teams, governance responsibilities, and measures such as change failure rate, defect removal efficiency, severity-one incidents, false positives, and turnaround time. CITY Furniture reported improving defect removal efficiency from about 30% to 75–80% and limiting production bugs to one or two over roughly six months after adopting agent-assisted workflows, while keeping headcount stable through upskilling. Panelists agreed that AI has accelerated development and automation, but differed on the future of exploratory testing: one saw manual exploratory work shrinking sharply, while another argued human experts remain necessary to investigate failures, validate agent output, and improve automated systems. They emphasized that business stakeholders ultimately make release decisions using quality evidence, while QE increasingly acts as a cross-functional governance layer. Because AI agents are non-deterministic, teams must evaluate their consistency, accuracy, hallucination rates, speed, and failure patterns, maintain detailed execution evidence, and create feedback loops that allow agents to learn from mistakes. The discussion concluded that durable QE careers will depend on combining AI fluency with system design, release-process knowledge, business understanding, and the ability to both perform work and train agents to assist with it.

Rhea Goel’s Testμ Conf 2026 session examines how large language models can augment production ranking systems by replacing hand-crafted objective functions with natural-language instructions while retaining established behavioural models and deterministic controls. She explains that rankings depend on user context and must balance relevance, business goals, diversity, price, and quality, making nuanced product requirements difficult to translate into fixed thresholds and weights. Base LLM prompting can reason about intent but lacks platform-specific knowledge such as loyalty, pricing, and cancellation preferences, so effective systems require supervised fine-tuning on ranking data and preference alignment through methods such as RLHF or direct preference optimisation. Hard requirements should remain outside the model as deterministic constraints, while LLMs can handle ambiguous soft preferences consistently through alignment and evaluation. To meet subsecond production latency and cost limits, the approach uses distillation, prefix caching, routing, and cascaded models, while evaluation combines standard ranking metrics such as NDCG and MRR with rubric-driven LLM judges. Goel concludes that LLMs are unlikely to replace traditional rankers outright; instead, hybrid architectures can combine behavioural prediction, semantic reasoning, controllability, and efficient infrastructure, with agentic components potentially expanding ranking systems through external data retrieval and multi-step reasoning.

Salesforce’s Service Cloud quality engineering team has adapted its testing strategy for AI agents through four phases: evaluating each stage of an agent’s reasoning and execution rather than only its final response, creating industry-specific environments based on real customer scenarios, using AI-driven agents to test complex voice interactions, and integrating customer-facing evaluation tools into Agentforce Studio. John Liang emphasized that deterministic assertions remain necessary alongside LLM judges, trace validation, metrics, and verification that an agent actually completed the task it claimed to perform. Production deployments, including work with Singapore Airlines, showed that internal tests often missed domain terminology, multi-part requests, and customer behavior patterns. Voice testing introduced further challenges such as accents, noise, interruptions, emotional states, agent handoffs, retrieval, latency, recovery, and human escalation. Salesforce simulates high call volumes with virtual calls and evaluates selected conversations for accuracy, completeness, conciseness, tone, and voice quality. Liang also identified retrieval among similar knowledge documents as a persistent weakness, arguing that continuous, representative, multi-turn testing in CI/CD is needed to measure and improve AI-agent performance across complete customer journeys.

A Testμ Conf 2026 discussion featuring Infosys quality engineering CTO Harleen Bedi argued that humans remain accountable for software releases even when AI writes, runs, and evaluates tests. As AI systems introduce nondeterminism, model drift, AI-generated test artifacts, and unpredictable failure impacts, traditional coverage metrics and one-time quality gates are becoming insufficient. Bedi advocates a continuous governance layer built around explicit thresholds, traceability of AI decisions and prompts, and post-production observability to detect and mitigate drift quickly. She describes a shift toward pipeline-integrated controls and calculated risk, while enterprises standardize approved models and tools through centrally governed internal platforms. Testing AI agents requires multiple layers covering user-facing business outcomes, responsible AI concerns such as fairness and bias, security threats including prompt injection and jailbreaks, and agentic efficacy, which examines an agent’s decisions and tool-use path rather than only its final response. The discussion concludes that quality engineers can remain central by contributing domain expertise, developing AI evaluations, and building trust and governance practices for AI-enabled and agentic systems.

Mesut Durukal’s Testμ Conf 2026 session examined how a four-hour, unreliable test pipeline became a delivery bottleneck through accumulated design and process decisions rather than a single failure. He identified root causes including excessive system-level testing, repeated setup and checks, unnecessary infrastructure, unreadable over-abstraction, hidden verification logic, overly complex helper functions, unproven test stability before merges, swallowed errors, duplicated interaction code, and inadequate reporting and result retention. Durukal argued that flakiness is primarily an implementation and engineering issue rather than a tooling issue, emphasizing readable tests, appropriate test-pyramid placement, pipeline-based stability checks, explicit failures, and reports that preserve historical evidence. He also described three AI-supported capabilities for defining tests from requirements, implementing them according to repository guidelines, and checking pull-request readiness, while cautioning that AI output requires review, performance scrutiny, cost control, and continued human understanding. He concluded that automation debt is unlikely to be addressed through pauses in feature delivery and instead must be reduced alongside development through shared responsibility among QA and engineering teams.

At Testμ Conf 2026, Senior QA Consultant Wessel Braakman argued that tester-developer conflict often stems less from process than from how bug reports are introduced, illustrating his point through five informal developer personas: drivers, analysts, supporters, enthusiasts, and systems thinkers. He showed that the same defect can receive different responses when framed around user impact for drivers, evidence and reproducible steps for analysts, collaborative discussion for supporters, bounded creativity for enthusiasts, and wider dependencies for systems thinkers. Reproduction steps were presented as a universal requirement, while the personas—four loosely influenced by DISC and one from Braakman’s experience—were described as practical observations rather than a formal model. He encouraged testers to combine traits from all types, balancing analytical rigor, system awareness, collaboration, creativity, and decisiveness, and noted in the Q&A that testers can help align quality goals among developers and business leaders, adapt support for junior and senior developers, collaborate on testable requirements, and ensure AI-driven knowledge-base changes remain traceable and subject to human review.

At Testμ Conf 2026, Capgemini insurance testing leader Lisha Rakesh described the “bionic workforce” as a maturity model for human-AI collaboration in software testing, arguing that organisations should not pursue immediate headcount reductions before progressing through assisted, augmented, and autonomous modes. In assisted mode, AI provides analysis and recommendations while testers retain all decisions; augmented mode shares work, with AI generating broad test coverage and humans supplying context, prioritisation, ethical judgment, and domain expertise; and autonomous mode lets agents execute workflows such as environment provisioning, regression testing, failure analysis, and defect creation under human governance. Rakesh said adoption is increasingly widespread and AI is shifting testers from activity-based tasks toward business risk, customer impact, and decision-making, but trust, explainability, privacy, and governance remain greater barriers than technology, particularly in regulated sectors. She proposed measuring progress through decision velocity, human effort reallocation, agent autonomy, and a trust index, rather than traditional counts of scripts, tests, or defects, and noted that work should shift from roughly 80% execution in assisted environments to 80% strategy and orchestration in autonomous ones. Her leadership recommendations included focusing on capability density rather than headcount, managing ecosystems of people and agents, developing AI and domain skills, and establishing trust-centered governance, while emphasizing that AI should remove repetitive work rather than replace the human curiosity, judgment, empathy, and accountability required for quality decisions.

At Testμ Conf 2026, Piramal Finance AI leader Jaydeep Chakrabarty presented “Context Engineering 2.0,” an approach intended to extend retrieval-based AI context layers by continuously deriving traceable conclusions from relationships among organizational records. He argued that conventional retrieval and GraphRAG systems can retrieve only explicitly recorded information, whereas a knowledge graph can connect dispersed signals—such as a skipped login test, increased logout complaints, and a session-timeout deployment—to identify a likely underlying defect. His Nodeex library builds such graphs through document acquisition, extraction, entity resolution, schema creation, and required human review, while preserving provenance for each source and derived claim. The system uses multi-hop relationships and eight analytical lenses, including contradiction, dependency, timing, and recurrence, to generate and score potential insights that downstream systems can use. Chakrabarty emphasized that generated ideas should remain grounded in graph evidence, with human feedback, access controls, masking, deduplication, and authorization serving as safeguards against hallucinations, poor-quality data, and inappropriate disclosure.

Rushabh Mehta’s Testμ Conf 2026 session argued that evaluating AI agents differs from conventional testing because long, tool-driven tasks can silently compound an early error across hundreds of steps, creating costly and difficult-to-diagnose failures. He described an eval harness as a combination of models, tools, tasks, captured execution trajectories, graders, and measurements such as cost and token usage, designed to verify both desired behavior and prohibited behavior across regressions and deployment environments. Because LLM behavior remains inherently non-deterministic, teams should reduce variability through fixed settings, schemas, repeated pass@k trials, and deterministic checks rather than expect to eliminate it. The session emphasized testing hallucinations by making necessary information inaccessible and rewarding honest abstention, while safeguarding real-world tool calls through idempotency, backoff, compensation mechanisms, detailed failure classification, checkpoints, and rollback. Mehta also highlighted memory management, including provenance, expiration, confidence levels, and structure-aware chunking, as essential for avoiding stale or misleading agent behavior. He recommended using code-based graders wherever possible, supplementing them with rubric-driven model judges and human review for nuanced judgments, and noted that benchmark results from GAIA 2 show strong progress in search and execution but persistent weaknesses in temporal reasoning, ambiguity, and adaptability.

A Testμ Conf 2026 session led by Salesforce engineering manager Sobhitha Neelanath argued that conventional deterministic testing metrics, such as a 94.2% pass rate, can obscure serious failures in probabilistic agentic systems, illustrated by a recursive refund loop that issued $400,000 before alerts activated. She proposed measuring trust through an equation combining alignment and predictability while minimizing blast radius, supported by indicators including conversational accuracy, intent continuity, output variance, and mitigation capability. The session categorized risks by agent autonomy, from context hallucinations in assistants to cascading workflow failures in autonomous agents, and recommended progressively stronger controls such as grounded assertions, human review gates, and transaction isolation. Neelanath also urged leaders to replace broad test coverage and productivity counts with impact-based regression, shorter feedback cycles, defect leakage measures, and commercial reliability outcomes, citing a reported 78% gap between executive expectations and actual agentic performance. Her team’s use of “chaos audits,” which rewards engineers for deliberately inducing failures and improving safeguards, was associated with reported reductions in production loops and burnout. She described an evolving quality-engineering path from script automator to evaluator, trust architect, and governance-focused guardian, requiring semantic evaluation skills, statistical rigor, and compliance awareness while retaining human involvement for complex customer workflows and system design decisions.

At Testμ Conf 2026, Databricks technical evangelist Viktoria Semaan argued that controlling LLM costs and proving model quality require the same foundation: a task-specific evaluation set that measures accuracy, latency, and cost across models. She recommended starting with the smallest suitable model, escalating only when evaluations show it fails, and using a model-agnostic gateway to avoid provider lock-in while centrally managing routing, budgets, access policies, and traffic experiments. Her demonstration of 16 models found that smaller open-weight models can match proprietary alternatives on straightforward classification tasks at far lower cost, although frontier models remain more effective for complex reasoning tasks such as writing human escalation briefs. Fine-tuning should be reserved for cases with recurring errors, several thousand labeled examples, and unsuccessful prompting or retrieval-augmented generation attempts; it improved classification performance but did not replace frontier reasoning in every scenario. Semaan also emphasized continuous evaluation using production traces, smart routing based on task complexity, and caching, prompting, and RAG as additional optimization methods, while identifying model choice as the largest potential source of savings.

A Testμ Conf 2026 panel on AI in banking, insurance, and financial services argued that the principal obstacle to moving AI pilots into production is not technology but trust, including whether outputs can be explained, audited, legally justified, and assigned clear accountability. Because AI systems are probabilistic and agentic systems can take consequential actions such as denying insurance claims, traditional pass-or-fail testing must evolve into “trust automation,” using multi-dimensional scorecards, adversarial and bias testing, uncertainty detection, and end-to-end review of an agent’s actions and their consequences. Panelists emphasized that human oversight should remain at defined points throughout development and deployment, while agents earn greater autonomy gradually through low-risk use cases, guardrails, independent verification, and ongoing monitoring for data-driven drift. Testing was presented as a practical proving ground for AI adoption because it can improve development velocity without directly affecting customers, while broader workforce changes will place greater value on domain expertise, judgment, orchestration, and governance. The discussion concluded that access to advanced models will become commonplace, making institutional readiness, data strategy, technology modernization, governance frameworks, and regulatory alignment the key differentiators for organizations seeking to deploy AI confidently and responsibly.

Amanda Martin’s Testμ Conf 2026 session examined how system-prompt design can substantially affect voice-agent reliability, efficiency, and caller experience, using three restaurant-booking agents with identical models, tools, and test scripts but different prompts. A personality-focused prompt produced a pleasant call but failed to confirm critical details, while a rigid numbered prompt caused the agent to narrate its procedure and create an overly long, impractical conversation. A structured best-practices prompt combining personality, knowledge, state and ambiguity handling, workflow and tool rules, examples, and recovery instructions was the only one to clarify the ambiguous phrase “next Friday,” while also delivering the best quality and shortest calls. Martin recommended evaluating agents through a layered process of manual demos, chat simulations for tool sequencing, and full voice simulations for interruptions and turn-taking, with atomic pass-fail criteria and additional deterministic transcript checks where risk warrants. She also emphasized that model selection and platform settings are central to handling accents, multilingual conversations, and overlapping speech, while prompts can support these capabilities through pronunciation examples and clear operational boundaries.

Microsoft’s Francesca Lazzeri argued at Testμ Conf 2026 that generic AI benchmarks can overlook critical agent failures, such as violating refund policies, approval limits, or instructions embedded in tool results, even when scores for helpfulness, groundedness, and safety are high. She presented ASSERT, Microsoft’s open-source Adaptive Specification-driven Scoring for Evaluation and Regression Testing framework, which turns plain-language requirements about what an agent must and must not do into behavior taxonomies, generated tests, and scored evaluation results. Her four-layer evaluation loop begins with written specifications, adds shared baseline metrics such as groundedness, retrieval quality, relevance, fluency, and safety, incorporates domain-specific measures including task adherence and tool-call accuracy, and uses production observability to identify unexpected failures that become future tests. Using a support assistant with refund capabilities as an example, she emphasized that evaluating only final answers can obscure unsafe or noncompliant action sequences. The observability layer also links quality with token costs, workflow outcomes, revenue, and controlled or causal experimentation to assess business impact. Lazzeri concluded that reliable agent evaluation requires both standardized measures for comparison across systems and custom metrics developed with product teams, domain experts, and users.

A Testμ Conf 2026 panel argued that AI-generated code has shifted software engineering’s main constraint from writing code to validating, securing, reviewing, and trusting it at scale. The speakers described an “agentic software factory” as a broader system rather than a standalone coding agent, requiring organisational context lakes, guardrails, dedicated verification environments, telemetry connected to SDLC artifacts, and deterministic tools that agents can invoke. They cautioned that probabilistic agent behavior may be acceptable for bounded tasks with human oversight but compounds into a significant trust problem across thousands of repositories, making compiler-accurate code models, tested workflows, and risk-based controls important. Rather than measuring token consumption, license use, or AI-generated code volume, the panel recommended tracking outcomes such as changes that merge, remain in production, support strategic initiatives, and reduce bugs, incidents, and human effort. They concluded that autonomy should be introduced gradually in areas with strong verification loops and low risk, with humans retaining judgment and oversight while organisations determine their own acceptable boundaries.

Nishant Gupta’s Testμ Conf 2026 session argues that AI agents should be treated as probabilistic planners operating within deterministic infrastructure, rather than being trusted to execute production actions directly. He describes risks including incorrect actions, unbounded retries, cost escalation, and false reports of success, and recommends an execution runtime that validates typed tool requests, enforces policy, manages retries, records durable state, verifies outcomes, and supports rollback. Key controls include strict tool contracts, scoped credentials, idempotency, rate and spending limits, approval gates for high-risk actions, and zero-trust authorization based on identity, data sensitivity, tenant boundaries, and change policies. Gupta also distinguishes agent evaluations from chatbot evaluations by emphasizing tool selection, permissions, failure recovery, budget compliance, and escalation behavior, while calling for semantic observability that can explain why an agent acted. He recommends progressive autonomy, beginning with suggestions and dry runs before moving toward tightly bounded production execution, with advancement based on evaluation results, incident rates, approval patterns, rollback reliability, budget stability, and audit completeness.

At Testμ Conf 2026, QA instructor Rahul Shetty discussed how AI agents are reshaping testing careers, arguing that professionals should first master browser automation, API testing, testing fundamentals, and a programming language before adding agent-related skills such as MCP servers, skill files, context engineering, and AI evaluation. He recommended Playwright for beginners because it combines browser and API automation, while suggesting JavaScript or TypeScript for newcomers and advising experienced Java users to retain their existing language expertise. Shetty said meaningful AI experience in interviews requires practical knowledge of enterprise model hosting, data privacy, governance, token routing, and secure use of open-source and paid models rather than familiarity with consumer tools alone. For organizations adopting agentic QA, he contrasted building self-hosted infrastructure with subscribing to enterprise platforms that provide models, cloud capacity, and token management. He predicted that SDET roles will evolve toward designing agent goals, scenarios, and validation processes, potentially reducing traditional headcount but creating work in testing AI agents, chatbots, and RAG systems. He identified ISTQB as the most widely recognized certification, noted growing demand for AI testing frameworks such as DeepEval and Ragas, and emphasized that rapidly changing AI course material requires continual updates.

At Testμ Conf 2026, Capgemini executive Kishor Wikhe argued that AI is not eliminating software testing but transforming it into “trust engineering,” a continuous practice focused on proving how AI-enabled systems behave after deployment rather than merely confirming pre-release functionality. He said green CI pipelines validate only the checks they contain and cannot guarantee resilience against real-world risks such as sudden demand spikes, data exposure, model drift, hallucinated customer responses, bias, or unsafe autonomous-agent actions. Wikhe characterized hallucinations as governance liabilities when they affect business, legal, or public decisions, and described agent failures primarily as authority and access-control problems rather than matters of malicious intent. Drawing on aviation-style incident investigation, he noted growing regulatory emphasis on continuous monitoring, incident reporting, documented evidence, and post-deployment accountability. He distinguished testing with AI, which improves testing speed and coverage for deterministic software, from testing AI systems, which addresses non-deterministic outputs and seeks evidence of reliability, security, fairness, resilience, and appropriate human escalation over time.

At Testμ Conf 2026, TestMu AI VP of Engineering Prince Verma argued that agentic coding systems can create misleading “green checks” when the same agent writes code, generates tests, and approves results, allowing incorrect behavior or disabled tests to go undetected. He proposed an independent validation layer in which tests derive from requirements, PRDs, tickets, and acceptance criteria rather than generated source code, supported by human review, immutable execution records, change-drift tracking, and explicit reporting of both coverage and untested gaps. Verma presented Kane CLI as a harness-agnostic validation tool that uses original product context to run tests in browsers, emulators, or simulators and produces evidence packs linking requirements, builds, screenshots, logs, test outcomes, and environmental failures. A demo using a property-search PRD illustrated its generation of use cases, acceptance criteria, scenarios, gaps, and downloadable records for successful and failed runs, while emphasizing that release decisions should rely on inspectable evidence and coverage rather than passing tests alone.