September 2026 Summaries

LLM output quality concerns whether responses are correct, factual, relevant, coherent, safe, and stylistically consistent, failures that conventional infrastructure metrics such as latency, token use, and error rates often cannot detect. The text recommends evaluating these six dimensions through a combination of human rubric reviews for nuanced and high-stakes cases, automated measures such as BLEU, ROUGE, BERTScore, safety screening, and LLM-as-judge scoring for scalable monitoring, and public or internal benchmark datasets for consistent comparisons. Organizations should establish representative test sets, run controlled prompt-tuning experiments, and use balanced scorecards that account for quality, cost, and latency trade-offs. Improvement approaches range from prompt engineering and retrieval optimization to guardrails, model routing, and fine-tuning, while an effective production system continuously logs interactions, evaluates sampled outputs, conducts periodic human reviews, and feeds findings into updates to prompts, retrieval, policies, and model choices.

Corios helped an $82 billion U.S. regional bank modernize its Current Expected Credit Losses reporting before its SAS technology reached end of support, migrating credit-risk models across 10 loan portfolios to a Dataiku and Databricks environment without missing quarterly reporting deadlines. The project translated SAS models into Python and PySpark, centralized inputs and scenarios, automated execution, reporting, approvals, and stakeholder notifications, and used Databricks MLflow to manage model versions. Individual model runs that previously required multiple days were reduced to under 30 minutes, while preserved data, code, assumptions, adjustments, and model versions improved reproducibility and auditability. Before deployment, Corios supported two live-data dress rehearsals and validation checks comparing legacy and new outputs, leading to approval from the model risk team, Internal Audit, and KPMG, with production results approved on March 30, 2025. The migration also eliminated SAS licensing costs, addressed a manual publishing control weakness, and trained internal teams to operate and expand the platform for other risk and analytics uses.

Dataiku’s AI Success Frontrunner Awards recognize organizations that have moved beyond AI proof-of-concepts to achieve operational scale, governance, and measurable business results through integrated human and AI-agent workflows. Bank Hapoalim won the Agents at Work Award for automating 90% of foreign-currency transfers, while Standard Chartered and SLB shared the Return on AI Award for preventing about $90 million in potential money laundering and avoiding $7.8 million in anticipated equipment spending through predictive maintenance, respectively. Other winners include Aviva for enabling audit research in seconds through AI agents, Pfizer for automating AI bias detection and audit documentation, and One Acre Fund for streamlining document review to support smallholder farmers. ExxonMobil received the flagship AI Success Award after enabling business experts to build reusable supply-chain workflows across 40,000 railcars and more than 800 marine vessels, producing over $10 million in operational savings. Industry distinctions also recognized Morgan Stanley Wealth Management, Novo Nordisk USA, and partner Corios, with Dataiku planning to announce winners at its Succeed summit and publish additional case studies.

SLB, a global oilfield services company, developed a Dataiku-based operational intelligence ecosystem to improve the reliability of high-value equipment operating in harsh field conditions. Beginning as a 2022 pilot involving fewer than 10 assets, the initiative expanded over four years to more than 110 assets, 24 equipment models, and operations in five countries. The system automatically ingests equipment and maintenance data, interprets it according to operational stages such as pumping or mixing, and uses context-specific models to identify developing failures more accurately than conventional threshold alarms. Developed by a cross-functional team of data, engineering, and reliability specialists, the platform retains human oversight by requiring reliability engineers to validate findings before field action is taken. SLB reports that the program reduced equipment downtime by about 55%, increased fleet availability from 84% to 93%, avoided roughly $7.8 million in capital investment, and generated an estimated $2.5 million to $15 million in economic impact over three years.

Novo Nordisk’s Rare Disease Insights & Analytics team developed an AI-powered, text-to-SQL assistant in Dataiku to help analysts answer business questions from healthcare claims and prescription data in plain language. Built by a senior analyst with domain expertise and supported by leadership and a Dataiku consultant, the tool integrates data ingestion, preparation, GPT-4, Visual Agents, and daily refreshed pipelines to provide current first-pass insights in under a minute. Used by a small team across the United States and India, it automates routine querying, exploration, and reporting work, allowing analysts to focus on interpretation, hypothesis development, and stakeholder communication. The team estimates that the assistant has increased overall analytical efficiency by about 50%, reduced time spent on exploratory analysis and routine reviews by as much as 50–70%, and reclaimed capacity equivalent to roughly half of an additional analyst without increasing headcount. Human review remains central to the workflow, with analysts validating and refining AI-generated results before sharing them commercially, and the framework may be expanded to other rare-disease therapeutic areas.

Bank Hapoalim developed Project Sentinel, an AI agent that automates the evaluation of incoming foreign currency transfers, a previously labor-intensive process requiring bankers to collect customer, transaction, compliance, foreign exchange, and risk information before deciding whether to approve, reject, or escalate cases. Built on Dataiku, Sentinel compiles a 360-degree customer view every 30 minutes, applies operational and risk rules, labels transactions for regulatory reporting, and can send instructions to robotic process automation systems. It now handles more than 90% of such transfers automatically and has reduced average handling time from about six minutes to two. The system includes confidence scores, evidence, reasoning, and audit trails, allowing high-confidence decisions to proceed automatically while incomplete, inconsistent, or lower-confidence cases are routed to bankers with supporting context. Bank Hapoalim introduced autonomy gradually, beginning with human validation of recommendations before advancing toward straight-through automation, and views the governed workflow as a model for deploying AI agents in other regulated banking functions.

Semantic drift is the gradual shift of an AI system’s interpretation or output away from its intended meaning, often occurring without declines in conventional accuracy metrics or visible system failures. It commonly appears in extended conversations, where accumulated context causes agents to lose the user’s original intent, and in business reporting, where teams apply divergent definitions to shared metrics such as active customers, revenue, churn, or customer lifetime value. Detecting it requires meaning-focused methods including embedding distance, semantic similarity, entailment checks, governance conformance scoring, and LLM-based evaluation, ideally combined rather than used in isolation. Key causes include multi-turn context loss, evolving business definitions, changing retrieval corpora in RAG systems, and outdated embedding models. Prevention involves periodically refreshing original instructions, using version-controlled semantic layers and centralized glossaries, running regression tests, maintaining current retrieval sources, assigning business owners to terms, and establishing approval, auditing, alerting, and cross-functional review processes. The text emphasizes that retraining alone does not resolve semantic drift because the underlying issue is often contextual or organizational meaning rather than model accuracy, and recommends continuous monitoring and governance to identify and correct shifts before they affect decisions.

Semantic observability is presented as an evidence layer for AI governance that complements operational monitoring by capturing meaning-level signals such as intent alignment, reasoning quality, policy adherence, output accuracy, and behavioral drift. It is positioned as relevant to requirements under frameworks including the EU AI Act, whose specified high-risk-system obligations were reportedly deferred to December 2027, ISO/IEC 42001, and sector-specific rules in finance, healthcare, and insurance, where auditors require traceable explanations for AI decisions. The approach relies on five metric pillars covering cognition, traceability, performance, security, and governance, with indicators such as hallucination rates, audit-trail completeness, policy adherence, PII exposure, and escalation response time. A proposed five-step implementation process involves instrumenting interactions, applying policy tags, evaluating outputs through automated and sampled human review, creating threshold-based alerts, and generating audit-ready reports, potentially beginning with a 30-day pilot. The text also recommends using standardized telemetry such as OpenTelemetry, integrating AI alerts with existing security systems, applying retention and data-minimization policies, and maintaining detailed records of model inputs, context, versions, reasoning, policy checks, outputs, and human reviews to support continuous compliance monitoring and remediation.

Manufacturers considering agentic reasoning face a trade-off between operational value and technical risk, particularly because industrial data is fragmented across SCADA, MES, ERP, quality, maintenance, and supplier systems. SCADA provides detailed real-time signals with limited business context, while ERP offers business and cost information without enough operational immediacy, forcing engineers to manually combine incompatible records during investigations. Rather than applying agents to real-time machine control, where latency and errors could create safety concerns, the proposed approach is to use them for data synthesis and cross-system reasoning supported by semantic models that connect sensor data to products, batches, suppliers, and service histories. Potential early applications include analyzing aftermarket service and warranty records, retrieving knowledge from design and product lifecycle data, and accelerating root-cause analysis for production or asset alerts. One industrial equipment manufacturer reportedly used such a system to unify service logs, manuals, repair histories, and inventory data, reducing historically lengthy diagnostic resolution times by more than half while improving repair prioritization and lowering warranty and field-service costs. Successful adoption depends on selecting problems that require employees to gather information across multiple systems, ensuring data is accessible and results can be verified, and retaining human oversight where needed.

AI hallucinations are fluent but false or unsupported model outputs that can create legal, compliance, reputational, and decision-making risks, as illustrated by cases involving Air Canada’s inaccurate chatbot policy and fabricated legal citations submitted in Mata v. Avianca. Because even retrieval-augmented systems can hallucinate and aggregate accuracy scores can conceal high-impact failures, the text argues that enterprises need dedicated detection layers alongside human oversight and governance. It compares four methods: LLM-based groundedness evaluation, which handles nuanced claims but adds cost and latency; semantic similarity scoring, which is fast but can miss factually close errors; stochastic BERT-based consistency checks, which can identify uncertain fabrications but require repeated model runs; and token-overlap measures, which are inexpensive pre-filters but have limited coverage. Effective production systems combine these methods, capture prompts, contexts, model versions, and outputs for semantic observability, track groundedness and drift, route flagged results to reviewers, and recalibrate thresholds by domain and after model changes. The central recommendation is to pilot detection on representative outputs and build layered, governed workflows that reduce rather than claim to eliminate hallucination risk.

A GxP AI governance framework based on Good Machine Learning Practice (GMLP) is designed to capture compliance evidence throughout development rather than reconstructing it after a prototype is completed. Each AI use case follows four governed stages—ideation, project, model version, and bundle—with recorded sign-offs required before progressing, while risk tier and residual risk are tracked from the outset. Independent validators must approve every model version, providing separation between development and validation, and the same adaptable process covers both conventional machine-learning models and autonomous agents. After deployment, centralized records of sign-offs, changes, and versions support inspection readiness, while daily production metrics and defined thresholds enable continuous monitoring, drift detection, and revalidation.

Agentic AI is portrayed as a shift from supplying employees with better software to redesigning organizations around AI workers that can perform complex, continuous tasks, requiring companies to decide what humans retain, what agents handle, and how accountability works. The proposed approach emphasizes studying actual work practices, including tacit knowledge and meaningful human judgment, while creating ongoing roles for agentic research, technology evaluation, and employee enablement. Each agent should have a business owner, measurable purpose, and governance oversight, supported by AI literacy and tools that let employees build and manage agents. Organizations are encouraged to integrate data, models, agents, and human review into connected operational systems, particularly for unattended critical processes, while embedding expert knowledge into workflows. Resilience is presented as essential because AI systems depend on volatile technical, geopolitical, legal, and infrastructure environments, making portability, orchestration, and risk assessment important. Finally, the text argues for a new operating model in which executives share responsibility for organizational redesign, business functions own agent outcomes and ROI, IT enables secure adoption, and governance evolves to address rapid agent decisions, red-team risks, logging requirements, human review thresholds, and irreversible actions.

AI agents require an independent observability and governance layer because infrastructure monitoring and individual platform consoles do not provide a complete enterprise-wide view of agent ownership, costs, performance, or risk. Dataiku Agent Management, planned for general availability in October, is presented as a platform-agnostic supervision layer with three functions: discovering agents across tools such as Copilot Studio, Agentforce, Bedrock, Vertex, Databricks, Snowflake, and n8n; monitoring each agent’s usage, costs, operational health, quality, and behavioral drift; and managing risk through certifications, recurring tests, mitigation requirements, and audit-ready records. The approach is intended to help organizations identify previously unknown agents, distinguish low-use agents from widely adopted ones, detect issues such as changing user requests or declining success rates, and apply stronger controls to agents that handle sensitive data, serve customers, or execute transactions. Rather than operating or enforcing guardrails on agents, the layer records and evaluates their behavior while leaving runtime controls to the platforms where the agents are built and deployed.