September 2026 Summaries
98 posts from Factory
Filter
Month:
Year:
Post Summaries
Back to Blog
Tenant isolation testing verifies that one customer in a shared application cannot access, modify, or affect another customer’s data, including through overlooked paths such as background jobs, caches, search indexes, queues, and object storage. Effective testing traces the tenant identifier from authenticated identity through every data-access layer, ensuring client-provided tenant IDs are not treated as authority and that all queries, events, cache keys, and file paths remain tenant-scoped. Teams should use deliberately similar fixtures for at least two tenants, test both standard and privileged roles against documented policies, and alter individual boundaries such as URLs, tokens, payloads, and storage prefixes to confirm denied requests reveal no cross-tenant information. Coding agents should receive explicit, executable security requirements defining tenant authority, protected resources, roles, expected denial behavior, and targeted tests, while using synthetic rather than production data. Reviews should examine final resource lookups instead of middleware alone, collect sanitized evidence of authorization decisions, run concurrent tests where shared workers or caches are involved, and confirm that every shared subsystem has both permitted and denied access cases.
Sep 29, 2026
602 words in the original blog post.
CORS testing for agent-generated web services should validate the browser-visible access policy rather than merely confirm that a server configuration resolves an integration error, since overly broad origin rules or credential settings can expose authenticated responses to untrusted sites. Effective tests define an explicit matrix of permitted schemes, hosts, and ports, include malformed, null, sibling-domain, and unrelated origins, and distinguish public endpoints from credentialed or administrative routes. Both OPTIONS preflight and actual requests must be tested for allowed and denied origins, checking headers such as allowed origins, methods, headers, credentials, cache behavior, and response variation by Origin, while recognizing that successful status codes do not establish browser access. Credentialed requests require an exact allowed origin rather than a wildcard, and reflected origins must be validated against an allowlist. Teams should provide coding agents with specific acceptance criteria, run isolated server and browser-level tests using synthetic accounts rather than production credentials, preserve evidence for reviewers, inspect redirects, and maintain a clear separation between browser CORS controls and server-side authentication and authorization as service boundaries evolve.
Sep 29, 2026
578 words in the original blog post.
Factory’s Luke Alvoeiro describes a multi-agent coding architecture designed to reduce the supervision burden of coordinating multiple AI agents by separating planning, implementation, and validation into distinct roles with explicit handoffs and shared state. Orchestrators define goals, milestones, boundaries, and completion conditions before work begins; workers receive bounded assignments with sufficient context; and independent validators test behavior against requirements rather than merely confirming implementation choices. The approach emphasizes detailed validation contracts, reliable application setup, isolated test data, and structured handoffs that preserve completed work, failures, command results, untested conditions, and evidence for later review. To avoid conflicting edits and inconsistent design decisions, implementation tasks should generally proceed serially, while read-only research and reviews can run in parallel when they do not alter shared code. The framework also recommends evaluating models according to their specific roles and keeping reusable prompt-based procedures inspectable and testable, while maintaining stable acceptance criteria even as orchestration methods evolve.
Sep 29, 2026
1,200 words in the original blog post.
Recursive self-improvement in AI should be evaluated through measurable, bounded changes rather than assumed to imply rapid or unlimited progress. Matan Grinberg distinguishes a system’s ability to help improve itself from the speed, quality, and generality of those improvements, emphasizing that successful experiments or repairs do not necessarily demonstrate broader capability gains. For engineering teams, the recommended approach is to measure accepted outcomes, total AI spending, failed attempts, preparation time, and human review required for recurring workflows, while testing changes against representative tasks not used to create them. The discussion also advises separating model choice, provider flexibility, API compatibility, privacy controls, and safety claims, since available alternatives may behave differently or involve different data flows. Teams can improve workflows through clearer task specifications, reproducible test commands, reusable procedures, and feedback loops, but should retain stable acceptance criteria, preserve failures, and expand automation or permissions only when evidence supports doing so.
Sep 29, 2026
1,209 words in the original blog post.
OpenAI dots and Factory offer contrasting approaches to AI-assisted software development: dots combines customer-feedback monitoring, issue scoping, code changes, testing, and pull-request evidence in a GPT-6 Astra-based workflow, while Factory emphasizes planned, bounded Missions and model-agnostic execution through persistent Droid Computers or user-managed machines. Dots supports proactive read-only research across connected applications and uses separate permission boundaries for actions, whereas Factory can launch work through schedules, Slack, GitHub events, or webhooks. Factory provides custom model, provider, endpoint, and local-model options in its CLI and desktop products, while dots does not describe a bring-your-own-model capability. The comparison also highlights differing pricing structures, remote-compute options, and organizational controls, recommending that teams test both products on representative repository tasks, verify patches and test evidence, evaluate permissions and data environments, estimate real usage costs, and retain human responsibility for merge and deployment decisions.
Sep 29, 2026
1,085 words in the original blog post.
OpenAI launched dots on September 29, 2026 as persistent agents powered by GPT-6 Astra that can work across connected apps and dedicated cloud computers on tasks ranging from research and proposals to software development, while Factory positions itself as a model-agnostic alternative focused on configurable engineering workflows. Dots can conduct read-only proactive research to identify potential work, retain context across projects, and operate under user-defined application permissions, whereas Factory uses explicit automation triggers, persistent Droid Computers, structured Missions for approved engineering plans, and enterprise controls over models and tools. Factory also supports bring-your-own machines and custom model providers through its CLI and desktop products, while OpenAI’s launch materials do not describe a bring-your-own-model option for dots. Pricing differs through included dot access in eligible OpenAI plans with separate task limits, compared with Factory’s tiered subscriptions, rolling usage caps, prepaid extra usage, and organizational plans. Both platforms emphasize oversight, but dots rely on configurable action rules, activity visibility, and approval mechanisms, while Factory distinguishes between organizational restrictions, session settings, and separate permissions for preparing, reviewing, merging, or deploying code.
Sep 29, 2026
1,043 words in the original blog post.
OpenAI’s dots, launched on September 29, 2026, combines persistent projects, connected applications, and cloud computers into an agent experience powered initially by GPT-6 Astra, while Factory positions itself as a model-agnostic alternative for engineering teams needing provider choice, local models, or automatic routing based on quality, cost, latency, and cache state. The comparison highlights differences in pricing, with dots included in eligible OpenAI plans under usage allowances and Factory offering tiered subscriptions, organization plans, and prepaid extra usage. Dots runs work on OpenAI-managed cloud computers and supports proactive, read-only research across connected apps, whereas Factory offers managed persistent Droid Computers or team-controlled machines and starts automated work through schedules or configured triggers. For software projects, dots can preserve context, build and test improvements, and create pull requests, while Factory Missions emphasizes approved plans, milestones, worker coordination, explicit success criteria, and validation. Both platforms provide permission controls, but dots centers on user-managed app access, rules, and action review, while Factory adds enterprise-enforced restrictions intended to keep model and tool flexibility within organizational policies.
Sep 29, 2026
1,089 words in the original blog post.
Session cookie testing evaluates how web applications create, transmit, rotate, expire, and remove browser session state, especially after coding-agent changes to authentication middleware or host routing. Effective tests define an explicit cookie contract covering names, host and path scope, lifetime, and security attributes such as Secure and HttpOnly, while distinguishing authentication cookies from preference or anonymous identifiers. Testing should validate creation after sign-in, session-ID rotation after authentication and privilege changes, idle and absolute expiration, and complete invalidation on logout by replaying old tokens against protected endpoints. Evidence should come from real response headers and browser behavior rather than utility-level tests alone, with checks across redirects, errors, alternate hosts, and multiple browser contexts where applicable. The guidance also recommends isolated synthetic credentials, avoiding token exposure in logs or fixtures, pairing cookie checks with server-side authorization testing, and rerunning tests after framework, proxy, or identity-provider upgrades that may alter security defaults.
Sep 29, 2026
590 words in the original blog post.
Content Security Policy (CSP) testing helps ensure web pages load authorized resources while blocking unapproved scripts, frames, styles, workers, and other content, particularly when coding agents introduce changes that may conflict with existing security rules. Effective testing begins by documenting policies across representative routes and identifying the narrowest relevant directive rather than broadly relaxing default settings, while keeping development allowances separate from production policies. Tests should verify both permitted resources and deliberately blocked controls in the browser, with special attention to nonce-based inline scripts, authentication and error pages, and the combined effects of headers from applications, proxies, and content delivery networks. Teams can constrain coding-agent tasks by specifying the affected route, resource type, approved host, and evidence requirements, avoiding unsafe wildcards or unscoped global configuration changes. Final reviews should preserve response-header and browser-test evidence, protect sensitive data, and use report-only deployments with an observation period when telemetry is needed before enforcement, repeating checks after infrastructure configuration changes.
Sep 29, 2026
593 words in the original blog post.
Backup recovery testing verifies that backups can be restored into a functioning service, addressing risks that schema, encryption, storage, or dependency changes may make apparently successful backups unusable. Effective tests define required data, configurations, keys, dependencies, restoration order, and approved recovery point and time objectives, then restore production-format backups into isolated empty environments without access to production. The process should validate integrity, migrations, supported older backup versions, encryption-key failure handling, representative data and permissions, rebuilt indexes, and basic application reads and reversible writes. Coding agents should receive bounded tasks, approved runbooks, synthetic data, and explicit restrictions against production access, while logs and evidence should document timings, versions, results, failures, and manual interventions without exposing secrets or customer data. Recovery exercises should be repeated after relevant system changes, since a successful backup job demonstrates backup creation rather than the ability to restore usable operations.
Sep 29, 2026
579 words in the original blog post.
Selecting an AI coding harness should focus on whether it can complete and verify real repository tasks with acceptable cost, time, and human intervention rather than on model names or benchmark scores alone. Factory argues that its Droid harness supports comparisons across models and describes features such as environment discovery, model-specific editing tools, planning, background-process tracking, and routing between models, while its vendor-published Terminal-Bench studies report higher pass rates and lower cost per passed task than several competing harnesses under specified configurations. These results are limited to the tested versions, models, terminal-focused tasks, permissions, and methodologies, so they cannot determine the best product for every workload or isolate the effect of any individual feature. The discussion recommends evaluating candidate systems on representative tasks while recording accepted changes, failed attempts, elapsed time, validation results, configuration differences, and required approvals. It also emphasizes that verification time, service cleanup, sandboxing, least-privilege access, and separation of benchmark permissions from production rollout policies are essential parts of assessing an agent’s practical reliability.
Sep 28, 2026
1,840 words in the original blog post.
Choosing an AI coding agent should be based on realistic, well-defined engineering tasks and a consistent trial setup rather than product demonstrations, interface assumptions, or benchmark rankings alone. The comparison should evaluate each vendor’s documented workflows, deployment requirements, model options, repository integrations, execution controls, and ability to complete implementation, validation, and handoff work, while using the same repository revision, instructions, and task boundaries for all candidates. Factory’s reported benchmark results provide evidence for specific tested configurations but are not universal rankings, and teams should avoid combining incompatible metrics such as task resolution, review accuracy, and cost into a single score without a sound methodology. Evaluations should record human interventions, elapsed time versus active supervision, environment repairs, validation results, unresolved issues, and abandoned attempts to reveal the true operational effort involved. Model flexibility can support organizational requirements, but changes in models or routing should be documented and should not weaken repository policies, approval processes, or independent checks. A practical adoption decision identifies specific task classes and configurations where the agent has proven useful, preserves trial cases for future retesting, and maintains clear limits for higher-risk work such as architecture changes or sensitive migrations.
Sep 28, 2026
1,478 words in the original blog post.
Memory leak testing for agent-generated services uses repeatable, bounded workloads and explicit memory budgets to identify retained memory that functional tests may miss, often caused by listeners, timers, caches, closures, queues, or request objects that are not cleaned up. Effective tests warm up the service, run equivalent workload batches under stable conditions, release references, collect memory metrics after each batch, and evaluate both absolute growth and post-warm-up trends in heap usage and resident set size. Heap snapshots can help trace persistent objects to their retaining paths, but should be collected in isolated processes because they may block execution, require substantial memory, and contain sensitive application data. Teams should give coding agents precise acceptance criteria, including workloads, metrics, thresholds, garbage-collection permissions, and timeouts, and require them to report commands and measurements rather than general assurances. Suspected failures should be repeated to distinguish consistent retention patterns from environmental noise, while CI should retain compact measurements and securely restrict access to full diagnostic artifacts.
Sep 28, 2026
590 words in the original blog post.
Telemetry schema checks help teams detect observability regressions introduced by coding-agent or developer changes, such as renamed attributes, unbounded metric dimensions, or sensitive data in logs that may not affect application behavior but can break dashboards and alerts after deployment. Building on OpenTelemetry semantic conventions, organizations should maintain versioned local schemas defining signal names, attribute types, units, privacy status, cardinality limits, and required fields, then validate deterministic test telemetry before export. Checks should cover success and error paths, normalize expected variable values, reject unknown or unsafe fields, and use synthetic sensitive-looking data to test protections at the earliest practical instrumentation layer. Coding agents should receive explicit requirements for expected signals, schema updates, test commands, consumer justification, and cardinality bounds, while automated review can flag instrumentation changes lacking schema, fixture, or privacy evidence. CI reports should clearly identify added, removed, and changed telemetry fields with redacted examples and workload sources, and intentional breaking changes should be coordinated with dashboard, alert, and retention owners before temporary compatibility aliases are retired.
Sep 28, 2026
593 words in the original blog post.
SSRF testing for coding-agent changes should verify that features such as webhooks, URL previews, importers, image fetchers, and integration callbacks cannot be manipulated into accessing unintended network destinations. Effective testing traces untrusted input through parsing, normalization, DNS resolution, redirects, proxies, and the final socket connection, using allowlists and network policies rather than simple string checks or hostname denylists. Test suites should cover permitted destinations alongside loopback, private, link-local, multicast, reserved, IPv6, and alternate address formats, while validating every redirect hop and accounting for DNS rebinding and deployment-specific routing behavior. Coding agents should receive narrowly defined tasks, approved destination policies, relevant request entry points, existing tests, and commands to execute them, while security reviews based on STRIDE, OWASP, and supply-chain methods can supplement but not replace executable tests. Reliable evidence uses local deterministic servers, confirms blocked requests never reach prohibited listeners, assesses final connection targets, and logs redacted decisions without exposing sensitive internal details.
Sep 28, 2026
581 words in the original blog post.
API rate limit testing should verify that services enforce quotas fairly and reliably, particularly after coding-agent changes that may affect middleware ordering, identity handling, shared counters, or retry logic. Effective tests define the quota-owning identity, confirm isolation between tenants and authenticated versus anonymous users, and explicitly validate any privileged bypasses. They should exercise request thresholds around the configured limit, verify 429 responses and documented retry metadata such as both valid forms of the Retry-After header, and test clients’ bounded backoff behavior. Concurrent requests can reveal non-atomic counter updates, while failures in distributed counter backends should produce an intentional fail-open or fail-closed outcome. Coding agents benefit from executable acceptance criteria covering limits, windows, identities, concurrency, and reset conditions, and reviews should assess policies alongside tests for isolation and retry coverage. Failure evidence should capture identities, counter state, and response metadata without exposing sensitive data, while controlled contract tests should remain distinct from load tests that evaluate sustained system performance.
Sep 28, 2026
624 words in the original blog post.
File upload security testing should validate the entire lifecycle from multipart request handling and temporary storage through validation, transformation, permanent object storage, and authorized retrieval, especially after coding-agent changes that may alter MIME support, permissions, validation order, or URL exposure. Following OWASP defense-in-depth guidance, tests should cover extension and content-type checks, file signatures, generated filenames, size and processing limits, storage outside public web roots, malware scanning, cleanup of rejected uploads, and edge cases such as Unicode names, duplicate names, malformed documents, archives, and boundary-size files. Accepted uploads should use server-generated keys and be protected from path traversal, cross-tenant access, overwrites, public residual objects, and unsafe browser interpretation of active file formats. Teams should give coding agents explicit policies and existing test entry points, require tests for authorization and cleanup rather than response codes alone, and run fixtures in isolated storage while ensuring prohibited uploads fail without leaving retrievable artifacts and permitted files remain accessible only to authorized users.
Sep 28, 2026
597 words in the original blog post.
Webhook verification changes made by coding agents can introduce security weaknesses even when legitimate deliveries continue to work, particularly if request bodies are transformed before signature validation, signatures are compared unsafely, or replay protections are omitted. Secure implementations should verify the exact raw bytes received from providers, follow each provider’s distinct signing contract, use trusted secret configuration, reject malformed or unsupported inputs, and parse payloads only after authentication. Testing should cover altered bodies, invalid or missing signatures, incorrect secrets, timestamp tolerance boundaries, duplicate deliveries, framework body-parser behavior, and secret-rotation overlap and retirement policies without exposing real secrets or customer data. Pull requests should document the provider requirements, raw-body flow, comparison method, timestamp and deduplication policies, and executed tests, while combining security review with runtime API testing to provide evidence that framework behavior preserves the verification boundary.
Sep 27, 2026
594 words in the original blog post.
Queue consumer safety requires designs that account for at-least-once delivery, worker interruptions, delayed acknowledgments, and duplicate messages, as seemingly correct agent-generated code can otherwise cause repeated charges, notifications, or lost work. Consumers should define durable, retry-safe invariants, use stable message identifiers, and place idempotency controls near the side effect through transactional database operations or API idempotency keys, while detecting conflicting payloads for the same identifier. Testing should simulate duplicate delivery, crashes between writes and acknowledgments, worker restarts, transient and terminal failures, lease expiration, stale-worker commits, and partial batch failures. Code reviews should document queue semantics, acknowledgment timing, idempotency storage, retry and dead-letter policies, and failure-injection evidence, while keeping live replay and queue-purging decisions under operator control.
Sep 27, 2026
602 words in the original blog post.
Concurrency testing helps reviewers identify race conditions, deadlocks, stale reads, lost updates, and other scheduling-dependent defects that coding-agent changes may introduce despite passing ordinary tests. Effective testing begins by defining cross-worker invariants, identifying shared state and its protections, and using race detectors or sanitizers as an initial but workload-limited check. Tests should avoid timing-based assertions in favor of barriers, latches, fake clocks, and controlled execution orders that reproduce contested scenarios, while stress tests can expose starvation, contention, and resource leaks. Deadlock risks require separate bounded-completion tests and retained diagnostic output, since race detectors cannot prove deadlock freedom. Pull requests should document synchronization choices, exact verification commands, controlled schedules, stress-test settings, and practical regression tests, while avoiding weakened assertions or retries that conceal nondeterminism. When relevant environments or production conditions cannot be reproduced, teams should report those limitations clearly rather than overstate confidence in incomplete results.
Sep 27, 2026
595 words in the original blog post.
Time zone testing helps uncover date and scheduling defects that can remain hidden when development and CI environments use UTC, particularly in agent-generated code that may alter conversions or scheduling logic without accounting for daylight-saving changes, ambiguous local times, or evolving regional rules. Effective testing distinguishes fixed instants from local civil-time schedules, retains named IANA time zones rather than fixed offsets, explicitly defines behavior for nonexistent and repeated times, uses injectable clocks, and records time zone database versions when reproducibility is important. Test suites should cover representative supported regions, transition boundaries, bidirectional conversions between local values and stored instants, serialization requirements, and updated time zone data to expose dependencies on outdated rules. Code reviews should document clock sources, storage formats, supported zones, ambiguity policies, data sources, and boundary tests, while automated end-to-end checks can complement deterministic lower-level assertions; coding agents should not independently select business policies based solely on date-library defaults.
Sep 27, 2026
586 words in the original blog post.
Authorization testing verifies that users, services, and other actors can perform only policy-permitted actions, helping detect security regressions from coding-agent changes that may leave normal workflows functioning while exposing unauthorized routes or objects. Effective tests should derive from the actual policy source and cover meaningful distinctions involving roles, ownership, tenant boundaries, expired or revoked privileges, anonymous access, and resource changes, with clear fixtures that specify the permissions being tested. Each permitted scenario should have a closely related denial case that verifies both the response and the lack of side effects or data leakage across interfaces such as APIs, browser controls, GraphQL, background jobs, bulk operations, collection queries, and exports. Cached authorization decisions also require tests confirming that revocations and tenant changes take effect immediately under controlled cache conditions. Reviews should link permission changes to policy requirements, examine migrations and defaults that could expand access, and use security-focused automated tools as complements to—not replacements for—policy-level tests, while avoiding production credentials and pausing work when policies or safe representative identities are unclear.
Sep 27, 2026
578 words in the original blog post.
Kubernetes manifest validation should extend beyond YAML syntax to verify that rendered resources will be accepted by the intended cluster, including API compatibility, admission policies, defaults, and server-side field ownership. Coding agents should receive clearly scoped manifest paths, rendering commands, Kubernetes versions, namespaces, overlays, and policy sources, while limiting changes to the reported deployment issue rather than unrelated refactoring. The recommended process renders final manifests and submits them to a matching non-production control plane through server-side dry run, then uses server-side apply dry run to identify conflicts with fields managed by other controllers. Because these checks cannot establish runtime safety, they should be complemented by separate policy, image, and workload-testing gates. Pull requests should preserve reproducible evidence such as render commands, target versions, validation results, ownership conflicts, and rendered-object diffs, while humans retain authority over rollout timing, production credentials, and deployment scope.
Sep 26, 2026
574 words in the original blog post.
Retry safety in agent-generated application code requires more than adding retry loops, because overlapping retries across clients, proxies, queues, and job runners can overload dependencies, duplicate side effects, and exceed caller deadlines. Reliable changes begin by mapping the full request path, identifying the single layer responsible for retries, documenting timeouts and attempt limits, and classifying operations according to their side effects and idempotency requirements. Failure testing should use deterministic fakes or fault injection to simulate conditions such as connection failures, committed-but-unacknowledged requests, throttling, partial responses, and permanent errors while measuring attempts, elapsed time, final outcomes, and durable side effects. Effective retry policies use capped, jittered backoff, honor cancellation and total deadlines, and avoid retrying errors unlikely to change, such as authentication failures or malformed input. Changes should be reviewable through clear pull-request documentation, failure matrices, observability that avoids sensitive data, required fault-injection tests, and rollout metrics that assess not only visible error rates but also dependency traffic, duplicate work, and latency.
Sep 26, 2026
584 words in the original blog post.
Database query plan review helps validate whether coding-agent SQL changes improve performance under realistic conditions, since unit tests may miss costly shifts in join order, row estimates, index usage, or sorting. Effective reviews establish a clear baseline using the exact query, parameter patterns, schema and PostgreSQL versions, representative statistics, and a specific performance objective, while documenting whether evidence comes from production telemetry, staging, or synthetic data. Reviewers should compare estimated plans with measured execution carefully, using EXPLAIN for non-executing analysis and EXPLAIN ANALYZE only in isolated environments, with repeated controlled runs to account for cache effects and variability. Estimation errors may reveal outdated or unrepresentative statistics rather than a query defect, so agents should report uncertainty instead of overstating fixes. Proposed indexes and rewrites should also be assessed for write, storage, maintenance, locking, ordering, migration, and transaction implications, while pull requests remain narrowly focused on a measured regression. Controlled tools such as Factory’s Droid Exec and automated code review can support this process, provided credentials are protected and plan comparisons, integration tests, dataset characteristics, commands, results, and remaining variance are clearly recorded for database reviewers.
Sep 26, 2026
580 words in the original blog post.
Protobuf compatibility requires evaluating schema history, mixed-version deployments, and rollback behavior rather than relying solely on whether a current .proto file compiles. Coding agents should receive clear scope information, including previous schemas, affected services, generated language targets, supported client versions, external consumers, and regeneration commands, while classifying changes by their compatibility risk. The guidance emphasizes never reusing deleted field numbers, reserving removed field names and numbers, treating renames cautiously because they can affect JSON mappings and generated APIs, and avoiding incompatible changes such as switching between singular and repeated fields. Teams should compile old and new schemas with supported toolchains and test serialization in both directions across representative cases, including unknown fields, defaults, enums, nested messages, and JSON boundaries. Generated code should be verified against clean, deterministic regeneration, schema changes should be reviewed separately from generated-file noise, and toolchain upgrades should remain distinct when possible. Rollout planning should document deployment order, the duration of mixed-version support, rollback implications, and CI compatibility testing, while pull requests should provide schema references, test results, target languages, regeneration details, and rollout evidence for service owners.
Sep 26, 2026
579 words in the original blog post.
Effective log redaction requires tracing sensitive values from their input source through every potential logging destination, including application logs, exceptions, audit events, traces, metrics, browser telemetry, CI artifacts, and third-party SDKs that may capture headers or payloads automatically. Organizations should use OWASP guidance alongside their own data classification and retention policies to define whether each field is removed, masked, hashed, encrypted, or retained as a safe opaque identifier, while supplying coding agents only sanitized test fixtures and constrained repository access. Verification should test final emitted records across normal and failure paths, scan logs and artifacts with canary values for direct and encoded leaks, and confirm that useful diagnostic details such as event names, error classes, and correlation fields remain available. Repeatable pull-request reviews should document affected fields, checked sinks, transformations, and tests for nested values, while addressing retention or deletion of historical records separately. Coding agents and automated security review tools can help implement and validate bounded remediation tasks, but security and privacy owners should remain responsible for decisions about regulated-data retention and policy conflicts.
Sep 26, 2026
627 words in the original blog post.
Coding agents make test data privacy a software delivery concern because they can access fixtures, database snapshots, logs, and CI output that may contain sensitive information. The article recommends applying data minimization by classifying sources before agent access, favoring synthetic fixtures that preserve only the structural conditions needed to reproduce bugs, and removing unnecessary identifiers, credentials, contact details, and free-form text. Agent permissions should be restricted to required directories, databases, and commands, with production exports and credentials blocked, short-lived test credentials used, logs redacted, and human approval required for exceptions involving production-derived data. After changes, teams should inspect diffs, artifacts, reports, and agent outputs for sensitive material, verify tests work from a clean checkout using declared fixtures, and document fixture generators and seeds to support reproducibility without exposing protected data.
Sep 25, 2026
630 words in the original blog post.
API client generation converts an OpenAPI service contract into reusable client code, models, and documentation, and coding agents can help coordinate updates when the contract remains authoritative and generated files are not manually edited. Reliable workflows define the target language, pinned generator version, configuration, approved command, generated directory, and publication approach, while keeping API contract changes separate from generator upgrades to make reviews clearer. Validation should include clean, repeatable generation with no subsequent diff, review of public API behavior and dependencies, compilation, and contract-focused tests covering successes, errors, optional fields, pagination, authentication, and serialization. Generated changes should be reviewed as release artifacts, with agents summarizing contract, generator, dependency, and validation differences, CI detecting stale output, and independent scrutiny of templates and configuration. Package versioning and publication should reflect compatibility impact, and clients for multiple languages should be updated and tested independently despite sharing a contract.
Sep 25, 2026
573 words in the original blog post.
Container image hardening is most effective when coding agents are assigned narrowly defined, evidence-based tasks rather than broad security cleanup requests. The process should identify the affected image digest, Dockerfile, build command, architecture, vulnerable package, and introducing layer, then constrain changes to relevant actions such as updating a base image, removing unused software, separating build and runtime stages, or running as a non-root user. Teams should preserve rollback information, rebuild images without local cache reliance, scan the resulting image digest, and test production-like runtime behavior including entrypoints, permissions, networking, certificates, and native libraries. Pull requests should document findings, image digests, changed layers, commands, test results, and justifications while avoiding sensitive information, and unresolved risks should have documented controls, ownership, and expiration dates. Small, reproducible changes supported by separate scanning and runtime validation checks are presented as easier to review and safer to deploy.
Sep 25, 2026
592 words in the original blog post.
Build provenance provides authenticated metadata linking a released software artifact to its source materials, build instructions, and build platform, an especially important safeguard when coding agents can alter code, dependencies, and build configurations. The recommended approach is to have agents submit changes through pull requests while trusted CI builds the exact approved commit, generates attestations, and records immutable identifiers such as commit SHAs and container digests without exposing secrets. Before deployment or promotion, teams should verify that an attestation matches the artifact digest, expected repository and workflow, and approved commit, treating missing or mismatched provenance as a failure. Reproducible builds complement provenance by testing whether declared inputs can recreate an output, while provenance documents the specific build that occurred. Because agent-edited workflows can affect recorded inputs and signing identities, workflow changes should receive security-focused owner review, use pinned trusted actions, and be tested against valid and invalid artifacts, beginning with one enforced release-verification point before expanding coverage.
Sep 25, 2026
559 words in the original blog post.
Code ownership provides a structured human review path for coding agent pull requests by routing changes in sensitive repository areas to reviewers familiar with their operational and security implications. Effective CODEOWNERS rules should be based on risk boundaries such as authentication, payments, deployment configurations, schemas, and public APIs, with pattern behavior and team access verified before enforcement. CODEOWNERS should remain distinct from AGENTS.md files, which give agents scoped execution instructions rather than approval authority. Branch protection should require owner approval where needed and protect the ownership file itself, while coding tasks should encourage agents to identify matched owners, flag unowned files, and separate unrelated subsystem changes. Automated code review can supplement this process by finding likely defects, but designated owners remain responsible for judging whether changes fit system constraints. Maintaining accurate ownership requires regular updates as teams and services change, monitoring for unassigned or unavailable reviewers, and using appropriately sized teams with escalation paths to balance coverage and review workload.
Sep 25, 2026
585 words in the original blog post.
Factory distinguishes between EU regional AI deployment and customer-operated infrastructure, arguing that organizations should evaluate them as separate requirements. Its EU deployment keeps session content, inference, backend services, and databases in Europe, while organization records, user profiles, and billing data remain in the global deployment, requiring teams to assess data categories individually. Organizations must also review external integrations, provider eligibility, retention rules, and data paths through testing rather than relying solely on regional labels. Factory Private, by contrast, places the control plane in a customer VPC or on-premises environment, including possible air-gapped configurations, but shifts responsibility for operations, upgrades, recovery, and model availability to the customer. The recommended decision process documents data classes, approved regions, infrastructure operators, model services, exceptions, contracts, and test evidence, with reassessment needed whenever connectors, providers, or telemetry change.
Sep 24, 2026
568 words in the original blog post.
Coding agent productivity should be evaluated through delivery outcomes rather than activity measures such as token consumption, sessions, lines changed, or pull requests created, which do not establish whether software is delivered faster or remains reliable. Useful evaluation connects agent-assisted work to speed, quality, cost, and organizational goals using delivery metrics such as lead time, deployment frequency, recovery time, change failure rates, review waits, rework, defects, rollbacks, and incidents. Comparisons should use similar task types, credible baselines, pilot groups where possible, and annotations for factors such as staffing, model changes, policies, and CI capacity. Cost analysis should focus on accepted changes or completed tasks while accounting for model usage, compute, reviewer effort, and failed-validation reruns. Organizations are encouraged to review trends with engineers and code owners, use a small balanced set of delivery, quality, cost, and adoption measures, avoid a single productivity score, and retain only metrics that support concrete decisions rather than rewarding visible activity or judging individual engineers.
Sep 24, 2026
575 words in the original blog post.
Flaky test triage should treat intermittent failures as a distinct engineering problem rather than repeatedly rerunning jobs until they pass, since reruns can erase valuable evidence. Coding agents can support investigations by preserving the original failing run’s logs, environment details, seeds, artifacts, and timestamps; conducting bounded, hypothesis-driven reproduction efforts; and producing reviewable changes. Effective diagnosis varies one condition at a time, such as test order, worker count, timezone, database state, or network latency, while examining common causes including shared mutable state, inadequate cleanup, fixed ports, timing assumptions, unordered data, and eventually consistent systems. Recommended fixes address the underlying causal issue through observable waits, isolated records, frozen time, deterministic cleanup, or removal of order dependence rather than merely increasing timeouts. Pull requests should document the symptom, root cause, verification commands, and repeated-run results where practical, while avoiding unrelated refactoring; temporary quarantines should have clear ownership and exit conditions. Scheduled automated checks and browser-based QA can provide additional evidence and detect recurring failures, but automation should preserve failures for investigation rather than retrying until successful.
Sep 24, 2026
614 words in the original blog post.
Private AI package maintenance benefits from durable context that preserves build conventions, upstream exceptions, reviewer decisions, and prior failed approaches across related repositories, reducing repeated investigation during long-running work. Factory cites a Chainguard case in which one Droid session reportedly operated for two weeks across six repositories and built 80 packages, while noting that this customer experience is not a universal productivity benchmark. Its context-compression research emphasizes retaining intent, file changes, decisions, and next steps, but file tracking remains a limitation that requires engineers to verify working trees and test outputs. The article recommends pilots involving related packages, documented revisions and validation commands, and measurements of repeated investigation and reviewer corrections rather than package counts alone. For disconnected deployments, organizations must also provision local source archives, images, package indexes, signing tools, and fixtures, ensure inference and telemetry remain internal, prevent unintended public downloads, and restrict agent permissions so patch preparation does not automatically enable package publication.
Sep 24, 2026
555 words in the original blog post.
Test impact analysis helps coding agents identify the tests most likely to detect regressions from a code change, balancing rapid feedback with the need to avoid false confidence from overly narrow test selection. It should begin by identifying the base revision, changed code, generated files, configuration, and non-code inputs, then mapping them to components, dependents, and relevant tests using build graphs, package metadata, imports, ownership rules, and historical test results. Because these sources can miss runtime configuration, external systems, and dynamic behavior, selection must also account for risk factors such as authentication, public APIs, database schemas, concurrency, serialization, build tooling, and shared fixtures. Teams should define explicit acceptance criteria, escalation rules for broader package or integration testing, and a full CI suite that remains in place, while requiring agents to justify both selected and skipped test groups. Deterministic policies stored in scripts or CI configuration improve reproducibility, and focused tests should be followed by required broader validation, including behavioral checks for user-facing flows. Pull requests should document the revision, affected components, test-selection method, executed checks, results, and deferred CI work, with broader testing preferred whenever incomplete dependency information prevents a defensible impact assessment.
Sep 24, 2026
610 words in the original blog post.
Enterprise AI adoption can be hindered when the code repository lacks the tickets, design records, and discussions that explain why changes are needed, requiring engineers to reconstruct context while complying with access controls. A Factory case study involving Nav, which connected on-premises GitLab, Confluence, Jira, and Slack, reports a 60% reduction in context-switching time and feature cycles completed twice as quickly, though these customer-reported results are not a controlled benchmark. The article recommends piloting AI on real changes spanning multiple information sources, measuring time spent gathering context and assessing whether generated patches clearly document requirements, preserved behavior, and validation evidence. It also emphasizes that integrations using approved credentials must preserve least-privilege access, while deployment decisions for cloud-managed, hybrid, or air-gapped environments should separately determine where control planes, inference, runtimes, and retained evidence reside. Organizations should evaluate accepted changes, reviewer corrections, context-gathering time, and access interruptions together, retaining existing source-permission and merge-approval controls rather than relying on an agent’s instructions.
Sep 24, 2026
547 words in the original blog post.
Effective audit trails for AI coding agents should link a requested change to the requester and execution identity, repository and session context, tools and commands used, resulting commits, validation evidence, human approvals, and deployment records rather than relying on conversation transcripts alone. Using OpenTelemetry-style traces and stable identifiers can connect agent activity with source control and CI systems, while distinguishing service accounts from the people who initiated work improves accountability. High-value evidence includes blocked commands, file modifications, pull requests, exact commit SHAs, test results, review decisions, and deployment details, with branch protections helping verify that the approved revision was actually merged. Organizations should establish retention, access, redaction, and telemetry-granularity policies to avoid collecting unnecessary sensitive content, and should regularly test whether a complete change history can be reconstructed for both successful and failed agent runs.
Sep 24, 2026
605 words in the original blog post.
Least-privilege access for coding agents should be defined by the specific task rather than the permissions of the developer who initiated it, limiting exposure to only the necessary repository paths, tools, network destinations, credentials, and external systems. The approach separates read and write permissions, uses dedicated machine identities and short-lived credentials, and keeps production deployment rights outside routine coding workflows. While prompts can guide agent behavior, effective safeguards require enforceable controls such as operating-system-level sandboxes, restricted working directories, network allowlists, command limits, and organization-wide policy ceilings. Changes prepared by agents should remain subject to branch protection, automated checks, human review, and explicit authorization for remote actions, with service accounts providing stable audit identities. Logging both successful and denied actions can reveal unclear task scopes or missing approved dependencies, while temporary permissions and environments should expire when work is complete to preserve a small, reviewable security boundary.
Sep 24, 2026
586 words in the original blog post.
Self-improving software architecture should turn recurring agent failures into controlled, versioned changes rather than relying on temporary session memory, with corrections placed in the appropriate artifact such as application code, tests, tool definitions, repository instructions, or reusable workflow skills. Factory’s software-factory model emphasizes connecting engineering signals to implementation, validation, deployment, and monitoring while preserving model independence, sovereignty, and continual learning. Effective improvement requires retaining both failure evidence and separate preventive guidance, assigning ownership and approval paths, maintaining rollback capability, and limiting instructions to their intended scope. The approach also separates observed evidence from authority to act, using configured security controls for permissions, credentials, network access, and enforcement boundaries rather than allowing agents to expand their own powers. Changes should be evaluated against the original incident and broader held-out scenarios, with blocked tests recorded accurately rather than treated as successes. The article distinguishes documented internal research practices from product guarantees and recommends beginning with small, reviewable improvements that specify an evidence source, changeable artifact, acceptance test, approver, and reversible prior version.
Sep 24, 2026
1,355 words in the original blog post.
Evaluating coding agents should focus on verifiable changes rather than completed sessions, persuasive explanations, or large diffs, using real backlog tasks that reflect a team’s languages, dependencies, review standards, and potential edge cases. Teams should define independent acceptance checks before runs, provide reproducible environments with consistent commits, dependencies, permissions, and settings, and avoid leaking solutions or giving candidates unequal assistance. Evaluations should distinguish agent-facing development tests from reviewer-controlled validation, assess scope and security implications alongside functional correctness, and document setup conditions that may affect outcomes. Each run should record configurations, changed files, checks, interventions, elapsed and human time, failures, retries, and abandoned work so reported costs and reliability are not distorted by selective successes. Benchmark findings such as Factory’s Legacy-Bench and review studies can inform methodology, but their rankings and results should not be generalized to other repositories, models, or conditions. Pilot results should guide limited adoption in task categories supported by evidence, while ongoing rollout metrics can supplement—but not replace—independent correctness review and repeatable evaluations after meaningful product, model, or workflow changes.
Sep 24, 2026
1,190 words in the original blog post.
Evaluating self-improving software after release requires treating each proposed change as a controlled release decision, with a clear baseline, stable success criteria, comparable tasks, and evidence that required checks were completed. Measures such as speed, issue volume, or apparent cost can be misleading if task difficulty changes, tests are skipped, or engineer review and repair work is omitted, so evaluations should separate output quality from agent explanations and document unacceptable regressions in advance. Teams should record workflow versions, models, tools, repository and environment conditions, use representative and held-out tasks, distinguish excluded or blocked cases from implementation failures, and track human effort across preparation, review, repair, and follow-up. Post-deployment monitoring should assess whether improvements generalize to new work while accounting for changes in task mix, release cadence, staffing, and connected-system coverage, rather than attributing every trend to automation. A durable release record should identify the change, evidence, remaining uncertainty, human corrections, blocked checks, ownership, and rollback signals, enabling teams to retain, revise, or cautiously roll out candidates based on defensible results.
Sep 24, 2026
1,295 words in the original blog post.
Factory Droid and Cursor should be compared as coding-agent platforms with different editor, automation, deployment, and data-boundary options rather than as a simple agent-versus-editor choice. Droid integrates with existing editors including VS Code-family tools, JetBrains, and Zed, while Cursor offers an editor, CLI, headless execution, cloud agents, and self-hosted team pools, though its agent loop remains cloud-based in that model. Factory supports cloud-managed, hybrid, and fully air-gapped deployments, with its isolated option running Droid, model endpoints, and telemetry within a private network, making it potentially better suited to environments requiring no outbound runtime connectivity. Cursor provides enterprise controls such as organization-wide Privacy Mode, zero-data-retention agreements with providers, SSO, and SCIM, while Factory emphasizes configurable deployment boundaries, model gateways, centralized model allowlists, autonomy limits, and policy enforcement before user sign-in. The comparison recommends testing rule handling, permissions, authentication, CI behavior, context access, command logs, and enforcement controls using identical real-world tasks in separate clean environments. Teams should assess reviewed patches, test results, human correction and review time, setup failures, and actual usage and execution costs rather than relying on feature lists, vendor claims, or isolated demonstrations.
Sep 24, 2026
1,185 words in the original blog post.
A self-improving software workflow should begin with a narrowly defined, recurring maintenance problem, a reproducible failure, clear ownership, and an established human review process, using automation initially to gather evidence and propose corrections rather than expanding its authority. The illustrative example involves an agent incorrectly editing generated files instead of their source inputs, with success defined by changing the proper source, regenerating outputs, and passing consistency checks independently. Teams should establish explicit scope, permissions, stop conditions, and evidence requirements, while retaining sensitive information securely and avoiding automated decisions when ownership, failures, or duplicate reports are uncertain. Durable improvements should be small, evidence-based, versioned instructions or reusable procedures such as documented generator commands and validation steps, rather than vague reminders or records of every prior interaction. Validation should separately test the immediate patch and the reusable lesson through clean reproductions, neighboring cases, repository-specific checks, and continued human approval. After acceptance, teams should monitor later tasks, including blocked and failed runs, evaluate changes on tasks not used to develop the correction, watch for new costs or stale guidance, and expand to additional repositories or triggers only when the workflow demonstrably reduces repeated human redirection.
Sep 24, 2026
1,292 words in the original blog post.
AGENTS.md files provide coding agents with concise, verified repository instructions that emphasize executable commands, task boundaries, validation requirements, and safeguards against costly mistakes. Effective guidance begins at the repository root with tested install, test, type-check, lint, build, and application-start commands, while distinguishing quick focused checks from broader CI requirements and documenting necessary local dependencies without exposing credentials. The file should define clear, reviewable constraints around shared interfaces, generated code, deployment settings, production data, and scope, while linking to longer maintained documentation rather than duplicating architecture details. In monorepos, nested AGENTS.md files should be added only where packages have genuinely different tools or workflows, and teams should confirm how each coding tool discovers and prioritizes instructions. New guidance should be tested through a small, low-risk change, with reviewers checking that the agent identified applicable instructions, stayed within scope, ran the correct commands, and accurately reported results. AGENTS.md complements rather than replaces technical controls such as permissions, branch protection, security policies, and enforceable linters or tests, and it should be updated alongside changes to scripts, architecture, or generation processes.
Sep 24, 2026
1,206 words in the original blog post.
Enterprise software modernization should focus on preserving customer-critical behavior rather than merely generating large code patches, with clearly defined targets, reviewable implementation records, representative inputs, integration boundaries, exceptions, and rollback plans. Comarch’s case study reports substantial reductions in estimated migration timelines using Factory, including a 45-day XSLT modernization compared with a 10,000-person-day estimate, a database migration completed in six hours against a 45-day estimate, and an airline MVP delivered in one week rather than an estimated 18 months, though these examples are separate projects rather than controlled performance benchmarks. Factory’s Missions architecture divides work among orchestration, bounded implementation, and independent validation, requiring validation contracts and reproducible checks before changes are approved. Human engineers retain decision-making authority and review work produced during background “night shift” execution, while production data access and release permissions should remain separate from task execution. The account also highlights You.com’s use of isolated Docker testing to identify an MCP timeout as a client-side issue, illustrating the value of reproducible investigations. Teams are encouraged to begin with a limited modernization slice, measure accepted results and reviewer effort, and expand the workflow only when validation supports doing so.
Sep 24, 2026
554 words in the original blog post.
Effective coding-agent task specifications define a bounded, verifiable outcome rather than relying on vague requests, particularly for delegated or autonomous work. They should identify the relevant repository and area, describe the desired user or system behavior, provide reproduction steps or concrete inputs and outputs where applicable, establish scope boundaries, and include independently testable acceptance criteria with clear triggers and expected results. Specifications should also cover important negative cases, such as rejected unauthorized access or preserved existing behavior during migrations, while avoiding unnecessary implementation prescriptions. Stable project conventions, build commands, security rules, and validation expectations can be maintained in an AGENTS.md file, while task-specific context should link to relevant tests, incidents, designs, or requirements without overwhelming the agent. Before assigning work, reviewers should confirm the outcome, limits, evidence, required checks, remote-action approval authority, and pull-request review process so that implementation can proceed without guesswork and completion can be evaluated from observable evidence.
Sep 24, 2026
576 words in the original blog post.
Factory Private describes an on-premises, potentially air-gapped AI deployment model for research environments containing sensitive unreleased model weights, evaluation data, and proprietary training code. NVIDIA’s Nemotron research team used the platform during post-training of Nemotron 3 Ultra to support evaluation and optimization, with detailed task trajectories and verified outcomes informing later training cycles. The material emphasizes that protecting research requires controls over runtime systems, model inference endpoints, telemetry, tool outputs, and external gateway providers, rather than merely hosting execution on customer infrastructure. It also recommends reproducible evaluations that preserve model versions, dependencies, tools, test fixtures, and independent validation results, while treating telemetry content as optional sensitive data subject to explicit configuration, access, and retention policies. For initial adoption, organizations are advised to use bounded tasks with existing acceptance tests, compare results against current workflows, and document both the agent’s permitted access and actual activity without assuming universal performance or efficiency gains.
Sep 24, 2026
585 words in the original blog post.
Factory and Warp are compared as broader engineering workflow platforms rather than merely agent or terminal tools, with the key evaluation spanning task intake, context, execution, validation, review, delivery, observability, and repeated operation. Warp’s Oz Platform offers cloud environments, orchestration, observability, and CLI, API, and SDK access while supporting third-party agent harnesses such as Claude Code and Codex, whereas Factory centers on agent-native automation through Droids, configurable integrations and instructions, model selection, shared context, and continuous feedback. The comparison recommends testing both products on an existing repository and recurring task with equivalent requirements, documenting setup time, failure recovery, output quality, maintenance needs, permissions, secrets handling, and deployment constraints. Factory presents a seven-stage reference implementation covering intent intake, context resolution, planning, execution, review and policy, delivery, and observability, but notes that this illustrates Factory’s own architecture rather than a neutral benchmark or Warp equivalent. Organizations are advised to confirm security, private-network, contractual, deployment, cost, and quality considerations directly with vendors and make decisions based on the delivery loop they can effectively operate and govern.
Sep 24, 2026
556 words in the original blog post.
Factory’s agent readiness approach aims to reduce avoidable AI coding-agent token use by identifying missing repository documentation, setup instructions, validation commands, and engineering foundations that otherwise force agents to repeatedly reconstruct context. Its readiness reports and AGENTS.md support are intended to preserve accurate, concise project knowledge such as tested build commands, environment requirements, repository boundaries, and exceptions, while emphasizing that instructions must be verified in their intended environment. Organizations are encouraged to measure representative tasks before and after readiness improvements using comparable models, environments, and acceptance criteria, tracking failed commands, repeated repository reads, elapsed time, reviewer corrections, and model usage. The article cautions against claiming token savings solely from readiness scores or conflating them with other changes, noting that a cited 60% reduction in context-switching time at Nav concerns human workflow rather than measured token savings. It recommends prioritizing recurring issues affecting many tasks and confirming improvements through subsequent task outcomes rather than assuming success after documentation is added.
Sep 24, 2026
523 words in the original blog post.
SBOM automation creates a machine-readable inventory of the components included in a resolved software release, supporting vulnerability management, license analysis, and software tracking while acknowledging that package manifests alone cannot establish complete coverage. Effective processes define the specific artifact, build stage, output format such as SPDX or CycloneDX, generator configuration, and treatment of unknown ownership or license data, while providing coding agents with reproducible build, validation, and acceptance instructions. Inventories should be generated from resolved dependency graphs, container layers, or final artifacts rather than manifests, then compared with prior releases to identify unexpected changes and validated against format schemas. The guidance distinguishes SBOM generation from vulnerability scanning, since inventories describe components and relationships whereas scanners evaluate them against advisory data. Review evidence should include configuration, validation results, inventory differences, and the associated release artifact, with automated checks used to identify drift, while incomplete builds, unknown resolution environments, or unsupported package ecosystems should be reported as clear limitations.
Sep 24, 2026
589 words in the original blog post.
Factory Private architecture places the AI development control plane within customer-controlled VPC or on-premises infrastructure, giving organizations greater control over execution, data boundaries, and service dependencies while also requiring clear operational ownership. Effective deployment reviews should map every component, operator, connection, and data flow—including model gateways, source control, package repositories, telemetry collectors, and execution environments—because private placement of one component does not guarantee that downstream services remain private. The guidance emphasizes separately evaluating model routing, access permissions, network restrictions, sandboxing, and telemetry retention, with validation through representative tasks and negative security tests rather than assumptions. Organizations are advised to begin with a bounded pilot, limit access and data collection to what is necessary, establish upgrade, support, capacity, and recovery processes, and measure accepted work, validation failures, reviewer corrections, and operational effort before expanding deployment.
Sep 24, 2026
1,400 words in the original blog post.
Fast inference can make AI coding agents more practical for tasks such as repository discovery, codebase questions, configuration changes, and CI investigation by reducing the time to a useful, validated result. Groq’s case study with Factory’s Droid reports three-times-faster medium-complexity feature development and five-times-faster quick-turn tasks in a specific comparison using Kimi K2 on Groq against GPT-5 on Codex and Claude Code, though these results are not presented as universal model advantages. The discussion emphasizes that overall workflow speed also depends on repository access, tool execution, testing, and human review, so organizations should pilot small, verifiable workloads and measure both first useful output and accepted completion, including errors and repeated tool calls. Factory’s support for custom models and endpoints can enable testing approved providers without changing surrounding workflows, but enterprises must validate quality, data handling, deployment routes, failover behavior, and policy compliance. Parallel agent use may reduce elapsed time for independent work but can increase cost and review burden, making organization-specific results and operating constraints essential to adoption decisions.
Sep 24, 2026
561 words in the original blog post.
Context compaction for AI coding agents should be evaluated by whether an agent can accurately continue work after a long session, rather than by summary length or token reduction alone. Factory’s December 2025 study of 36,611 software-engineering messages compared retention across recall, artifact tracking, continuation, and decisions, reporting overall scores of 3.70 for Factory, 3.44 for Anthropic, and 3.35 for OpenAI, while noting that artifact tracking was the weakest area for all methods. The findings suggest that structured, persistent summaries can preserve technical details, but summaries should not replace repository evidence such as the working tree, patches, and test results. A Chainguard case study of a two-week, six-repository coding session illustrates the value of continuity but does not independently establish compaction’s cost or performance effects. Effective evaluation should record constraints, modified files, validation results, rejected approaches, and next steps before compression, then test whether the agent can resume correctly, incorporate revised decisions, and avoid redundant rereading or human intervention.
Sep 24, 2026
571 words in the original blog post.
Long-running coding agents can lose direction as scope expands, context fades, tests become stale, or failed approaches are repeatedly retried, so multi-day autonomous work requires explicit control points. Factory recommends defining a validation contract with bounded outcomes and observable behavioral checks, then dividing work into independently reviewable milestones with clear dependencies, required evidence, and isolated high-risk changes. Separate validators should assess each milestone against the original requirements rather than relying on the implementer’s reasoning. Durable shared state should record plans, decisions, changed files, failed attempts, test results, blockers, and repository conventions so new workers can continue without reconstructing prior conversations. Teams should also establish human intervention and stop conditions for repeated failures, unavailable credentials, scope changes, or unsafe migrations, documenting any resulting adjustments. Final review should compare delivered behavior with the original contract, verify named checks and code changes, preserve evidence, and identify deferred work.
Sep 24, 2026
603 words in the original blog post.
Model routing for AI coding agents should account for full session economics rather than token prices alone, particularly the value of warm cached context, prior tool results, and accumulated decisions that may be costly to reprocess after switching models. Factory reports separate production findings of 58% lower aggregate cost and 49-second median routed-session latency versus 81 seconds for frontier-pinned sessions, while its current documentation cites 43% savings against top-tier pricing; both figures use distinct baselines and are not guarantees. Routing decisions can use task signals such as test outcomes, recurring errors, and context needs to determine when inexpensive models are appropriate and when stronger models should remain active, as illustrated by a 67-turn Prisma migration that could not safely step down. Focused secondary workers may preserve a parent session’s warm context but add their own costs and integration overhead. Enterprises evaluating routing should measure billed costs, cache activity, chosen models, failed attempts, reruns, accepted changes, elapsed time, and cost per successful task under consistent acceptance criteria, while also confirming that model availability, deployment requirements, and enterprise controls fit their environment.
Sep 24, 2026
586 words in the original blog post.
AI coding agents working in monorepos need machine-readable repository boundaries because file proximity alone does not reveal dependencies, ownership, validation requirements, or deployment implications. Dependency and task graphs, such as Nx’s affected-command model, can connect changed files to impacted projects and required checks, while layered instruction files provide concise root-level rules and package-specific guidance for tools, tests, generators, and ownership constraints. Tasks should be scoped to an owning package and a defined dependency radius, with agents identifying consumers of shared code and preparing plans for cross-package changes that address interfaces, generated artifacts, and test targets. Running agents from narrow working directories and using isolated worktrees can reduce conflicts between concurrent changes. Affected-only validation should be supplemented by periodic full builds to identify missing graph edges, and failures should prompt metadata or dependency-graph fixes rather than broadly expanding routine checks. Stronger review and testing are recommended for shared libraries, generated clients, lockfile changes, and deployment-related packages, with production actions kept outside coding tasks unless explicitly authorized.
Sep 24, 2026
556 words in the original blog post.
Coding-agent observability changes require end-to-end review because seemingly minor instrumentation edits can disrupt dashboards, raise storage costs, expose sensitive data, or obscure failures. Effective work begins with a defined operational question, an appropriate OpenTelemetry signal such as traces, metrics, or logs, and clear requirements covering naming conventions, sampling, prohibited fields, expected queries, and verification steps. Reviewers should trace telemetry from application instrumentation through context propagation, SDKs, collectors, export, storage, and the dashboard or alert that consumes it, rather than relying only on unit tests. Stable semantic names, correct metric types and units, bounded attribute cardinality, and explicit redaction policies help preserve reliable and safe telemetry, while high-cardinality identifiers should generally remain in traces or structured logs when permitted. Changes should be tested with controlled successful and failing requests and validated in their exported form for trace relationships, metric behavior, log severity, error status, resource attributes, duplicate instrumentation, and privacy compliance; work should pause when data destinations, retention, or permitted fields are unknown.
Sep 24, 2026
583 words in the original blog post.
Agent-generated pull requests can increase development output faster than review capacity, making it important to preserve rigorous review standards while structuring changes so routine work is easy to verify and higher-risk work reaches appropriate experts. Each pull request should contain one coherent change, clearly explain its purpose and affected files, disclose completed and pending checks, include relevant test or reproduction evidence, and separate mechanical from behavioral changes where possible. Automated checks such as linters, type checks, tests, security scans, and repository-specific policy reviews should identify actionable defects before human review, but cannot replace accountable ownership and approval. Teams can use CODEOWNERS, branch protection, and stricter requirements for sensitive areas such as authentication, billing, deployments, and migrations, while reviewers assess behavior, permissions, compatibility, rollback readiness, and operational evidence. Monitoring metrics including review delays, requested changes, escaped defects, and reverts can reveal recurring problems and help teams improve tests and review guidance without incentivizing pull request volume over reliable merges.
Sep 24, 2026
574 words in the original blog post.
Factory presents customer examples of enterprise AI-assisted software work, including Comarch’s XSLT modernization and database migration, Chainguard’s multi-repository package builds, Nav’s financial-product integrations, Empower’s incident and pull-request workflows, You.com’s infrastructure investigations, and Groq’s launch, telemetry, configuration, and CI tasks. The company emphasizes that reported outcomes, such as faster development cycles or reduced response times, reflect distinct project scopes and measurement methods rather than a universal benchmark. It argues that effective pilots should begin with a team’s specific bottleneck, such as cross-repository context, review delays, or recurring research work, while retaining existing acceptance and review standards. Factory recommends defining clear project boundaries and tracking criteria including reviewer effort, elapsed time, operating cost, and delivery outcomes before expanding adoption, noting that a successful scoped migration or MVP does not necessarily predict results for broader production deployments.
Sep 24, 2026
584 words in the original blog post.
Feature flag cleanup requires more than deleting a conditional because stale flags can persist across configuration, evaluation calls, tests, analytics, documentation, and multiple repositories, while removing an incorrect branch can unintentionally restore outdated behavior. Before using coding agents for coordinated searches and edits, teams should verify the flag’s ownership, purpose, rollout status, production values, environments, evaluations, and any customer or regional exceptions, stopping when evidence is incomplete. Once retirement is confirmed, the change should narrowly remove the flag evaluation, obsolete branch, configuration, transition-specific tests, mocks, metrics, and comments while preserving the established permanent behavior. Cross-repository removals need an explicit sequence, generally ensuring consumers no longer read a key before its central definition is deleted, and standards such as OpenFeature can clarify evaluation behavior but cannot prove rollout completion. Validation should include targeted and adjacent-path tests, production builds, renewed searches for keys and aliases, and user-flow checks where applicable, with the final pull request documenting the retired flag, retained behavior, repositories reviewed, and verification evidence.
Sep 24, 2026
624 words in the original blog post.
Cross-repository coding agents are most effective when coordinated changes follow an explicit dependency and compatibility plan rather than parallel, isolated edits. Teams should identify every repository that produces or consumes an interface, define sources of truth, owners, checks, release order, and whether changes are additive or require transitional compatibility, such as expand-and-contract database migrations. Work can then be divided into milestones covering contracts, producer updates, consumer updates, and eventual removal of legacy paths, with a coordinator maintaining shared state while repository owners retain approval authority. Separate branches or worktrees prevent file conflicts, while durable records of interface versions, blockers, rollout decisions, and validation results help focused workers remain aligned. Because local tests may pass despite system-level failures, final validation should test exact producer and consumer revisions together, confirm generated clients and version constraints, and verify deployment manifests, with completion withheld until cross-repository evidence succeeds.
Sep 24, 2026
581 words in the original blog post.
Database migrations performed with coding agents require safeguards because production databases contain live state that is not represented in source code. The recommended parallel-change approach divides incompatible changes into expansion, migration, and contraction phases, preserving compatibility while application versions transition and giving agents bounded tasks such as adding compatible schema paths, running measured backfills, and removing obsolete code only after adoption. Migration changes should be tested in isolated databases with representative edge cases, while reviews assess migration ordering, reversibility, application compatibility, locking behavior, validation options, backfill performance, transaction size, and pause signals. Tools such as Factory Missions and automated QA can distribute coding, testing, and review work and retain evidence, but production execution should remain under human approval. Effective monitoring tracks migration-specific indicators including remaining rows, error rates, lock waits, replication lag, and use of legacy paths, since a completed job does not necessarily confirm correct data migration.
Sep 24, 2026
561 words in the original blog post.
Accessibility testing with coding agents is most effective as a layered process that combines static analysis, browser automation, and human evaluation, since automated tools can identify clear technical defects but cannot independently establish full accessibility compliance. Teams should convert relevant WCAG 2.2 requirements into specific, observable behaviors for defined user flows, including keyboard navigation, focus management, semantic roles, accessible names, error handling, and responsive behavior across relevant states and environments. Agents can run linters and deterministic scanners, exercise critical paths with keyboard-only interaction, inspect accessibility trees, and capture screenshots or other evidence, while keeping testing scope aligned with the code changes. They can safely fix issues such as unlabeled inputs, incorrect heading order, improper controls, and missing focus handling, but decisions involving alternative text, reading order, product intent, or assistive-technology usability require designers, content owners, accessibility specialists, and independent reviewers. Findings should document the tested route, conditions, steps, expected and actual outcomes, evidence, and tool versions, with blocked checks clearly identified rather than treated as passing, while recurring automated QA can be triggered through repository or GitHub workflows.
Sep 24, 2026
600 words in the original blog post.
Private AI-assisted legacy modernization should begin with a narrowly defined migration whose existing behavior, dependencies, acceptance criteria, and responsible maintainers are clearly identified. Factory Private is presented as a customer-controlled environment for running AI agents, builds, model services, telemetry, and tools within approved infrastructure and data boundaries, with reproducible baselines and required dependencies prepared before changes begin. For larger brownfield efforts, Factory Missions can structure work into reviewed plans and milestones that produce inspectable evidence, while simpler replacements may remain standard tasks. Teams are advised to provide agents only the necessary source, documentation, and examples; restrict filesystem and network access; and maintain approval controls for consequential actions. Compatibility should be tested using representative inputs, regression tests, target-like staging environments, manageable code reviews, and documented rollback plans rather than relying solely on successful builds. Progress should be measured through accepted changes, completed validation checks, failures, reviewer corrections, and reliable cost data, with successful pilot practices retained and expanded only after the process is demonstrably repeatable.
Sep 24, 2026
1,298 words in the original blog post.
Factory’s Agent Readiness framework assesses how repository structure, validation practices, documentation, security scanning, and operational signals affect coding agents’ ability to work autonomously, emphasizing actionable engineering gaps rather than adoption scores. Organizations are advised to evaluate active repositories and individual applications within monorepos, using maturity levels as a shared framework while targeting Level 3, Standardized, which includes enforced processes, end-to-end tests, maintained documentation, security scanning, and observability. Readiness does not replace authorization controls or grant production access, and teams should convert findings into reviewed, validated changes such as documented test commands and reproducible local checks. Platform teams can track remediation across repositories, assign ownership, and measure practical outcomes including setup reliability, validation success, and reduced reviewer intervention, while avoiding superficial changes intended only to improve scores.
Sep 24, 2026
624 words in the original blog post.
Factory argues that self-improving software should track recurring user and engineering friction rather than relying only on successful task completion, using evidence such as repeated rephrasing, rejected tool calls, and error recovery to identify failures. Its January 2026 Signals research describes an internal process that abstracts session patterns for human analysts, files issues when friction exceeds a threshold, and has Droid propose and review fixes, while retaining human approval before code changes are merged. Factory reported a 30% reduction in repeated-rephrasing friction within 48 hours for one internally observed change, but presents this as a specific result rather than a universal guarantee. The approach recommends reproducing failures, preserving errors and acceptance tests, integrating automation with existing review and release controls, and measuring post-release outcomes for retries, requirements compliance, costs, and regressions.
Sep 24, 2026
521 words in the original blog post.
Factory’s guidance for governing AI agents in private development networks emphasizes layered, testable controls that remain effective despite ambiguous tasks, unexpected tool output, or incorrect model actions. It recommends defining the agent’s execution identity and granting only the repository, filesystem, network, and service permissions required for a task, with separate reviews when workflows move between environments such as laptops and CI runners. Command allowlists, denylists, and blocklists should distinguish automatic actions, reviewable actions, and prohibited actions, while sandboxing and infrastructure controls limit the broader filesystem and network effects of permitted tools. Organization-managed settings should enforce hard policy boundaries across models, tools, hooks, and sandbox configurations, supported by deliberate policy testing and narrowly scoped exception processes. Hooks and Droid Shield secret scanning provide additional protections for lifecycle checks and staged Git diffs, but should be evaluated for their own permissions, data flows, and limitations. Effective deployments also require evidence from safe positive and negative tests, source-control and CI records, telemetry or audit logs, and periodic reassessment after changes to credentials, integrations, or runtime environments.
Sep 24, 2026
1,301 words in the original blog post.
Factory Managed and Factory Private differ chiefly in who owns the control plane and where development session data, orchestration, and operational responsibilities reside, rather than simply whether agents run on customer hardware. Managed uses Factory-hosted, tenant-isolated control-plane services while customers retain responsibility for local machines, CI/CD, and related access controls, whereas Private places the control plane in customer infrastructure and requires customers to manage updates, monitoring, recovery, certificates, and capacity. Organizations should separately evaluate inference routes, session data, telemetry, tool connections, retention, and diagnostic paths through an approved data-flow diagram, since an approved model provider does not necessarily approve hosted coordination services. The choice should reflect an organization’s data-boundary requirements and operational capacity, with mixed deployment patterns possible if workload placement rules are clear. Air-gapped deployments, GovCloud offerings, and customer-hosted environments address distinct needs and require separate validation, particularly for connectivity, feature availability, and authorization status. Evaluations should test equivalent tasks, permissions, models, policy enforcement, negative cases, review effort, and operator workload, then document permitted workloads, approved services, data classes, responsibilities, exceptions, and criteria for expansion in a versioned acceptance record.
Sep 24, 2026
1,318 words in the original blog post.
Factory describes a software factory as a continuous delivery model in which signals from bug reports, feedback, requirements, and production monitoring inform subsequent work rather than remaining isolated in individual investigations. Its Factory product is presented as tooling for operating this loop across intent intake, context resolution, planning, execution, review and policy, delivery, and observability, while the broader software-factory concept remains independent of any one platform. The article recommends beginning with a specific recurring issue, such as a repeated CI failure, defining what automation may inspect and change, and establishing when human judgment is required. It emphasizes that reliable completion requires testable evidence from running applications, repository tests, and review rules rather than an agent’s explanation, and that failures should be retained to improve instructions or validation. A Chainguard example, in which a Droid session worked across six repositories for two weeks to build 80 packages, illustrates sustained automation but is not presented as proof of universal outcomes.
Sep 24, 2026
605 words in the original blog post.
Reproducible environments for coding agents make results easier to verify, repeat, and trust by recording the source revision, operating system, toolchain, dependencies, commands, outputs, and resulting code changes for each run. The approach relies on starting from a clean named commit, using checked-in lockfiles and executable environment definitions such as containers or machine templates, and storing standard installation, testing, linting, and build instructions in the repository. Each parallel agent should use an isolated worktree or disposable workspace to prevent mutable files, caches, or dependencies from interfering with other runs, while shared caches should remain read-only or carefully namespaced. Agent outputs should preserve commands, exit statuses, test summaries, and relevant logs or screenshots, with discrepancies between local and CI results treated as evidence of missing environment assumptions rather than reasons for unexplained retries. Consistent environments also support software supply-chain assurance, clearer failure analysis, fairer comparisons between agents or prompts, and more concrete evidence for reviewers evaluating changes.
Sep 24, 2026
558 words in the original blog post.
Token efficiency for coding agents depends on returning tool results at the smallest scope that still supports the next correct action, rather than sending broad repository searches, full conversations, or unbounded logs that increase context overhead. The guidance recommends starting with relevant files, symbols, repositories, or time windows, expanding only when specific questions remain, and preserving essential evidence such as commands, errors, affected files, file state, and prior decisions. Factory’s MCP integration and project instructions can help constrain access to needed external data and document important code locations and validation commands. Citing Anthropic and Factory research, the piece notes that compression and smaller outputs should be evaluated by retained context quality and total task cost through accepted completion, including retries, worker activity, recovery calls, validation, elapsed time, and reviewer intervention. It recommends controlled experiments that modify one recurring tool call or instruction while holding models and acceptance tests constant, with token counts kept distinct from billed cost.
Sep 24, 2026
511 words in the original blog post.
API compatibility checks help prevent service changes that pass unit tests yet break clients through altered fields, inputs, defaults, authentication, or error responses. Maintaining an OpenAPI specification in version control provides coding agents with a baseline contract, but effective reviews also require explicit compatibility policies, deprecation rules, consumer tests, SDKs, gateway behavior, and evidence of how real clients use the API. Proposed changes should be compared against the baseline with tools such as OpenAPI Diff, while agents also trace behavioral effects across validation, authorization, serialization, storage, events, generated SDKs, and documentation. The recommended approach favors additive changes, explicit and owned compatibility shims, consumer migration before removal, and a defined deprecation period. Pull requests should document contract differences, affected consumers, migration sequencing, and focused test evidence, with provider tests run against old requests and consumer tests against the new service, particularly for authentication and failure paths. Human approval remains important for accepted breaking changes, and agents should report blocked or incomplete evidence rather than make unsupported claims when they cannot establish a baseline, identify consumers, or reproduce prior behavior.
Sep 24, 2026
603 words in the original blog post.
Release note automation uses coding agents to turn merged software changes into audience-focused summaries, but it requires strict evidence and review to avoid unsupported claims. Effective workflows define immutable release boundaries using tags, branches, build identifiers, repositories, and cutoff times, then derive change ranges from commits and pull requests rather than date windows. Agents should inspect merged diffs, tests, configuration, APIs, migrations, feature flags, and documentation to distinguish implementation details from verified user impact, while citing sources for each entry and grouping related changes by outcome. Notes should address the distinct needs of end users, operators, API consumers, and security stakeholders, highlighting breaking changes, deprecations, security fixes, and required actions. Before publication, teams should validate drafts against the final release artifact, exclude reverted or disabled features, obtain specialist review where needed, and retain human approval because automated drafting and publishing carry different risks.
Sep 24, 2026
617 words in the original blog post.
Factory and Claude Code may generate different costs for the same successful engineering task because of variables such as model selection, cache use, context handling, tool output, retries, and failed attempts, so neither product can be declared cheaper without controlled testing. A fair comparison should use the same repository revision, environment, task description, predefined acceptance criteria, and documented model, provider, reasoning, and configuration settings, while separating product-native workflows from same-model harness tests. Evaluations should track input, output, cache reads and writes, reruns, corrections, human intervention, elapsed time, reviewer effort, and both accepted and failed tasks across varied work types. Factory’s support for custom models, external providers, local endpoints, and routing makes model choice and routing important variables, while Claude Code also provides documented controls that should be assessed according to their configuration and date. The recommended headline measure is total measured model cost per accepted task, with subscription fees distinguished from metered API spending, and any savings claim should be supported by an inspectable, reproducible record rather than a demonstration alone.
Sep 24, 2026
567 words in the original blog post.
AI coding agents can help prepare infrastructure-as-code changes, but safe review requires more than syntactically valid configurations because small diffs can affect networks, identities, storage, and production capacity. Changes should be narrowly scoped with explicit permitted modules, accounts, regions, environments, expected resources, and rules for deletions, replacements, or privilege expansions, while tool versions, provider locks, backends, workspaces, variables, and validation steps should be fixed before editing. Agents should generate a fresh Terraform or OpenTofu plan against the target environment, and reviewers should evaluate resource additions, modifications, replacements, destruction, IAM, public access, encryption, retention, networking, data resources, and unresolved values against the intended architecture and local conventions. Plans are useful evidence rather than approval because they can contain unknown apply-time values and depend on the current state and execution context. The guidance recommends retaining automated policy checks and provider validation in CI, separating planning identities from production apply identities, protecting credentials, documenting plan results and rollback considerations in pull requests, and stopping work when workspace selection, state access, or destructive changes cannot be clearly verified.
Sep 24, 2026
596 words in the original blog post.
AI coding agents can make dependency updates more effective by handling compatibility work beyond automated pull requests, including API migrations, configuration changes, generated files, and targeted regression tests. Updates should be narrowly scoped to a dependency family and target version, supported by maintainer release notes, and verified through clear evidence such as expected lockfile changes, removal of deprecated APIs, focused tests, and successful type, lint, test, and build checks. The approach emphasizes reviewability and supply-chain security by separating mechanical and behavioral changes, scrutinizing new transitive packages, registries, checksums, and lifecycle scripts, and using isolated environments with failure-aware CI automation. Existing package alerts and bots should remain the intake mechanism, while agents are triggered only for migrations or deeper validation, operate with least-privilege service accounts, and cannot merge their own changes, leaving final approval to branch protections and human reviewers.
Sep 24, 2026
563 words in the original blog post.
Test fixture maintenance can improve test-suite speed, clarity, and reliability by reducing unnecessary setup, shared mutable state, obsolete data, and external dependencies that cause slow or misleading failures. The recommended process begins by mapping each fixture’s consumers, scope, setup and teardown behavior, generated data, and dependencies, while measuring baseline runtime and confirming actual usage through focused test runs rather than static searches alone. Problems such as oversized, order-dependent, remote, or stale fixtures require distinct remedies, including splitting fixtures by tested behavior, preserving cleanup for resources such as transactions and containers, using shuffled or isolated test execution to reveal coupling, and making generated data reproducible through recorded random seeds. Coding agents can assist with dependency tracing and repetitive changes, supported by repository-specific instructions in files such as AGENTS.md, but changes should remain narrow and avoid combining fixture work with production refactoring. Validation should test direct consumers and the full shared suite, compare performance and test results against baselines, explicitly review snapshots and generated records, and document the fixtures changed, commands run, measurable effects, and any remaining shared state.
Sep 24, 2026
620 words in the original blog post.
Runtime upgrades with coding agents can reduce manual effort, but successful migrations still depend on carefully managing compatibility across version declarations, dependencies, native modules, CI, containers, deployment settings, and local development tools. Teams should choose an explicit supported target runtime, establish a reproducible passing baseline, inventory every runtime selector, and preserve repository-specific build and test instructions for agents to follow. The recommended approach separates mechanical version updates from compatibility repairs, addresses failures incrementally using upstream migration guidance, and avoids unnecessary dependency refreshes or unrelated refactoring. Validation should include the complete repository gate, deployment-equivalent artifacts, clean container builds, supported version ranges for libraries, and runtime behaviors such as startup, networking, serialization, background tasks, and native extensions. Pull requests should document all changed selectors, tests performed, required dependency updates, and unsupported environments, while long-term maintenance should consolidate configuration sources, remove obsolete pins only after adoption, and add drift detection between local, CI, and deployment environments.
Sep 24, 2026
598 words in the original blog post.
Localization testing must extend beyond successfully compiling translation files to verify that supported user journeys work across languages, scripts, regional formats, and interface directions. Effective coverage identifies supported locales and flows, includes long-text and right-to-left cases, uses deterministic synthetic data with controlled time, currency, and measurement settings, and validates translation catalogs for missing keys, placeholder errors, and syntax issues. Coding agents can automate browser-based tests that change locale settings, confirm persistent preferences, inspect formatting and navigation, capture visual evidence of clipping or overlap, and test non-ASCII input behavior such as composition, cursor movement, and paste. Failures should be categorized by source content, locale formatting, layout, or directional behavior, then retested in the affected, source, and additional supported locales at relevant viewport sizes. Combining functional assertions, visual regression checks, and documented testing assumptions helps ensure localization quality beyond a successful catalog build.
Sep 24, 2026
606 words in the original blog post.
Visual regression testing with coding agents helps teams identify interface changes that may affect specific pages, states, and viewport sizes, but it relies on stable screenshot baselines created in consistent browser, operating system, font, and rendering environments. Reliable tests require deterministic data, fixed locale and time settings, loaded fonts, disabled irrelevant animations, carefully limited masking, and focused component or region snapshots alongside selected full-page checks. Agents can investigate failures by examining source diffs, baseline and current images, browser logs, and automated browser recordings to distinguish environment noise, such as missing fonts or GPU differences, from product defects like clipping, spacing errors, hidden controls, or responsive failures. Proposed fixes should be validated first with targeted tests and then against nearby states, while pull requests should provide design references, changed screenshots, viewport information, reproduction steps, and explanations for approved differences. Human reviewers retain responsibility for confirming that baseline changes match intended design, with visual checks complemented by accessibility and functional testing.
Sep 24, 2026
583 words in the original blog post.
Configuration drift repair with coding agents relies on comparing observed infrastructure and service settings against trusted, documented baselines while recognizing that detected differences do not automatically authorize changes. Effective workflows identify authoritative sources for each setting, collect read-only and sanitized drift evidence, classify differences such as unauthorized changes, approved emergency changes, or provider-generated values, and avoid overly broad ignore rules. Remediation should either update code to preserve an approved live change or deploy a narrowly scoped, reviewed patch that restores the declared baseline, with policy checks, infrastructure plans, and human approval retained as required safeguards. Validation includes rerunning formatting, policy, and planning checks, closely reviewing potentially high-impact changes, confirming convergence after authorized deployment, and investigating persistent drift caused by provider defaults or competing management systems. Long-term reliability depends on addressing the underlying causes through stronger permissions, improved procedures, documentation, and scheduled detection.
Sep 24, 2026
585 words in the original blog post.
Backporting production fixes with coding agents requires more than successfully cherry-picking a commit, since older release branches may differ in dependencies, APIs, build tools, schemas, and release policies. Teams should define the affected branches, isolate worktrees, identify the minimal portion of the original change needed to fix the defect, and give agents clear acceptance criteria, verification steps, and boundaries against unrelated modifications. Conflicts should be resolved by preserving the original root-cause correction while adapting implementation details to the older architecture, such as using compatible utilities, migrations, or dependency APIs. Each backport should be tested with the target branch’s documented toolchain through regression, package, integration, and release checks, ideally confirming the test fails before the patch is applied. Separate pull requests for each branch should document the original fix, version-specific adaptations, provenance, and exact test results, while automated review can help detect unrelated changes, weak tests, and violations of support policy.
Sep 22, 2026
588 words in the original blog post.
Dead code removal with coding agents can lower maintenance costs and reduce build and search clutter, but it requires caution because apparently unused code may still be activated through configuration, reflection, framework conventions, plugins, dynamic imports, or external API consumers. Reliable decisions should combine compiler and lint signals with reference searches, dependency and package-consumer checks, test coverage, version history, deployment configuration, and confirmation from code owners, while documenting the strengths and limits of each signal. Removals should be made in small, reviewable slices without unrelated formatting or redesign, with generated artifacts regenerated through documented tooling and shared tests preserved. After deletion, teams should run type checks, linting, targeted tests, builds, and runtime checks for discovery mechanisms such as routes, scheduled jobs, plugins, command registries, and dependency injection, while comparing observable behavior and bundle contents where relevant. Automated and local code review can supplement deterministic build gates by checking for hidden registrations, public exports, initialization effects, and weakened tests, and pull requests should clearly record removed entry points, evidence, generated changes, validation steps, and any uncertain consumers that should block deletion.
Sep 22, 2026
582 words in the original blog post.
Performance reviews for coding-agent-generated changes should rely on measurable baselines, representative workloads, and explicit performance budgets rather than visual inspection of diffs alone. Comparable tests must control variables such as datasets, concurrency, cache state, runtime, hardware, and sample size, while metrics should match the affected area, including bundle size and interaction timing for frontend changes or query counts, execution plans, latency, throughput, and memory for backend work. Reviewers should account for normal measurement variance, retain raw benchmark and profiling artifacts, and trace changed code to risks such as repeated requests, unbounded loops, eager loading, lock contention, or cache invalidation. Tools including profilers, bundle analyzers, query planners, load tests, and browser measurement systems can provide reproducible evidence when paired with repository-specific instructions, named workloads, thresholds, and commands. Findings should identify the affected input, metric, comparison commit, environment, and explanatory code path, while optimizations must still undergo correctness testing to avoid altering freshness, failure behavior, or other acceptance criteria. Stable, decision-relevant measurements can become required CI checks, whereas noisy benchmarks should remain advisory until their variance is understood.
Sep 22, 2026
592 words in the original blog post.
Coding agents can accelerate Git merge-conflict resolution, but successful resolution requires preserving the intended behavior of both branches rather than merely removing conflict markers. Effective workflows pin the exact source and target commits, reproduce conflicts in a clean worktree, provide agents with relevant requirements and repository rules, and account for generated files, lockfiles, schemas, and API compatibility. Resolvers should examine both changes alongside their common ancestor, tests, and call sites, combine compatible behavior where appropriate, avoid unrelated edits, and escalate incompatible product decisions instead of making assumptions. Validation should include checking for remaining markers, reviewing staged changes, running targeted and broader tests, and documenting the conflicting commits, resolution rationale, affected files, and commands used so reviewers can assess the merged result confidently.
Sep 22, 2026
597 words in the original blog post.
Coding agents can inadvertently expose credentials by reading sensitive configuration, logs, fixtures, or tool output and copying them into source files, generated artifacts, or Git history, so secret protection should begin before an agent edits code and continue through commit and push. Recommended practices include granting only necessary access, using short-lived and narrowly scoped credentials, relying on environment injection and sanitized test data, identifying sensitive paths in repository guidance, and preventing secrets from appearing in prompts, snapshots, issue comments, or diagnostic uploads. Factory’s Droid Shield scans added lines during agent-driven Git commits and pushes, but manual Git activity requires additional safeguards such as CI secret scanning, pre-commit checks, hosting-provider scanning, and server-side push protection. Teams should also inspect non-source artifacts such as source maps, notebook outputs, recordings, archives, and encoded fixtures, maintain narrowly reviewed allowlists, and treat confirmed exposure as an incident by blocking publication, revoking or rotating affected credentials, documenting remediation without restating the secret, reviewing access logs, and improving the path that exposed the value.
Sep 22, 2026
603 words in the original blog post.
Self-hosted AI agents require enterprises to evaluate not only model placement but also ownership of orchestration, session handling, inference, analytics, connected tools, maintenance, and support access. Factory presents Managed as a hosted control-plane option and Private as a customer-hosted alternative, with deployment choices determined by specific requirements for data boundaries, access controls, and operational responsibility rather than by labels alone. Organizations considering private deployments should map data flows, gateways, retries, diagnostics, and dependencies; establish processes for releases, recovery, certificate management, model validation, and incident response; and test complete workflows rather than isolated model responses. Airgapped deployments impose additional restrictions by eliminating runtime connectivity and disabling cloud-dependent features, while GovCloud hosting and customer-operated infrastructure do not by themselves demonstrate completed authorization, particularly as Factory’s FedRAMP status was described as in progress. The recommended approach is to compare options through bounded tasks, approved permissions, known test suites, output quality, review effort, and the organization’s ability to operate and validate the chosen system.
Sep 18, 2026
587 words in the original blog post.
Deploying AI coding agents in IL4 and IL5 environments requires an end-to-end assessment based on the data classification, authorized system boundary, and responsible authorizing official’s decision, rather than relying on labels such as private network or airgapped deployment. Evaluations should account for source code, prompts, model endpoints, connected tools, telemetry, identity systems, package sources, source control, and operational controls, with each integration remaining within the approved configuration. Factory describes support for private, sovereign, and airgapped arrangements, but these statements represent deployment capabilities rather than blanket authorization for specific workloads or services; as of September 18, 2026, its FedRAMP authorization was still in progress. Private deployments place the control plane in customer infrastructure, while airgapped deployments use customer-configured models and exclude Factory-bound and cloud-dependent functions such as Slack integration, session sharing, and hosted analytics. Organizations should validate complete workflows using synthetic data, document data flows and responsibilities, test rejected connections and failure recovery, and maintain acceptance records identifying the workload decision, service scope, active configuration, and verification evidence.
Sep 18, 2026
599 words in the original blog post.
AI code data residency depends on the full path of source files, prompts, model inference, session records, telemetry, integrations, logs, and diagnostic outputs rather than solely on where a Git repository is hosted. Factory states that Droid works on files locally and routes inference context through the configured model path, while its cloud-managed, hybrid, and airgapped options differ in their data boundaries; organizations must also verify whether gateways, proxies, or fallback routes send data to external providers. Model hosting and control-plane placement are separate considerations, requiring review of session handling, administrative records, support diagnostics, retention, and deletion procedures. Telemetry, MCP services, hooks, and test tools can create additional data destinations, particularly during failures, and Factory notes that content logging is disabled by default but sends unredacted raw content to a customer-configured collector when enabled. The recommended approach is to use representative synthetic tasks to observe and document every data destination, operator, purpose, retention policy, and enforcement point, repeating the assessment whenever models, collectors, integrations, or execution environments change.
Sep 18, 2026
608 words in the original blog post.
Private AI security remediation should focus on reducing verified risks through narrowly scoped, reviewable changes rather than generating large volumes of unvalidated findings or patches. Effective workflows begin by confirming reachability, exploitability, existing controls, and false positives, while using synthetic data and disposable environments instead of live systems or real credentials. Organizations should limit repository access, keep production credentials out of task environments, apply sandboxing and secret-scanning controls, and avoid granting deployment authority solely because an AI agent can prepare a patch. Each remediation should include evidence, focused code changes, regression testing for the unsafe behavior, confirmation that legitimate behavior still works, and transparent reporting of any tests that could not run. Reviewers should independently inspect changed data paths and possible bypasses, since passing tests may not cover every entry point. Success should be measured through validated findings, accepted fixes, false positives, and review burden, with records documenting why a finding was real, how the patch addressed it, which checks were performed, and who approved the result.
Sep 18, 2026
619 words in the original blog post.
FedRAMP AI deployment should be evaluated based on the exact named service, its current authorization status, and the full architecture of the proposed workflow rather than assumptions based on product families, GovCloud hosting, or vendor claims. As of September 18, 2026, Factory describes its FedRAMP control-plane and analytics hosting in GovCloud as authorization in progress, so organizations should obtain current, dated evidence and involve responsible agency reviewers before committing to deployment. Reviews should cover dependencies such as repositories, model services, identity systems, telemetry, scanners, and ticketing integrations, while assigning owners to unresolved questions and distinguishing commercial, private, federal, connected-cloud, and air-gapped configurations. Air-gapped deployments require customer-managed internal services and lack certain cloud-dependent capabilities, creating additional responsibilities for releases, certificates, model availability, incident response, and recovery. Organizations should validate a bounded, approved workflow with permitted data, document permissions, destinations, tests, audit evidence, and review gates, test both successful and failure scenarios, and preserve the authority’s decision for the precise production configuration rather than a broader product category.
Sep 18, 2026
595 words in the original blog post.
Factory’s air-gapped deployment guidance argues that offline AI development must be validated as a complete workflow, since tools such as test runners, dependency resolvers, model gateways, and plugins may otherwise attempt external connections. Its enterprise airgap build of Droid disables Factory services including sign-in, updates, telemetry, synchronization, hosted models, and routing, requiring organizations to configure and test approved internal model endpoints using representative repositories and realistic development tasks. Network and infrastructure controls should independently restrict all destinations and local access, while approved internal artifact processes must supply source code, runtimes, packages, tools, fixtures, policies, and credentials. Organizations are advised to test model failures and credential expiry, account for unavailable cloud-based collaboration and analytics features, preserve evidence through internal source control and validation records, and rehearse updates and rollbacks. A successful pilot should demonstrate that teams can consistently build, secure, review, and recover within the offline environment rather than merely complete a one-time isolated test.
Sep 18, 2026
599 words in the original blog post.
SOC 2 evaluation for AI coding tools should focus on whether a report’s defined system scope actually covers the selected control plane, model services, integrations, and workflow rather than treating a security badge as proof of compliance. Reviews should examine report periods, exceptions, customer responsibilities, and the distinction between an auditor’s attestation and real-world configuration choices such as model routing, regional deployment, repository permissions, execution identities, and network controls. Organizations should map end-to-end data flows, including local file access, prompts, model context, telemetry, session records, diagnostics, and error paths, while separately assessing vendor-operated and customer-operated components. Useful evidence includes current model policies, retention settings, sandboxing and approval controls, and bounded tests conducted in disposable environments, with sensitive session content collected only when necessary. The resulting acceptance record should identify the reviewed service, configuration, unresolved risks, responsible owners, and reevaluation triggers, since changes in model routes, integrations, or deployment ownership can alter the compliance posture.
Sep 18, 2026
615 words in the original blog post.
Self-hosted AI models can help keep confidential source code within approved environments, but organizations must verify the complete inference route, including gateways, downstream services, credentials, logs, retention policies, and fallback behavior. Factory recommends documenting model identifiers, endpoints, providers, operators, and data-handling policies, while using enterprise controls to limit developers to approved model destinations rather than relying only on default selections. Credentials should be managed through secure, organization-controlled mechanisms such as keyless endpoints or runtime helpers, with testing for expiration, revocation, identity failures, and log exposure. Teams should validate models using realistic development workflows involving file access, tool calls, edits, tests, and error correction rather than simple chat responses, and should maintain repeatable evaluations when changing models or gateways. In air-gapped environments, customer-configured internal models are required, while outages should halt work or move it to another approved internal route.
Sep 18, 2026
596 words in the original blog post.
Factory’s guidance on AI agent telemetry emphasizes collecting only the operational evidence needed, distinguishing lower-sensitivity activity metrics from potentially sensitive raw content and approval records. Metrics-only customer exports are enabled by default, while optional content spans may include prompts, files, commands, and tool outputs without automatic redaction, requiring approved collectors and centralized policies. Organizations are advised to secure collector credentials, verify configurations on actual execution accounts, test export failures, and review downstream storage, access, and retention practices. The guidance also recommends deliberately selecting identity granularity, validating exported schemas with non-sensitive samples, and accounting for air-gapped environments where no hosted fallback exists. To assess outcomes rather than mere activity, telemetry should be combined with CI results, source-control history, and reviewer approvals, while cost data should come from appropriate inference or infrastructure sources. Pilot programs should use stable definitions for measures such as accepted changes, validation failures, corrections, and operator intervention to avoid mistaking measurement changes for productivity gains.
Sep 18, 2026
591 words in the original blog post.
On-premises AI development for private repositories requires evaluating every dependency in the workflow, including the execution environment, control plane, model endpoint, package sources, operational-data collection, and permitted network routes. Factory describes cloud-managed, hybrid, and fully air-gapped deployment patterns, with Factory Private offering a customer-controlled control plane in a VPC or on-premises environment while still requiring approved inference services, tools, and connectivity. Teams are advised to first make repository builds reproducible using internally approved artifacts, avoid unreviewed public downloads, and verify where file content travels when used for model inference. Initial agent access should be narrowly scoped through disposable workspaces, limited permissions, command and network restrictions, sandboxing, and continued source-control review, with tests confirming that prohibited actions fail. Successful operation also depends on clear ownership of releases, model availability, certificates, recovery procedures, and measurable results from bounded maintenance tasks, with artifact versions, endpoint settings, policy tests, and reviewed changes recorded before expanding use to additional repositories.
Sep 18, 2026
601 words in the original blog post.