October 2026 Summaries

19 posts from Bright Data

Filter
Month: Year:
Post Summaries Back to Blog
A tutorial demonstrates how to analyze San Diego’s housing market using Bright Data’s Python SDK to retrieve up to 1,000 recently sold Zillow property records filtered by city, state, sale status, and date. It explains how to authenticate the SDK, request and export a dataset snapshot, load the resulting CSV into pandas, and clean fields such as prices, dates, ZIP codes, living areas, and lot sizes while converting measurements to square meters. The workflow creates derived metrics including price per square meter, Zestimate error, estimated gross rental yield, price bands, and property age bands, then uses matplotlib, seaborn, and Plotly to visualize market patterns. Example findings show a right-skewed price distribution, strong variation across home types and ZIP codes, higher prices in coastal and northern areas, and neighborhood effects that can outweigh home size in explaining price per square meter. The analysis also finds that Zestimate closely tracks sale prices in the sample, listing duration has little relationship to price, and estimated ZIP-code rental yields fall within a relatively narrow range. The tutorial concludes that the SDK streamlines access to filtered, structured web data so analysts can focus on exploratory analysis, visualization, predictive modeling, and comparisons across housing markets.
Oct 22, 2026 3,953 words in the original blog post.
DSPy can build web-enabled research agents by converting tools from Bright Data’s Model Context Protocol server into DSPy tools for Google search and Markdown page scraping, with the server handling bot protection and prompt-injection elicitation requests. The guide describes configuring a minimal two-tool agent, keeping API keys in headers, restricting tool selection to reduce prompt size, and using synchronous wrappers around a background asyncio event loop because asynchronous MCP tools may silently fail or hang when DSPy’s Evaluate and GEPA run work on threads. In tests using gpt-5.4-mini, web tools increased performance on 60 SimpleQA Verified questions from 14 correct without tools to 54–58 correct with tools, and improved results on recent-event questions, though repeated runs and LLM-based judging introduced measurable variability. It recommends caching, retries, timeouts, output-length limits, domain blocking during benchmark evaluation, source verification, and MLflow tracing to make tool use more reliable and auditable. Error analysis found that many failures came from answers located beyond page truncation limits, while added find and pagination controls improved long-page performance and reduced token use. GEPA prompt optimization did not measurably outperform the baseline agent in the reported tests and sometimes copied training answers into instructions, so optimized prompts require careful review and evaluation on held-out data.
Oct 22, 2026 5,214 words in the original blog post.
Bright Data’s CLI and Python SDK provide access to the same web-data infrastructure and core services, including protected webpage scraping, search-engine queries, structured extraction from major platforms, remote browser control, custom scrapers, and account information, with a shared free tier of 5,000 monthly credits. The CLI, requiring Node.js and authenticated primarily through OAuth, is aimed at terminal-first workflows, quick one-off tasks, shell pipelines, cron jobs, account monitoring, and AI coding agents, returning formats such as Markdown, JSON, HTML, and screenshots through standard output. The Python SDK, requiring Python and an API key or existing CLI credentials, offers synchronous and asynchronous typed clients for application development, concurrent batch scraping, data pipelines, backend integrations, and direct processing of structured response objects. Although most CLI commands closely correspond to SDK methods, the CLI offers persistent browser control and fuller Scraper Studio management, while the SDK provides native concurrency, programmatic error handling, and easier integration with Python tools such as data-processing libraries. The comparison concludes that the CLI is preferable for simple command-line automation and interactive tasks, whereas the SDK is better suited to complex, scalable Python workflows, and the two can be used together.
Oct 20, 2026 3,613 words in the original blog post.
Microsoft Agent Framework (MAF) is an open-source Python and .NET SDK for building AI agents that can connect to Model Context Protocol servers, including Bright Data Web MCP for live search and web scraping. The guide demonstrates how to configure this connection securely by pinning compatible MCP SDK versions for MAF 1.21–1.22, sending the Bright Data API key in an authorization header rather than a URL, limiting available tools and prompts, and capping scraped page sizes to control token use. It outlines a research-agent design that searches before scraping, limits function calls, supports selective approval for batch scraping, audits tool activity without logging page contents, and can pass only concise research notes rather than full scraped conversations to a tool-free writer agent. It also discusses prompt-injection protections, including Bright Data’s untrusted-content markers, URL allowlisting for agents with access to sensitive data, and avoiding unnecessary secrets in research agents. In tests involving 20 current GitHub release questions, a restricted Bright Data tool configuration achieved 20 correct answers with lower model-input costs than the default toolset, while models without web access answered none correctly. The guide additionally provides an equivalent .NET setup and recommends evaluating configurations continuously on verifiable, frequently changing questions before using agents in production.
Oct 20, 2026 5,900 words in the original blog post.
Bright Data’s Model Context Protocol server extends the open-source Cherry Studio desktop AI client with country-specific web search, access to anti-bot-protected pages, and structured data from pre-built web scrapers. The guide explains how to add the hosted Streamable HTTP server through JSON configuration and browser authorization or API-key headers, or alternatively run the open-source local package through Node.js, while emphasizing that a Connected server must also be manually enabled for each assistant or agent. Recommended configuration includes enabling only needed tools, keeping scraper runs subject to approval, using a tool-capable model, disabling Cherry Studio’s built-in browser when all web requests should use Bright Data, and setting long-running timeouts for content-gate interactions. Suggested system-prompt rules require models to verify prices, dates, and other figures from scraped source pages rather than search snippets, cite every claim, validate article dates for time-limited news requests, and treat webpage instructions as untrusted data. Demonstrations cover cross-country product price comparison, 24-hour news summaries, and documentation notes, while noting that additional verification improves accuracy but increases tool usage, cost, and response time. The guide also outlines agent-specific permissions, optional skills and CLI use, troubleshooting for authorization and transport errors, and spending controls through Bright Data usage limits and a monthly free tier for new MCP users.
Oct 20, 2026 4,082 words in the original blog post.
Bright Data’s Python SDK is a typed PyPI library that provides synchronous and asynchronous clients for accessing Bright Data services, including web scraping, search engine results, platform-specific structured-data extraction, datasets, Scraper Studio, and browser connections. It abstracts direct API calls through Python methods, supports authentication via tokens, environment variables, or stored CLI credentials, and can automatically configure required API zones unless users choose to manage them manually. The guide explains installation, authentication, single-page and concurrent scraping, synchronous versus asynchronous request behavior, polling for long-running scraper jobs, and exporting results in JSON, CSV, or JSONL. Examples show practical uses such as collecting LinkedIn profiles, comparing Google and Bing results, retrieving Amazon product and review data, and combining unblocked page retrieval with BeautifulSoup parsing.
Oct 18, 2026 3,573 words in the original blog post.
Claude Code routines are scheduled cloud-based agent sessions that can clone repositories, use configured environments and connectors, and run without a laptop, but their default network sandbox restricts most public websites and a successful routine status does not ensure its intended task succeeded. The guide describes building a daily pricing monitor using Bright Data’s Web MCP connector for Claude’s page reading and the Web Unlocker API, secured through a network secret, for a Python script that fetches and validates public pricing pages. The monitor stores snapshots and reports in GitHub, checks redirected URLs, expected text, and minimum price-line counts, detects content drift even when sites return HTTP 200, compares current prices against prior snapshots, updates a private Claude artifact dashboard, and sends notifications only for changes or failures. Setup involves connecting Bright Data, configuring an API key and Web Unlocker zone, editing a watchlist, publishing a dashboard, and scheduling a routine with a repository skill that commits results directly to main. Testing across nine real pricing pages found all 1,948 simulated single-price changes and detected a real Together AI Kimi K3 promotion removal, while the guide emphasizes least-privilege connector access, treating scraped content as untrusted, preserving state between isolated routine sessions, monitoring routine continuity, and accounting for Bright Data usage and Claude subscription limits.
Oct 18, 2026 6,047 words in the original blog post.
Google closed its Custom Search JSON API to new customers in January 2026 and plans to end existing access on January 1, 2027, leaving no self-service Google product that provides full-web search result lists to new developers. Google’s remaining offerings are limited to domain-specific Agent Search, Gemini Grounding answers with restrictive reuse terms, or an unpublished full-web solution available through an interest form. The guide argues that direct Google scraping is unreliable because search pages require JavaScript and may trigger CAPTCHAs, and presents Bright Data’s SERP API as an alternative that returns parsed Google results, including organic links, ads, People Also Ask questions, images, and optionally AI Overviews. It compares pricing and capabilities, maps Custom Search parameters to Google URL parameters, and provides a Python adapter intended to preserve common cse.list() workflows while handling pagination, country and language settings, site filters, image search, retries, and errors. It also outlines integrations for MCP tools, LangChain, and n8n, recommends testing against saved search-result baselines behind a feature flag, and advises developers to account for expected differences between Custom Search’s limited index and live Google results.
Oct 18, 2026 4,740 words in the original blog post.
Bright Data’s SERP API and Parallel’s Search API serve AI agents through different approaches: Bright Data retrieves live search-engine results pages, including Google rankings, map packs, AI Overviews, ads, shopping results, and city-specific results, while Parallel searches its own index and returns ranked URLs with Markdown excerpts tailored to an objective. Tests across 30 queries found that Parallel’s Advanced mode overlapped with 35–43% of Google’s first-page URLs, while faster modes had substantially lower overlap, particularly for local searches where city-level results varied sharply and Parallel supported country rather than city targeting. Bright Data also offers full-page retrieval through Web Unlocker, including bot-protected pages and location-aware content, whereas Parallel provides excerpts in search responses and separately offers full-page extraction but may be unable to access blocked sites. The comparison recommends Bright Data for use cases requiring the live Google experience, local targeting, rank or brand monitoring, news coverage, and SERP features, while Parallel may suit open-ended research where Google’s specific rankings are less important and concise, ranked excerpts are useful. It also emphasizes evaluating total per-answer costs, including API calls and model-token usage, and suggests testing each tool against an agent’s own logged queries before selecting or combining them.
Oct 18, 2026 4,548 words in the original blog post.
Bright Data and Snowflake can be combined to turn large-scale external web data into AI-ready business intelligence, with Bright Data handling web collection, filtering, and delivery while Snowflake stores, joins, processes, and activates the data through Cortex AI. The workflow emphasizes reducing downstream storage and compute costs by filtering datasets before ingestion, illustrated through an Instagram influencer-scoring example that narrows hundreds of millions of profiles to a targeted subset before exporting it to Snowflake. Within Snowflake, Cortex Agents can use Cortex Analyst for natural-language SQL over structured data, Cortex Search for semantic and keyword retrieval, and Cortex Sense for business context, enabling agents to combine web-derived profiles with internal campaign data. The integration supports two complementary access models: persistent batch datasets for repeatable, auditable, join-heavy analysis and Bright Data MCP-based live web access for current, ad hoc, or time-sensitive research. Bright Data MCP can also extend Snowflake CoCo CLI with web search, scraping, structured extraction, and browser automation, allowing organizations to use stored and live web data together in hybrid AI architectures.
Oct 15, 2026 2,817 words in the original blog post.
Free SERP APIs provide structured search-result data while handling proxies, CAPTCHAs, and parsing, but their free offerings vary substantially in renewal schedules, credit-to-search conversion rates, card requirements, supported engines, and post-free-tier pricing. The comparison ranks 10 services for 2026, identifying Bright Data as offering the largest clearly stated recurring allowance for Google results, with 5,000 monthly requests without a credit card, followed by alternatives including Decodo, Apify, Oxylabs, SerpApi, Serper, and DataForSEO. It distinguishes traditional SERP APIs that reproduce live Google or other engine results from AI-oriented search services such as Tavily, Exa, and Brave, which use proprietary retrieval systems and are better suited to agent research than SEO rank tracking. The guide also notes that Google Custom Search JSON API is closed to new customers, Microsoft’s Bing Search APIs were retired, and self-hosted SearXNG may be unreliable for production use because upstream engines can block it. Bright Data is presented as particularly accessible for coding agents through a hosted MCP server, CLI, REST API, SDKs, and shared credits that can also fund page retrieval, while the text advises retry logic for occasional CAPTCHA or rendering failures. It concludes that users should select a service based on whether they need recurring Google SERP data, a one-time testing credit, or AI-ranked web content, then assess actual usage before moving to paid plans.
Oct 14, 2026 4,746 words in the original blog post.
OpenOSINT is an MIT-licensed, AI-powered open-source intelligence framework that combines OSINT utilities, external intelligence APIs, search services, and web-scraping capabilities for authorized investigations involving emails, usernames, domains, IP addresses, phone numbers, and public web content. It offers 20 integrated tools for account discovery, breach exposure, WHOIS and DNS analysis, subdomain enumeration, infrastructure and threat-intelligence checks, GitHub research, geolocated news, and targeted search queries, while supporting concurrent execution, persistent session histories, Markdown and PDF reporting, multiple AI backends, and MCP integration. Bright Data expands the framework through live SERP searches, target footprint discovery, and protected-page extraction into Markdown, allowing users to retrieve search results and source material directly instead of manually opening generated Google dork links. The tutorial describes installing OpenOSINT with Python, configuring OpenAI and optional Bright Data credentials, investigating a permitted target through its REPL, and using live search tools, while also noting support for CLI, Docker Compose, a web interface, local Ollama inference, and MCP-compatible AI clients.
Oct 14, 2026 2,404 words in the original blog post.
Bright Data’s web-data tools are now available through DigitalOcean Action Gateway, allowing developers of DigitalOcean Managed Agents to access live web search, public webpages, structured extraction, interactive browsing, and refreshed datasets across more than 100 domains through an existing gateway endpoint. The integration is intended to address the difficulty agents face in obtaining reliable, current information such as competitor pricing, inventory levels, and reviews from websites that may resist automated access. Bright Data and DigitalOcean describe potential uses including retail pricing agents that combine internal sales history with real-time market data to make and refine recommendations. Future phases are planned to integrate Bright Data datasets with DigitalOcean Agent Data Services for combined public and private data analysis, followed by Bright Data deploying its own agents on DigitalOcean Managed Agents, creating a managed workflow from data access to scalable agent deployment.
Oct 13, 2026 468 words in the original blog post.
Booking.com scraping tools convert public hotel listings, prices, room availability, amenities, and reviews into structured data for uses such as rate monitoring, market research, and travel-product development, but data collection is complicated by AWS WAF challenges, JavaScript-rendered pages, price variations by dates, guests, currency, and visitor location, and a 1,000-result search limit. Based on vendor websites reviewed in October 2026, the guide compares 10 services with Booking-specific products on data coverage, anti-bot handling, parameter support, output formats, cost, and user-review scores. It ranks Bright Data as the overall option because of dedicated APIs, datasets, CLI access, and a 5,000-record monthly free tier, while highlighting Apify and Octoparse for no-code use, HasData for parsed search and property JSON, Crawlbase for geographically localized prices, Outscraper for lower-volume-based costs, Oxylabs for enterprise batch workflows, Decodo for multi-travel-site coverage, ScrapingBee for AI-assisted extraction, and ScraperAPI for scheduled collection. The guide also distinguishes these tools from Booking.com’s official Demand API, which is restricted to contracted managed affiliate partners and intended for selling inventory, whereas scrapers are positioned for analysis of publicly displayed information.
Oct 08, 2026 3,716 words in the original blog post.
Bright Data Web MCP is presented as a lightweight approach for grounding RAG systems and AI agents with current web information, beginning with a single Google SERP search tool and escalating to full-page Markdown scraping only when snippets are insufficient. The guide reports October 2026 benchmarks showing that the search-only schema uses 168 tokens and Google results typically add 736–885 tokens, while adding page scraping raises the schema to 261 tokens and scraped pages have a median size of roughly 4,400 tokens; exposing the full tool catalog costs 8,414 schema tokens. It recommends filtering tools through hosted URL parameters or local environment settings, using Google with a specified geographic location for compact and consistent results, and enforcing search-first behavior through prompts that limit searches and page reads. The hosted and local MCP configurations work with Claude Code, Cursor, and other MCP-compatible clients, while Web Unlocker handles bot protection, CAPTCHAs, and proxy rotation for page reads. The guide also emphasizes API-key security, spending and rate limits, treating retrieved content as untrusted to resist prompt injection, and a free allowance of 5,000 monthly requests, with additional browser or structured-data tools available only when tasks require them.
Oct 07, 2026 2,860 words in the original blog post.
Octop is an open-source, self-hosted Tencent AI assistant that supports multi-agent collaboration, private knowledge bases, persistent memory, scheduled automation, external integrations, and isolated multi-user workspaces. The tutorial explains that while Octop includes a general-purpose browser, production-scale web research and automation may require stronger infrastructure to handle JavaScript rendering, rate limits, CAPTCHAs, fingerprinting, and other anti-bot measures. Bright Data extends Octop through its CLI, which provides commands for search, scraping, browser control, screenshots, and structured-data pipelines, or through its MCP server, which exposes more than 70 specialized web-data tools including geo-targeted retrieval and domain-specific scrapers. It outlines installing and configuring Octop, connecting an LLM provider, installing and authenticating the Bright Data CLI, importing the associated agent skill, storing CLI availability in agent memory, and testing the setup with an NFL schedule research task that produces a CSV output. As an alternative, it describes connecting Octop to Bright Data’s remote MCP endpoint through the Octop Connectors interface, enabling agents to use Bright Data tools directly for scalable web search, extraction, and browser automation.
Oct 07, 2026 2,872 words in the original blog post.
Kasada is described as an anti-bot platform that identifies automated traffic through IP reputation, TLS and HTTP/2 fingerprints, an obfuscated JavaScript challenge, token management, and behavioral signals. The text explains that sites using Kasada can be identified by x-kpsdk headers and related cookies, while its JavaScript challenge runs a self-checking virtual machine that gathers browser characteristics, performs proof-of-work, and generates the short-lived x-kpsdk-ct token required for subsequent requests. It argues that copied challenge scripts and simple TLS impersonation are insufficient because the payload, decoding keys, fingerprints, and tokens change frequently, and it discusses browser automation signals such as webdriver flags, headless user agents, framework artifacts, and CDP behavior. The piece contrasts maintaining an in-house solver, including browser and proxy infrastructure, with using managed services such as Bright Data’s Web Unlocker or Browser API, and notes emerging standards for authenticated AI agents and verified bots.
Oct 07, 2026 4,181 words in the original blog post.
Kimi Code CLI is an open-source, terminal-based AI coding agent from Moonshot AI that can analyze code, edit files, run commands, and handle multi-step development tasks, but its built-in web access may be limited by model training cutoffs and anti-bot protections. The article explains how Bright Data can extend the agent with live web search, scraping, structured-data extraction, and browser automation through either the Bright Data CLI or Bright Data MCP, including access to tools designed to handle JavaScript-heavy sites, CAPTCHAs, proxy rotation, and geographic targeting. It provides setup instructions for installing Kimi Code CLI, authenticating a Kimi model, installing and logging into the Bright Data CLI, adding the official brightdata-cli Agent Skill, and testing the integration with a product-discovery and scraping workflow that creates JSON output files. It also describes a faster MCP-based configuration using a Bright Data connection URL in Kimi Code CLI’s MCP settings, enabling access to a broad catalog of web-data tools.
Oct 07, 2026 2,779 words in the original blog post.
Auto parts and tire data combines product listings, pricing, fitment, identifiers, availability, and marketplace signals, but its inconsistent formats and cross-reference part numbers make accurate matching difficult across retailers. Brand and manufacturer part number are central matching fields for known SKUs, while complex fitment comparisons may require additional attributes such as vehicle year, model, side, color, and licensed ACES and PIES terminology, which provide manufacturer product and fitment data but not competitor prices, inventory, or local pricing. Competitive data collection from retailer sites requires ongoing handling of anti-bot measures, dynamic pages, retries, monitoring, and clear separation of genuine assortment gaps from collection or matching failures. Tires require a separate schema because size codes, load markings, and original-equipment designations can distinguish otherwise similar products. Bright Data presents pre-collected datasets, scraper APIs, Scraper Studio, and its managed Bright Insights service as options ranging from self-managed collection to full-service matching, normalization, quality assurance, and delivery, with the appropriate choice depending on technical resources, required data freshness, source coverage, and fitment complexity.
Oct 05, 2026 3,285 words in the original blog post.