Fair samples from uneven workers

APL’s new `sample n` operator returns an exact number of rows selected uniformly from all query matches, addressing the bias of combining probability-based filter sampling with an early `take` limit, which tends to favor data scanned first. It uses reservoir sampling, specifically Algorithm R, to process streaming data with constant memory while giving every matching row an equal eventual chance of selection. In distributed queries, each worker maintains a reservoir and reports its total number of matched rows, while a weighted coordinator merge uses hypergeometric allocation to determine how many sampled rows each worker should contribute according to its share of the overall population. This avoids the bias of generic merges that treat equal-sized worker reservoirs as representing equal amounts of data despite differing match counts. The implementation required a dedicated distributed sampling plan and makes count propagation part of the operator contract, although it currently applies only to single datasets and still scans every matching row; future work aims to support unions and reduce scanning without sacrificing fairness.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.