The NeurIPS 2024 Preshow: A Label is Worth a Thousand Images in Dataset Distillation

Harvard researcher Sunny Qin’s NeurIPS 2024 paper argues that effective dataset distillation depends more on informative probabilistic “soft labels” than on sophisticated synthetic-image generation. Dataset distillation compresses large training sets into small subsets that retain enough information for models to achieve comparable performance while reducing storage and computational costs. Soft labels convey relationships and semantic similarities among classes, regularize training to limit overfitting, and can transfer knowledge from expert models trained on full datasets; experiments found that even randomly paired images and expert-generated soft labels can rival leading methods. The research frames dataset distillation as closely connected to knowledge distillation and identifies a trade-off between data quantity and label-encoded knowledge, with smaller data budgets benefiting from richer labels generated by less extensively trained experts and larger budgets benefiting from simpler knowledge. Although the work focuses primarily on image classification, it suggests applications in detection and natural-language tasks while noting that bias and safety concerns require care, particularly for small or noisy datasets.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.