September 2023 Summaries

Torchmetrics v1.2 introduces a clustering subdomain with 11 metrics for evaluating unsupervised clustering results, distinguishing between eight extrinsic metrics that compare predicted clusters with available ground-truth labels and three intrinsic metrics that assess cluster cohesion and separation using embedded data without labels. The release explains clustering as a technique for finding hidden data structures when the number and meaning of groups are not known in advance, contrasting it with supervised classification. It also expands MeanAveragePrecision for object detection with micro averaging across classes and selectable computation backends, including faster-coco-eval alongside pycocotools. Additional fixes address duplicate dictionary keys in MetricCollection, missing or incorrect metric metadata, and issues affecting BootStrapper, PearsonCorrCoef, and RecallAtFixedPrecision. The project recommends upgrading, notes adoption in more than 2,400 projects and over 1,600 GitHub stars, and invites community feedback and contributions.

Instruction finetuning improves large language models’ ability to follow prompts by training them on instruction, optional input, and desired response pairs through supervised next-token prediction, with reinforcement learning from human feedback presented as a separate optional stage. The discussion contrasts LLM-generated datasets, including Self-Instruct, Alpaca, and instruction backtranslation, with human-created data, noting that synthetic-data-trained models can imitate an upstream model’s style without necessarily matching its factual accuracy. It highlights evidence from LIMA and AlpaGasus that smaller, carefully selected or filtered datasets can outperform much larger collections, emphasizing data quality over quantity. The article also outlines how to prepare datasets and efficiently finetune open-source models such as Llama 2 using Lit-GPT and LoRA, including support for custom CSV-formatted instruction datasets and model-specific tokenization. In the context of the NeurIPS LLM Efficiency Challenge, which prohibits LLM-generated training data and limits training to one GPU for one day, it recommends eligible human-generated datasets such as LIMA, Dolly, Open Assistant, Natural Instructions, P3, and FLAN, while proposing experiments involving dataset merging, ordering, repeated training epochs, and quality filtering.

PyTorch Lightning and Lightning Fabric are complementary frameworks for scaling PyTorch training across multiple devices and nodes, sharing consistent configuration options for accelerators, device counts, node counts, and distributed strategies such as DDP, FSDP, and DeepSpeed. Lightning Fabric is the lower-level, lightweight option, centered on a Fabric class that adds capabilities including mixed precision, distributed execution, checkpointing, and communication utilities while preserving an existing PyTorch training loop with minimal changes. PyTorch Lightning provides a higher-level managed training experience through LightningModule, Trainer, and LightningDataModule, reducing boilerplate by organizing model logic, training processes, and data handling into standardized components. Fabric is intended for users who need close control over custom training workflows, while PyTorch Lightning is suited to conventional models and training loops where concise, reusable code is more important. Both form part of the broader Lightning AI ecosystem alongside TorchMetrics and are used in projects such as Lit-GPT, SDXL, NVIDIA NeMo, HyenaDNA, and other research and production tools.