DataRater: Meta-Learned Dataset Curation

The quality of foundation models depends heavily on their training data. Consequently, great efforts have been put into dataset curation. Yet most approaches rely on manual tuning of coarse-grained mixtures of large buckets of data, or filtering by hand-crafted heuristics. An approach that is ultimately more scalable (let alone more satisfying) is to \emph{learn} which data is actually valuable for training. This type of meta-learning could allow more sophisticated, fine-grained, and effective curation. Our proposed \emph{DataRater} is an instance of this idea. It estimates the value of training on any particular data point. This is done by meta-learning using `meta-gradients', with the objective of improving training efficiency on held out data. In extensive experiments across a range of model scales and datasets, we find that using our DataRater to filter data is highly effective, resulting in significantly improved compute efficiency.

Authors

• Dan Andrei Calian
• Greg Farquhar
• Iurii Kemaev
• Luisa Zintgraf
• Matteo Hessel
• Jeremy Shar
• Junhyuk Oh
• András György
• Tom Schaul
• Jeff Dean
• Hado van Hasselt DeepMind
• David Silver

Keywords

No keywords are indexed for this paper.