Language Models Do Not Embed Numbers Continuously (Student Abstract)

We evaluate how well large language model embeddings represent continuous numerical values across different precisions and ranges. Using linear models and principal component analysis on models from major providers, we show that while embeddings can reconstruct numbers with high fidelity (R2 ≥ 0.95), they introduce substantial noise, with principal components explaining less than 40% of embedding variance. Performance degrades with increasing decimal precision and mixed-sign values, revealing fundamental limitations in how these models encode numerical information.

Authors

• Alex Davies University of Bristol
• Roussel Nzoyem Ngueguin University of Bristol
• Nirav Ajmeri University of Bristol, Bristol, UK
• Telmo de Menezes E Silva Filho University of Bristol

Keywords

No keywords are indexed for this paper.