Blog Link: https://talkie-lm.com/introducing-talkie
Talking to someone from the past with no knowledge of the modern world, what would you ask them?
This is really cool because they can test LM’s abilities to come up with new ideas. They can test if LMs can arrive at inventions or scientific discoveries we know would arise after their knowledge cutoffs
Could a model trained up to 1911 discover General Relativity as Einstein did in 1915?
Vintage LMs are contamination-free by construction, which allows you to do cool experiments such as testing if model with no knowledge of digital computers can learn to code in a modern programming language
Vintage models dramatically underperform compared to models trained on web data but improve at this task with scale