In this post, we’ll take a closer look at the question “Do LMs represent space and time?” inspired by a recent paper. We’ll look at how spatial and temporal information has been encoded in LMs, what this means for practical applications, as well as other aspects such as encoding of fine-grained spatial information and across cultures.
Thank you for sharing! This work may be relevant: https://arxiv.org/abs/2308.15197. It is interesting that it does something about time-aware prediction.
It goes back much longer than the "classic observations" from 2013 you mention. Here's a quite extensive paper from 2009 from Louwerse & Zwaan that might be helpful in reconstructing the intellectual history of these ideas a bit further back: https://onlinelibrary.wiley.com/doi/full/10.1111/j.1551-6709.2008.01003.x (it uses LSA, which was the state of the art embedding technique before word2vec)
Thank you for sharing! This work may be relevant: https://arxiv.org/abs/2308.15197. It is interesting that it does something about time-aware prediction.
It goes back much longer than the "classic observations" from 2013 you mention. Here's a quite extensive paper from 2009 from Louwerse & Zwaan that might be helpful in reconstructing the intellectual history of these ideas a bit further back: https://onlinelibrary.wiley.com/doi/full/10.1111/j.1551-6709.2008.01003.x (it uses LSA, which was the state of the art embedding technique before word2vec)