Title: Non-identifiability of word embeddings, and connections to shape analysis
Authors: Simon Preston - University of Nottingham (United Kingdom) [presenting]
Karthik Bharath - University of Nottingham (United Kingdom)
Rachel Carrington - University of Nottingham (United Kingdom)
Abstract: In statistical shape analysis the shape of a centred configuration of landmarks is the information invariant to orthogonal and scale transformations. We will make the connection between shape analysis and word embeddings. A word embedding is a construction of landmarks such that each landmark represents a distinct word and the relative positions characterise word meaning. Many different models have been proposed for constructing word embeddings, and it is typical to compare different embeddings in terms of how well they perform in word similarity and association tasks. Performance is quantified via a function (of the embedding and some test data) that happens to be invariant to orthogonal and scale transformations of the embedding, and hence can be interpreted as a measure of shape. The criteria optimised when constructing word embeddings are, however, invariant to a wider class of transformations. We will explain this and discuss the problematic consequences for interpreting reported performance of word embeddings.