A0402
Title: One sample, one label: Learning from labels with different degrees of informativeness
Authors: Matthias Assenmacher - LMU Munich (Germany) [presenting]
Abstract: Unlike classical tabular data commonly used in statistics, data sets utilized for text mining or natural language processing (NLP) can exhibit a wildly different structure, and so do the labels. For most tasks of interest to NLP research, it is not as easy as just measuring the values of the target variable; instead, human labour often has to be employed for this purpose. This can have several implications when developing and training models: Human annotations of text can be highly subjective (depending on the task and the data), they might incur different costs, or they could altogether be challenging to come up with, as the existence of only one ground truth gold label itself is highly debatable. The probably most prominent example of the latter is the case of open-ended text generation, a task in which the model's capabilities have recently made a significant leap upon the introduction of large language models (LLM). However, the challenge of evaluating the LLM-generated text is still far from being solved due to subjective criteria for the desired outputs. A further example is topic modeling, where the target itself (i.e. the topic distribution in a document) is a latent variable to be estimated before it can be associated with additional covariates.
