In this publication in the Workshop on Argument Mining, Julia Romberg develops a method to incorporate human perspectivism in machine prediction. The method is tested on the task of argument concreteness in public participation contributions.
Abstract
Although argumentation can be highly subjective, the common practice with supervised machine learning is to construct and learn from an aggregated ground truth formed from individual judgments by majority voting, averaging, or adjudication. This approach leads to a neglect of individual, but potentially important perspectives and in many cases cannot do justice to the subjective character of the tasks. One solution to this shortcoming are multi-perspective approaches, which have received very little attention in the field of argument mining so far.
In this work we present PerspectifyMe, a method to incorporate perspectivism by enriching a task with subjectivity information from the data annotation process. We exemplify our approach with the use case of classifying argument concreteness, and provide first promising results for the recently published CIMT PartEval Argument Concreteness Corpus.
Key findings
- Machine learning often assumes a single ground truth to learn from, but this does not hold for subjective tasks.
- PerspectifyMe is a simple method to incorporate perspectivism in existing machine learning workflows by complementing an aggregated label with a subjectivity score.
- An example of a subjective task is the classification of the concreteness of an argument (low, medium, high), a task whose solution can also benefit the machine-assisted evaluation of public participation processes.
- First approaches to classifying the concreteness of arguments (aggregated label) show an accuracy of 0.80 and an F1 value of 0.67.
- The subjectivity of concreteness perception (objective vs. subjective) can be predicted with an accuracy of 0.72 resp. an F1 value of 0.74.
Publication
Romberg, Julia (2022, October). Is Your Perspective Also My Perspective? Enriching Prediction with Subjectivity. In Proceedings of the 9th Workshop on Argument Mining (pp.115-125), Gyeongju, Republic of Korea. Association for Computational Linguistics. https://aclanthology.org/2022.argmining-1.11