Author: Tobias Escher

Overview of Methods for Computational Text Analysis to Support the Evaluation of Contributions in Public Participation

In this publication in Digital Government: Research and Practice Julia Romberg and Tobias Escher offer a review of the computational techniques that have been used in order to support the evaluation of contributions in public participation processes. Based on a systematic literature review, they assess their performance and offer future research directions.

Abstract

Public sector institutions that consult citizens to inform decision-making face the challenge of evaluating the contributions made by citizens. This evaluation has important democratic implications but at the same time, consumes substantial human resources. However, until now the use of artificial intelligence such as computer-supported text analysis has remained an under-studied solution to this problem. We identify three generic tasks in the evaluation process that could benefit from natural language processing (NLP). Based on a systematic literature search in two databases on computational linguistics and digital government, we provide a detailed review of existing methods and their performance. While some promising approaches exist, for instance to group data thematically and to detect arguments and opinions, we show that there remain important challenges before these could offer any reliable support in practice. These include the quality of results, the applicability to non-English language corpora and making algorithmic models available to practitioners through software. We discuss a number of avenues that future research should pursue that can ultimately lead to solutions for practice. The most promising of these bring in the expertise of human evaluators, for example through active learning approaches or interactive topic modelling.

Key findings

  • There are a number of tasks in the evaluation processes that could be supported through Natural Language Processing (NLP). Broadly speaking, these are i) detecting (near) duplicates, ii) grouping of contributions by topic and iii) analyzing the individual contributions in depth. Most of the literature in this review focused on the automated recognition and analysis of arguments, one particular aspect of the task of in-depth analysis of contribution.
  • We provide a comprehensive overview of the datasets used as well as the algorithms employed and aim to assess their performance. Generally, despite promising results so far the significant advances of NLP techniques in recent years have barely been exploited in this domain.
  • A particular gap is that few applications exist that would enable practitioners to easily apply NLP to their data and reap the benefits of these methods.
  • The manual labelling efforts required for training machine learning models risk any efficiency gains from automation.
  • We suggest a number of fruitful future research avenues, many of which draw upon the expertise of humans, for example through active learning or interactive topic modelling.

Publication

Romberg, Julia; Escher, Tobias (2023): Making Sense of Citizens’ Input through Artificial Intelligence. In: Digital Government: Research and Practice, Artikel 3603254. DOI: 10.1145/3603254.

Expert evidence: State of research on opportunities, challenges and limitations of digital participation

As set out in the German Site Selection Act (StandAG), the Federal Office for the Safety of Nuclear Waste Management (BASE) is charged with the comprehensive information and participation of the public in regards procedure for the search and selection of a repository site for the final disposal of high-level radioactive waste. In this context, in February 2022 BASE commissioned an expert report on the “Possibilities and limits of digital participation tools for public participation in the repository site selection procedure (DigiBeSt)” from the Düsseldorf Institute for Internet and Democracy (DIID) at Heinrich Heine University Düsseldorf in cooperation with the nexus Institute Berlin. For this purpose, lead by Tobias Escher a review of the state of research and current developments (work package 2) was prepared has been summarised in a detailed report (in German).

Selected findings from the report are:

  • Social inequalities in digital participation are mainly based on the second-level digital divide, i.e. differences in the media- and content-related skills required for independent and constructive use of the internet for political participation.
  • Knowledge about the effectiveness of activation factors is still often incomplete and anecdotal, making it difficult for initiators to estimate the costs and benefits of individual measures.
  • Personal invitations have been proven to be suitable for (target group-specific) mobilisation, but the established mass media also continue to play an important role.
  • Broad and inclusive participation requires a combination of different digital and analogue participation formats.
  • Participation formats at the national level face particular challenges due to the complexity of the issues at stake and the size of the target group. Therefore, these require the implementation of cascaded procedures (interlocking formats of participation at different political levels) as well as the creation of new institutions.

Publication

Lütters, Stefanie; Escher, Tobias; Soßdorf, Anna; Gerl, Katharina; Haas, Claudia; Bosch, Claudia (2024): Möglichkeiten und Grenzen digitaler Beteiligungsinstrumente für die Beteiligung der Öffentlichkeit im Standortauswahlverfahren (DigiBeSt). Hg. v. Düsseldorfer Institut für Internet und Demokratie und nexus Institut. Bundesamt für die Sicherheit der nuklearen Entsorgung (BASE). Berlin (BASE-RESFOR 026/24). Available online https://www.base.bund.de/DE/themen/fa/sozio/projekte-ende/projekte-ende.html .

CAIS Working Group: AI in digital public participation

As participants in a workshop organised by the Center for Advanced Internet Studies (CAIS) in Bochum, Julia Romberg and Tobias Escher presented results of the CIMT research on AI-supported evaluation of participation contributions and discussed further possibilities for using artificial intelligence to support public participation with experts from research as well as participation practice. It became clear that the practitioners see potential not only in the evaluation (output), but also in the activation of participants (input) and in the support of interactions (throughput) in participation processes. Nevertheless, these potentials face challenges and risks, including the adequate technical implementation and ensuring data protection and non-discrimination.

The workshop was organised by Dr Dennis Frieß and Anke Stoll and took place from 8 to 10 February 2023 in Bochum. Further information can be found on the website of the Düsseldorf Institute for Internet and Democracy.

3rd workshop for practitioners on first results from surveys in case study municipalities

On 30 November we invited representatives of the municipalities with whom we cooperate in order to discuss the first results of the extensive surveys conducted by our research group. The focus was on the question of how the respective participation procedures are assessed by those participating and which aspects motivate or discourage such participation.

Despite the diversity of the five projects we examined (and the still small number of participants), the assessments of the people participating in such processes show a relatively high degree of agreement. Overall, the evaluations of the participation processes are rather positive with regard to the course of discussion and transparency. At the same time, however, there are also comparable challenges in all processes. For example, the representation of one’s own interests is rated as relatively good, but gaps in the representation of other opinions are perceived. Also, a balance of interests is not always achieved. Furthermore, the participants are rather sceptical about the actual impact of the participation results on the political process, even though they still deem such an impact possible.

There is more information available in German.

2nd workshop for practitioners on automated text analysis for citizen contributions

Part of the efforts of the research group is to develop tools that support the evaluation of citizen contributions from participation processes. On 10 December 2021 the research group hosted a workshop with practitioners (including local planning officials, participation officers and planning experts) to discuss our recent developments, part of which have been published in the Proceedings of the 8th Workshop on Argument Mining.

More information on the insights from the workshop is available in German.

Interdisciplinary course on exploring social status and language

This term we are offering a master course in which we use proposals from online consultation processes in conjunction with individual-level survey data to analyse if social status of participants is reflected in the language they use in their written proposals. To this end, we utilize AI-based methods of Natural Language Processing.

More information is available in German.

(Opportunities to) Mobilise for local political online participation

In this article in the Zeitschrift für Politikwissenschaft, Bastian Rottinghaus and Tobias Escher explore the question of who does (not) participate in digital participation formats for local mobility-related planning and to what extent personalised invitations can contribute to the mobilisation of a larger and more diverse group of participants.

Summary

Consistent findings of unequal political participation have been motivating different democratic innovations, including those that utilize the opportunities of information and communication technologies for political online participation. While previous research has established only a limited mobilizing potential of digital media, we still lack a good understanding of the mechanisms leading citizens to decide for or against engagement online. Therefore, we investigate who participates in opportunities for political online participation, what explains (non)-engagement and how effective personalized invitations are to increase and diversify participation. To address these questions, we conducted a comparative study of three almost identical instances of local online participation, relying on evidence from surveys of registered users and random samples of the local population.

Our results show that engagement in online participation is indeed significantly biased from the population towards resource-rich individuals who also differ in their assessment of the participation process and its results. This is despite the fact that knowledge of these participation opportunities is equally distributed among all social groups. While online aversion is a barrier for some, distrust in the participation process and lack of interest are more powerful reasons to refrain from engagement. Using a randomized-controlled field experiment we can confirm that personalized invitations are an effective instrument for mobilization that increased participation by a factor of four to seven and that can to a limited degree elicit participation from under-represented groups. These findings have a number of important implications for researchers and practitioners who aim to increase equality in political participation.

Key Findings

  • In 2017, largely identical online participation processes were carried out and analysed in three cities in North Rhine-Westphalia. In Bonn, Cologne-Ehrenfeld and Moers, the population was invited to submit suggestions for improvements to cycling on an online platform.
  • Initially, the usual participation patterns emerged, with above-average participation by highly educated and middle-aged men. One of the main reasons for participation was dissatisfaction with the cycling infrastructure.
  • A lack of knowledge about the participation process is the main reason for not participating. Our findings could show that all population groups were equally-well informed about the process, but in the end resource-rich groups were significantly more likely to decide to participate. Furthermore, there are some specific reservations about the online format, which represent an obstacle to online participation.
  • As part of a controlled field experiment, personalised letters were sent to a random selection of citizens with an invitation to the participation process. It was found that this increased participation by a factor of four to seven.
  • This invitation also mobilises additional groups that are otherwise under-represented in the consultation process. This applies, for example, to women and people with a lower level of formal education. They also have slightly different attitudes to ‘the usual suspects’ in that they are somewhat less critical of the existing transport infrastructure, but are also less positive about the results of the participation process.
  • Overall, this shows that personal invitations are an important means of mobilising a larger and more diverse group of citizens to participate in consultation processes, but they cannot eliminate the fundamental under-representation of people with fewer resources and political interests.

Publication

Rottinghaus, B., & Escher, T. (2020). Mechanisms for inclusion and exclusion through digital political participation: Evidence from a comparative study of online consultations in three German cities. Zeitschrift für Politikwissenschaft, 30(2), 261–298. https://doi.org/10.1007/s41358-020-00222-7

Kick-Off Conference of Junior Research Groups in Bonn

On 9 & 10 March 2020 the kick-off conference of all junior research groups took place that are funded by the BMBF programme on Research for Sustainable Development (FONA). At the moment there are about 20 junior research group that receive funding and that reported during the two-day conference.

As part of the latest funding round of 2019 we presented our group in the form of a presentation and a poster.

For an overview of all junior research group see the BMBF website.