Abstract

• 1. Introduction
• 2. Basics: LLMs and their Application
• 3. Scholarly Editing as Processing Steps: Expectations ex ante
• 4. Experiments
• 5. Conclusions
  • 5.1 The Practical Perspective 1: Workflow Orchestration
  • 5.2 The Practical Perspective 2: Evaluation
  • 5.3 Theoretical Perspective 1: General Strengths and Weaknesses
  • 5.4 Theoretical Perspective 2: Bias and Ethical Issues
• Bibliography
• List of Figures

1. Introduction

[1]New technologies are leading to fundamental upheavals both in scientific practices and their epistemological foundations. Where else could this be a greater cliché than in the digital humanities? After machine learning and artificial intelligence have stood on the horizon for many years as ›the next big thing‹, and machine learning methods have been used in various very special fields of application in the humanities such as scholarly editing,‍[1] the revolution was finally proclaimed for the wider audience. With the publication of ChatGPT by OpenAI in November 2022, generative AI based on large language models (LLM) became so accessible and versatile that ›everyone‹ sooner or later started experimenting with it and developing visions for possible applications. Further developments from GPT-3.5 to GPT-4 and other foundation models (such as Claude, Bard / Gemini, Llama, Mistral) quickly followed, and applications (BingChat, NotebookLM, Perplexity etc.) were added in steadily growing competition.

[2]In response to these developments, scholarship in general and the digital humanities in particular are fundamentally called upon to examine the applicability of new technologies, to adopt and further develop them as tools for their own purposes. Applications of generative AI naturally include scholarly editing, a fundamental practice in humanities research. The Institute for Documentology and Scholarly Editing (IDE) has been stimulating discussions on digital editing through various initiatives since its establishment in 2006. Therefore, the IDE set out to investigate the applicability of generative AI in the field and to critically reflect on its potential for supporting editorial practices and processes at the intersection of literary studies, archival research, and textual criticism. To this end, a workshop was organized at the Passau 2024 annual conference of the Association for Digital Humanities in the German-speaking countries (DHd), which was intended to further structure the field via a call for experiments and bring together the actors who have already dealt with the possibilities and challenges of AI-based tools in digital editions at this early stage.‍[2] Specific use cases that move along the editorial workflow confronting LLMs with editorial materials and tasks were presented and discussed. Experimental tests for a set of given scenarios were examined with regard to their potential, limits, and shortcomings, as well as their ethical and theoretical implications.

[3]This article‍[3] draws on a workshop report to document tentative initial attempts – likely to have a limited lifespan – aimed at clarifying key aspects of applying generative AI to scholarly editing. The main aim is to continue the synthesis work that has already begun before the workshop with, for instance, first experiments in summer 2023 by Elisa Beshero-Bondar on performing text collation‍[4] or the XML annotation of unstructured text by Christopher Pollin, Christian Steiner, and Constantin Zach.‍[5] The questions developing from these experiences are: What are the areas of application for generative AI in the context of digital scholarly editing? How can it be used as efficiently as possible? What best practices are emerging? Where does it make sense to actively strive for further development? Where do the evolving practices converge in a common methodology? How do we address bias and ethical risks in AI systems – from the initial training data, through reinforcement learning by human feedback (RLHF), to the fine-tuning stages of the training data? How can the results be evaluated?

[4]In addition, an immediate, contemporary historiography of scientific development is particularly important in this area. In moments when new technologies arise, the contrast to previous technical possibilities is very sharp. These periods offer a high potential for fresh thinking, conceptual innovation, and critical reflection. They are the most dynamic times of upheaval, whose subsequent reflections provide a deeper insight into the shape and development of scientific practices. This is particularly evident in the numerous contributions at events focusing on LLMs and generative AI in the digital humanities, where the topic is omnipresent. The interest comes from all directions, including discussions on methodologies, legal aspects, ethics, pedagogy, editing, text and image processing, analysis, and more. An immediate documentation helps to prevent the oblivion of the discussions held and alternative paths of development that were still open at the time – a risk that is always inherent to the implementation of practices that eventually prevail. With this article, we want to anticipate the past of the future and write the history of a transitioning scholarship ›in real time‹ – when it was 2024.

2. Basics: LLMs and their Application

[5]AI-based applications such as OpenAI’s Generative Pre-trained Transformer (GPT) have demonstrated transformative capabilities in text processing. Their strength lies in their capability to be used in dialogue systems (chatbot), natural language text generation in general, the inclusion of both local context (specific to the current interaction or document) and ›world knowledge‹ (as patterns learned by the Transformer architecture from vast amounts of training data), multilingual support, and the possibility of fine-tuning to specific tasks.‍[6] They can assist humans in (as of 2024) developing coded algorithms for processing information, though their output requires verification and they serve as aids rather than independent developers of algorithmic solutions. Related to scholarly editing, they can be applied in many activities, from the recording and digitization of the historical tradition to the creation of deeply indexed, critically annotated forms of representation. Despite the diversity of subjects and approaches, we do have a reasonably consensual theoretical, methodological and technical common ground for digital scholarly editing. Due to the lively exchange on editions in recent years, the field has a good foundation for exploring the possible uses and effects of the new technology. The IDE has been actively engaged in shaping this exchange through the structured, reflective, and quality-assuring discourse of criticism and evaluation of editions as editorial research projects, most visibly through the establishment of the review journal RIDE (since 2014) and the catalog of criteria for evaluation of digital editions on which it is based.‍[7]

[6]Christopher Pollin’s introduction to the Workshop‍[8] established a common ground and clarity about the situation we are facing. After machine learning based methods of text generation had shown promise with OpenAIʼs GPT-2 and GPT-3.5 models, it was the release of GPT-4 in March 2023 that marked the true breakthrough. While GPT-3.5 had already attracted widespread attention from both scholars and laypeople, GPT-4 demonstrated unprecedented capabilities across a broad range of tasks. Though initial results often fell short of expectations, leading some to quickly abandon their work with GPT, others became fascinated and driven to understand how to achieve better results through experimentation and refinement. This is precisely where the workshop comes in, emphasizing the importance of prompt engineering, which has now almost developed into a discipline in its own right. Prompt Engineering is the process of designing and optimizing prompts to effectively communicate with an LLM. This is an iterative process, in which prompts are adjusted based on the model’s output. Crafting precise prompts improves the accuracy and controls bias of the response. Therefore, knowledge of the behavior and limitations as well as the difference in prompting techniques of the model used is crucial. Examples of key prompting strategies include (a) ›few-shot prompting‹, providing examples enabling in-context learning, (b) ›chain-of-thought‹, where the model is instructed to break down problems into step-by-step reasoning, (c) ›persona / expert prompting‹, where the model is assigned a specific role or expertise, and (d) ›reflection prompting‹, which encourages self-evaluation and refinement of responses.

[7]A study on the critical analysis of the correctness and improvement across various LLMs on the ATLAS and other datasets‍[9] had shown that GPT-4 outperforms other LLMs in both correctness and quality. It improves the correctness of responses by 75-85 % and the quality of responses by 40-80 %. In fact, the generic GPT-4 outperforms even fine-tuned LLMs. Despite all surprising idiosyncrasies of some LLMs, research is establishing LLMs as best practice for many NLP (Natural Language Processing) tasks.

3. Scholarly Editing as Processing Steps: Expectations ex ante

Fig. 1: General model of scholarly editing as knowledge production workflow: from source material to data and publication, through intermediate ›products‹ (middle), concrete steps (right), and more general challenges (left). [Pollin et al. 2025]

[8]The transformative potential of LLMs makes their integration into scholarly editing practices highly desirable. For which editorial use cases should we find the appropriate combination of generative AI based tools? We suggest describing scholarly editing as special scientific practice that includes recurring steps.‍[10] Figure 1 illustrates one potential graphical representation of this practice. The graphic can be read from top to bottom. The middle column shows three main components framed in purple – source material, edition as data, and edition as publication – linked by intermediate processing steps. From the source material, the pipeline proceeds through digital images, raw transcription, structured and annotated text, leading to the critical text. All these outputs are part of the edition as data, which is fed into a publication system and perceived through the user interface. This creates the edition as a publication. The left and the right column break the pipeline of objects down to tasks. The left column gives generic labels to structural tasks, while the right column gives concrete tasks related to this: Defining the research object and research goals of the edition, i.e. the ›edendum‹,‍[11] is a major practice in planning the edition. Editors document the textual transmission by creating descriptive metadata of the sources. They apply automatic text recognition methods, recognize named entities, link them to identifiers, and normalize the transcription. All these tasks, executed by digital tools, can be integrated into one workflow. When scholars add commentaries, describe named entities semantically, or translate texts, they incorporate knowledge external to the source material into the edition and enrich the information in the source. The conversion of printed editions into digital formats creates data that must meet quality standards as part of the edition’s output.

[9]This edition data is processed through a specific technology stack into a user interface, which should be implemented deliberately along with a user experience design. The published digital edition is reviewed through the external scholarly evaluation procedures and should include documentation as scholarly reflection and justification of its creation – and as support for re-use and long term preservation. These steps are certainly not the only way to conceptualize digital scholarly editing, but they help to integrate the concrete examples of application of generative AI to digital scholarly editing. None of the experiments presented later discuss edition planning, but several integrate the technology into the processing of source material into edition as data.

[10]The experiments presented at the workshop and described below can be related to the editorial steps and allow for a critical examination of generative AI’s application. We need to identify the particular strengths and potentials of the new tools within the tension between the need for problem-solving in science and the specific affordances of these tools for practical application. This includes the conversion of unstructured text (e.g., transcriptions) into structured text (via structural and semantic markup), the conversion of structured text into explicit data structures, named entity recognition, normalization and enrichment, context-specific annotations, error management, and data control. We are particularly interested in three complementary aspects: first, the integration of AI into the editorial workflow, including planning, design, and evaluation of digital editions as well as code development for web applications and user interfaces; second, prompt engineering for effective model interaction through natural language; and third, AI Engineering – encompassing technical enhancements like retrieval-augmented generation (RAG), vector databases, and tool integration to extend the base capabilities of LLMs.

4. Experiments

[11]While figure 1 is just one possible rough map of the field of editorial practices, the case studies we received in response to our Call for Experiments‍[12] show where first applications of generative AI try to tackle specific editorial tasks. The case studies can be organized along the typical editorial workflow, though not always in strict sequence:

• documentation of textual transmission,
• post-processing of retro-digitized editions,
• text establishment (transcription, Document Layout Analysis, OCR/HTR post-processing, basic markup),
• normalization,
• named entity recognition (NER) and deeper semantic annotation,
• information enrichment,
• translation, and
• summarization.

[12]The case studies whose discussion now follows are documented in Generative KI, LLMs und GPT bei digitalen Editionen‍[13] as single slide sets within the larger Zenodo package.

[13]Regarding the practice of documentation of textual transmission, the conversion of spreadsheets into TEI data appears to be an easily automatable task. However, Gerrit Brüning and Felix Schenke‍[14] demonstrate for the Goethe LYRIK project how much human understanding is needed to handle the documentation of the transmission of all known poems by Goethe that has been previously organized in a tabular format. These spreadsheets contain basic information about manuscript witnesses as well as poems within these witnesses alternating in consecutive rows but in shared columns. With TEI / XML as target format, the Oxygen XML Editor was used with the AI Positron plugin, providing access to the paid OpenAI API. It helps to execute not only standard tasks like summary, correction, and translation, but also conversions described by examples. The application of the prompt on manageable chunks facilitates human supervision and the creation of data, including identifiers designed to interlink the multitude of data fragments, which is extremely close to the planned target format and requires only minor refinement. Even though the TEI requires rather detailed descriptions and larger inputs are less stable, Brüning and Schenke demonstrate how an integration of AI-based activities in established tools such as Oxygen can be effectively used to handle and transform messy table data whose structure is not fully captured by the columnar format.

[14]Generative models offer two promising capabilities for manuscript research: the transformation of human-written queries for use in structured datasets, and access to semantic searches through vector embeddings of manuscript descriptions and metadata. However, these applications are still in early stages of exploration for scholarly editing. The application of generative AI to discovery service – as, for instance, proposed by Xu‍[15] – is under development in the library community, while the manuscript description community has not yet taken this up.

[15]Experiments in the testbed of retro digitization of printed scholarly editions show promising results regarding the evaluation and improvement of OCR output. In 2009, the University of St. Gall started to digitize the collection of Swiss Law Sources. The scanned books have been processed by OCR software and published in PDF on the SSRQ Online platform without correction or post-processing. At various points of the digitization workflow, Bastian Politycki‍[16] tested the application of generative AI models (GPT-3.5/4) to create an integrated OCR post-processing pipeline.‍[17] Politycki identifies five relevant areas: error correction of OCR results, structural TEI mark-up, annotation of dates and named entities, integration of data sets, and integration in the digital edition.

[16]Based on a training set of three volumes (two editions, one index), two different approaches have been compared. The first approach started from the PDF files, providing the respective edition texts and basic information about the layout and structure in order to train the correction of errors, the classification of segments as edition text, annotation and paratext, and the encoding of structural and semantic units. A zero-shot prompting approach has been refined by enriching the prompts through RAG and the provision of contextual knowledge.‍[18] Still, the results remained problematic, especially regarding text classification due to the unstructured nature of the OCR text and its prevalent character as print pages obscuring the actual document structure. The second approach starts from an HTML export of the PDF instead of plain text, adding further input such as titles, page numbers, dates, and item numbers from the table of content in XML format as well as layout information regarding typography of the original print. The pipeline combines a series of Python scripts with LLM few-shot prompts. The results of each step are validated by a set of tests. This provides significantly higher accuracy regarding both the recognized text and the classification of its sections and semantic units.

[17]For a long period, the label ›AI‹ was reserved in scholarly editing for the application of text transcription through automatic text recognition, particularly that associated with the activities on recognizing handwritten texts since 2015.‍[19] Although neural networks were already introduced into the scholarly editing discourse through an explorative paper by Daniele Fusi,‍[20] handwritten text recognition (HTR) was the first successful application of this technology. The successful application of recurrent networks to optical text recognition by Enrique Vidal’s team in the tranSkriptorium project (2013–2015) made large scale automatic transcription in ready-made tools available to every scholar. Generative AI applications in this field aim at specific subtasks, such as the creation of synthetic data for training purposes‍[21] or the reconstruction of damaged texts.‍[22] HTR is becoming a ›conservative‹ field, in which classical methods of deep learning (training and fine-tuning of existing models) dominate the research. Even if the recently discussed TrOCR methodology‍[23] is based on the same transformer architecture as the generative machine learning of large language models, it has not yet reached the same level of semantic richness and quality that enables advanced generative AI applications in text processing. However, HTR can be seen as a first example of multimodal enrichment of LLMs, combining the CLIP style prediction of text from images and vice versa.‍[24] The role of the implicit language models in the visual part of HTR are not yet studied in sufficient detail, although many pre-digital edition practices show the close connection between linguistic and paleographical interpretation made by editors when they ›normalize‹, for instance, punctuation or ›u‹ and ›v‹. The multimodal extensions of generic models and ›AI‹ services have not yet challenged the established HTR algorithms, but the developments in multimodal LLMs seem to be a starting point in doing so.‍[25] Transformer-based models play a well established role in post-OCR correction,‍[26] leading to recent tests with LLMs.‍[27] However, as demonstrated by Jacob Möhrke, Sandra Balck and Anna Ananieva in their workshop contribution‍[28], this approach carries a fundamental risk of ›over-normalizing‹,‍[29] discussed in detail below.

[18]Normalization of texts transcribed from historical documents to improve readability, searchability, and not least, the application of NLP tools, is yet another important step of the editorial workflow. Over the past decade, several tools and approaches for the post-processing and automated error correction of OCR-generated text have been developed based on reference materials, word lists or linguistic tools‍[30]. Two experiments tested the potential of LLM-based approaches to facilitate normalization of historical orthography. The first experiment, conducted by Yannic Bracke‍[31], presented the newly developed transnormer (sic) application (available as github repository). This application follows the basic concept of using an LLM that was pre-trained on a massive dataset, fine-tuned on a domain-specific dataset – specifically, a manually evaluated subset of the DTA corpus containing 5 million tokens of historical texts from 1780–1901. Intermediate results show a good accuracy (98.93 % word accuracy), encouraging enough to train more specific models for smaller historical time periods.

[19]The other experiment, presented by Kay-Michael Würzner and Robert Sachunsky‍[32], attempted to address the needs of three different user profiles: the corpus linguist with no interest in historical spelling, hyphenation or line breaks; the textual scholar with an interest in historical spelling without hyphenation or line breaks; and the OCR specialist with an interest in faithful and accurate transcriptions as training data.‍[33] In the first case, ChatGPT-4 can increase the quality of full texts derived from large digitization projects by addressing errors and ensuring consistent normalization. For the other two user profiles, ChatGPT was tasked with de-normalizing previously standardized text versions (e.g., from Wikisource or other digital libraries) to better align these texts with original historical documents. The denormalization attempts demonstrated relatively higher historical and material accuracy compared to standard normalization methods, particularly when guided by detailed prompts and supplemented with scans of the originals. Nevertheless, the resulting quality remained insufficient for training data purposes. Significant improvements were achieved through iterative prompting with more specific instructions and the inclusion of original document scans, though the outputs still did not meet the standards required for ground truth data.

[20]A major benefit of LLMs is their ability to process semantic relationships and contextual patterns in texts. Named Entity Recognition (NER) is a prominent application for generative AI, but it has to compete with established methods. Pia Schwarz, Florian Barth, and Lennart Keller‍[34] attempt to map academia in newspapers and compare SpaCy transformer-based fine-tuned models with OpenSource LLMs (Llama 2 13B, OpenOrca Platypus-2). Their results indicate the need for computational power, the length of context, and the risk of unsystematic and invented results.‍[35] It is well known that Named Entity Linking (NEL) is problematic for LLMs, which tend to invent formal identifiers. Inspired by the results of Tanti Kristianti and Laurent Romary‍[36] and Delpeuch,‍[37] the HIPE 2022 experiment could demonstrate that models trained on Wikidata or DBpedia sets are already successful in this task.‍[38] Consequently, they suggest fine tuning LLMs on structured data from Wikidata that includes explicit references to the identifiers. This comparative analysis reveals two critical requirements for effective LLM implementation in entity recognition tasks: integration with structured data sources to minimize hallucination of identifiers, and sufficient context window size to maintain accurate entity recognition and linking.

[21]Traija Nisha, Franziska Pannach, and Jörg Wettlaufer‍[39] tackle a more complex task involving NER and NEL: extracting information on the itineraries from travelogs.‍[40] This task needs deeper semantic understanding because the system has to distinguish between narrative on the travel itself and mentioning of other places. The corpus of Middle East travelers lists ca. 800 travelogs with ca. 180 full texts. While a previous study on grounding characters and places in the narrative‍[41] supports results by Schwarz, Barth, and Keller‍[42] that plain LLM based methods were then not yet performing better than a fine-tuned BERT model, the Göttingen team built two custom GPTs – specialized versions of GPT configured for specific tasks through custom instructions and knowledge bases – on OpenAI’s platform (Travelogue Annotator and Itinerizer) for few-shot prompting and including graphical user interfaces with a spatial display on a map.

[22]Jacob Möhrke, Sandra Balck, and Anna Ananieva‍[43] hypothesize that NER in historical texts can be reformulated as a text generation task using the inherent strengths of LLM.‍[44] Encouraged by experiments with modern language,‍[45] they created a custom GPT called DH Assistant to insert special characters marking named entities in a travelog by F. X. Bronner from 1810 / 1817. The GPT-based model performs text processing in three steps: decomposition, annotation, and recomposition. Untrained standard NER like flair‍[46] resulted in F1 scores of 0.18 for this multiclass annotation task. In a zero-shot-experiment, the custom GPT performed significantly better with a F1-score of 0.43, rising in both precision and recall. Their experiment also included normalization tasks for dates and currencies, in which the LLM identified problems to be solved that were not explicitly stated in the prompt. However, this ability also resulted in a hypercorrect output of the annotation task, where the reference text for the NER labels were normalized, although this task was not requested  – an effect which might be controlled in a prompt.

[23]Nina C. Rastinger‍[47] has experimented with GPT-3.5 in her project to extract information about people arriving in Vienna in the 18th century as documented in the Wiennerisches Diarium‍[48]. In a standard prompting scenario, she tested a one-shot-prompt to process these semi-structured lists. The results showed poor accuracy in identifying where named entities began and ended within the text, with the system generating 139 false entities, resulting in a ›hallucination quote‹ of 0.35 %. Most of these errors followed a ›hyper-correct‹ pattern we have also seen in other experiments: change in case, graphematics, and punctuation, but also semantic inferences, such as resolving a co-reference to the name of the entity. For historians, such cases would not necessarily count as false positives. Overall, the experiment shows potential, particularly in its good semantic referencing and the ability of the generic GPT model to adapt to the source material, despite using only few examples in the prompt.

[24]While the NER experiments show promising results in entity identification, they do not address the subsequent challenge of information enrichment through entity linking. Entity linking can be addressed with RAG, a method being applied in many industry tasks such as customer support or health care. In the workshop discussion, Andreas Kuczera reported that the Regesta Imperii team is leveraging their internal knowledge base stored in a Neo4j graph database for this task.

[25]Machine learning can also be part of a pipeline not using any of the LLMs used for generative AI applications. Carina Geldhauser, Ipek Tuncel, and Saahil Sundaresan‍[49] build an eScriptorium-based HTR pipeline to annotate nomina sacra in greek majuscule manuscripts, incorporating specialized XML markup. The application uses the implicit language model created by fine-tuning the kraken model, which uses an established deep learning architecture combining convolutional layers with long-short-term-memory (LSTM). Like Rastingerʼs approach, they enhance the system with manually curated lists while exploring how modern language processing can support this specialized task.

[26]Christopher Pollin‍[50] demonstrates the capabilities of LLMs in TEI annotation through a semantic markup of plain text letters. He used the Hugo Schuchardt Archive as a case study‍[51] to trace the evolution of a process for supporting the creation of digital editions of correspondences, progressing from simple ›bad prompts‹ to sophisticated workflows. Pollin’s experiment compares GPT-3.5 and GPT-4 models. A simple task description in a ›one shot‹ setting yields unsatisfactory results. It produces well-formed XML with some reasonable tagging but almost never valid TEI. However, a more sophisticated prompt including examples – a so-called few-shot approach – changes this significantly. Building on research in prompting techniques,‍[52] Pollin refines this further by using persona modeling (›You will act as a skilled expert automaton …‹), leveraging context (›...that is proficient in transforming unstructured text, specifically multilingual letters from or to Hugo Schuchardt (1842–1927), into well-formed TEI XML‹), assigning a clear task (›Analyze the provided text based on the mapping rules I have shared and then execute the transformation to produce TEI XML ensuring you adhere to the guidelines and only annotate if certain‹), and implementing few-shot prompting by adding examples for in-context learning. This approach is supported by TEI mapping rules that list elements for correspondence modeling along with their application, such as ›<salute> Salutations within the letter‹, and modeling instructions, such as ›Preserve the original text or produce well-formed TEI XML according to the TEI standards‹. Finally, emotional prompting‍[53] improves quality by adding urgency, such as ›This is very important for my career!‹.

[27]Comparing the two models, GPT-4 showed stronger ›reasoning‹ capabilities in ›understanding‹ task logic and followed TEI Guidelines more closely to produce valid XML. The model also successfully identified and normalized names and dates.

[28]Combining these findings with the best performing LLMs on reasoning tasks, including a human-in-the-loop and integrating external knowledge bases via RAG, Pollin proposes a workflow that combines several subsystems into an conceptual framework for transforming unstructured text into TEI-compliant XML documents – the underlying data structure required for digital scholarly editions. This follows the concept of interacting LLM-based agents invoked by other AI components.‍[54] Pollinʼs proposal considers interacting agents for analyzing visual features in digital facsimiles (GPT-4 Vision), adding TEI annotations to unstructured text (teiCrafter‍[55]), conceptualizing the target structure – the text model (teiModeler‍[56]) – and verifying the resulting TEI encoding (teiVerifier). Particularly in the modeling and verification steps, the system incorporates knowledge bases such as entity stores, schema descriptions, and glossaries. It delegates analytical processes and decisions to predefined subsystems called ›actions‹. These interacting systems need the human-in-the-loop to check the results, maintain communication, and feed further information into the system.

[29]The final step of an ideal and complete editorial workflow is the translation and summarization of the edited text. The experiment presented by Dominic Fischer, Martin Volk, Patricia Scheurer, and Phillip Ströbel‍[57] used ChatGPT to produce English translations of the original Latin letters of the reformer Heinrich Bullinger (1504–1575) as part of the Bullinger Digital edition project hosted at the University of Zurich. Compared to ›traditional‹ machine translation tools incl. Google Translate, results have been satisfactory. In particular, ChatGPT-3.5 and even more so GPT-4 are capable of dealing with text parts mixing German and Latin passages. Additional prompts introducing glossaries (such as ›Translate cesar as emperor‹) helped to secure accuracy for historical terms. Equally useful for the end user of a scholarly edition proved the results of prompting to summarize individual letters of the collection.

[30]As these experiments demonstrate, generative AI can significantly enhance various tasks within the editorial process. This observation has been further supported by experiments conducted since spring 2024. DeRose, in his paper at the Balisage conference 2024, explores how LLMs often disregard important markup and structural information during both training and output generation, emphasizing the implications of this limitation and proposing methods for improved integration and utilization of these features.‍[58] Lang, on the other hand, advocates for an editor-in-the-loop approach, where LLM-assisted corrections enhance the quality of Transkribus OCR outputs, enabling editors to efficiently review suggested changes using git diff.‍[59] Scholger, Strutz, and Pollin presented experiments on integrating LLMs into scholarly editing workflows, demonstrating experiments with GPT-4 and RAG for TEI encoding, handling editorial interventions, normalization, named entity recognition, and translation, using the example of the Austrian orientalist Joseph von Hammer-Purgstall.‍[60] At a conference on ChatGPT and generative AI in medieval research in September 2024, Schonhardt explored the LLM-assisted enrichment of medieval correspondence.‍[61] Abel and Pultar presented studies on converting indexes into knowledge graphs,‍[62] while Armbruster and Kuczera developed a RAG workflow for content processing.‍[63]

[31]The feasibility of integrating these applications into real editions has been demonstrated. These include prompt refinement, external knowledge bases for RAG, delegated subtasks feeding back into the main task (Actions), and, in particular, the human-in-the-loop guiding the AI system in a conversation. Comparing the experiments with an ideal set of tasks (see figure 1) demonstrates that more detailed work is still necessary to assess the possibilities to apply automatic text generators and AI in several areas relevant to digital scholarly editing. The definition of the subject matter (›edendum‹), for instance, can be part of the brainstorming use of generic Chatbots,‍[64] but can also include publically available manuscript and archival databases for RAG. Multimodal approaches can add context to an automated text recognition (ATR) process in context-specific definition of transcription methods, e.g., by describing more complex transcription practices for orthography and abbreviations. Textual criticism can use code creation to include existing deterministic tools like CollateX. A brief experiment‍[65] with the stemma codicum of Pier Aimone-Braida’s edition of Simon of Bisignano’s Summa Decretorum‍[66] demonstrates that text generators are capable of making informed individual suggestions on textual criticism because we can conceptualize these decisions as a combination of probabilities of a textual sequence and reasoning on textual tradition. RAG technologies were not as much in use in February 2024, but their application for entity linking tasks is obvious. The extraction of documentation from annotation practices constitutes a typical text generation task, constrained only by the text generatorʼs context window size. The use of code generation to set up technical systems for publishing has been demonstrated by Christopher Pollin in September 2024.‍[67]

5. Conclusions

[32]These experiments and the related research answer several of the questions asked in the introduction. The areas of application for generative AI in the context of digital scholarly editing are mostly allocated in the processing of source material into editions as data: automatic transcription, structural and semantic annotations, retro-conversion, normalization and translation. Knowledge in prompt engineering methods can increase their efficiency. While Christopher Pollin already suggested a deeper integration in early 2024, the scholarly community has not yet reached best practices, and in the following, we want to suggest some research lines to follow up from the experiments, trying to contribute to the development of a common methodology through a thorough evaluation of the methods tested. These proposals have to highlight biases and ethical risks.

[33]The global debates on the developments and trends observed in early 2024 can be considered under two perspectives: a practical (productive) and a theoretical (reflective) one, each partitioned into two sub-perspectives.