Stuck in the Middle: Developing Research Workflows for a Multi-scale Text Analysis

There is a serendipity to historical research; the slow, often piecemeal gathering of ideas over a lifetime of reading can create strange and wonderful connections between seemingly disparate facts and, by extension, lead to new and unexpected insights. Laurel Thatcher Ulrich alluded to this process in the 1997 documentary American Experience: A Midwife’s Tale. In a voiceover, she relates how one passage of Martha Ballad’s diary seemed particularly ‘out of place’ to her. 1 Later, she ‘began to do more research into Martha’s life and into the incidence of epidemics in the eighteenth century and suddenly it just all fell together in one place’; the connection between the birth of Martha’s daughter and the death of another child many years later was suddenly clear. This level of access to a historian’s analytical process was unusual for me. It did not appear in the book – which provided the conclusion but not the journey to it; had I not seen the documentary I might never have understood fully the nuance of her interpretation.2 I began to wonder: how many of my connections and assumptions do I make clear to my readers? Although commonplace in modern historical documentaries, Ulrich’s intertwining of the historical narrative with her narrative of discovery is not without criticism. Restricted by a processual framework, certain evidence can be unjustly foregrounded, distorting the analysis. Stylistically, it can be obtrusive or unnecessary. But is the invisible author any less problematic? However carefully we craft our methodologies, the explicit framework in which we place our evidence, our day-to-day experience of research, our workflow of discovery, is littered with innumerable, and often unconscious, choices on how to select, organize, contextualize and interrogate sources. Few, if any, of these choices are made explicit in the summative analysis. Is it, therefore, any wonder that two historians may come to radically different interpretations of the same evidence or that they may be at a loss as to understand the other’s conclusion? The use of digital methods and tools complicates matters even further. Most historians accept that any historical interpretation is shaped by the particular questions asked and is therefore only one of many possible conclusions that could be derived from a specific pool of evidence. In response, most have developed reading strategies to tease out these implicit aims or perspectives. Yet, these are less effective when critiquing digital analyses – a re-examination of the sources or an analysis of the author cannot always shed sufficient light on the workflow that shaped that analysis. Moreover, historiographical conventions have encouraged authors to exclude seemingly mundane activities, such as data cleaning and mark up, when describing their research. Yet the choices made during these activities, and the assumptions that underpin them, often have dramatic consequences on the final analyses. Large-scale corpora cannot simply be re-read to reverse-engineer the researcher’s point of view. If an interpretation is to be considered plausible, the author must leave behind not only a citation to the dataset used but detailed documentation of both their methodology and workflow, the specific choices made at each stage of gathering and processing, and the analysis of the data, as well as the rationale behind these choices. For example, a note that quantitative evidence was regularized is meaningless without knowing the software with which it was regularized, the specific changes made to the data by that software, and the stage of research in which that regularization took place. With this information, we can critique and refine the questions and assumptions made at every stage of our research and improve our understanding of the past. As an example of the minimum level of documentation required for a digitally augmented analysis, the following is a narrative of the workflow of the Scissors and Paste Project. 3 Before telegraphy, the dissemination of national and international news relied upon a global system of authorized and unauthorized copying, a process generally referred to as scissors-and-paste journalism. Scissors and Paste utilizes the British Library Newspapers collection to explore the possibilities of mining large-scale corpora for these reprinted texts. As news content was time-sensitive, and more directly descriptive than miscellany or literary content, it was often reprinted in an uninterrupted chain of high-fidelity copies. Research into these dissemination pathways has led to the development of a suite of tools and methodologies to identify reprint families and to suggest both directionality and branching within them. From these results, detailed analyses of additions, omissions and wholesale changes can be made, offering insights into the mechanics of reprinting that left behind few if any other traces in the historical record. Like many digital projects, the seemingly mundane choices made in obtaining, cleaning, processing and analysing the original dataset have affected my final conclusions. Moreover, because the analysis takes place at multiple scales, this project is particularly susceptible to hidden or non-intuitive workflow practices; by providing full documentation of my methodology on the project website, my interpretations can be better understood and refined as we develop a clearer understanding of this oft-discussed but rarely documented journalistic practice. As the size and number of digitization projects grow, researchers can theoretically access more information with the click of a mouse than their predecessors could in a lifetime of archival research. However, the inconsistent storage, dispersal and retrieval mechanisms for this information may negate many of the advantages of digitization. Particularly egregious are digital periodical and newspaper collections. Despite being limited by the same curatorial factors that affect all digital materials, these archives often give the mistaken impression of completeness.4 When hunting for specific, known documents, these problems are manageable, but when making an initial sampling of a resource, or when attempting large-scale distant reading, the limitations of these seemingly full-text repositories becomes clear. Despite these limitations, Scissors and Paste relies upon digitized databases to map the dissemination of newspaper reprints on a previously unachievable scale. Although historians have long known about scissors-and-paste journalism, systematically recording instances of reprinting would have been unviable before digitization.5 With the advent of full-text searching, articles and their reprints have been found across multiple databases using a combination of specific phrases and chronological or geographical filters.6 However, this method is both inefficient and unreproducible. With the notable exception of Trove (National Library of Australia), digitizers have produced the machine-readable (that is, searchable) versions of their newspaper collections using optical character recognition (OCR) rather than manual transcription. This process produces many errors that would effectively prevent reliable, full-text searching if a user’s queries were applied directly. For this reason, many searches allow, or automatically apply, an error tolerance to the search terms, allowing it to return possible matches even if the transcriptions were flawed. The exact methods by which a provider applies these imprecise, or fuzzy, search parameters are usually unknown; consequently, those attempting to replicate or build upon digital research cannot be sure that they will obtain the same selection of results, nor can the original researcher be sure he or she is obtaining representative samples in different databases. When working with collections indirectly through pre-defined search interfaces, this uncertainty can be partially mitigated by recording queries as well as results, that is, how you search, not just what you find.7 In this case, a single spreadsheet will often suffice – a table listing the database and the date of the search, the particular search parameters used, and the specific results obtained. This process of recording serves two important functions: for the researcher, it ensures that the specific keywords and limiting factors, such as date, geographical location, or title, are uniformly replicated in different sessions and in different databases; for those critiquing the work, it provides a greater understanding of the choices and assumptions that underpin the final analysis. Such tables do not guarantee that others can reproduce your search; they do, however, shed light on how the search was constructed and why it returned these specific results. If a conclusion appears unfounded, examining the search parameters may explain whether the researcher selected inappropriate parameters, the dataset was incomplete on the date of the search, or the search algorithms employed by the interface altered the query. Although this method may suffice when undertaking a close reading of a well-defined case study, it is unlikely to fulfil either function on a larger scale. For distant reading, direct access to a static, self-contained and well-documented dataset may be required. With this, the results of any computer-aided analysis can be placed in a specific context and, more importantly, the results can be fully replicated. Whether the dataset is obtained from a third-party or created by the researcher, what is important is that the specific version of the dataset is recorded and that the data themselves are securely and sustainably stored. Through the support of British Library Labs, Scissors and Paste was able to obtain and make use of this type of dataset, namely British Library Newspapers, Part I: 1800–1900.8 As a static collection of public domain data, with strong claims to geographical and political representativeness, this collection facilitated the development of well-defined and reproducible methodologies and results. The remainder of this article will discuss how to document the cleaning, processing and analysis of fully accessible and self-contained datasets. Digital analyses require a significant amount of data cleaning before any actual analysis can take place, with surveys suggesting that it can account for up to 80% of the total time expended on research, regardless of the source or nature of the data.9 Whether stored in tabular, graph, relational, hierarchical or long form, most databases will need to be at least partially transformed and this will affect any subsequent analyses; by focusing on the most relevant data or organizing it in the most appropriate format for the specific questions being asked, these transformations inevitably bias the shape, and possibly the content, of the final results. Moreover, no dataset is unimpeachable – experimental and survey data are subject to variances in their collection and all data are vulnerable to input errors, whether at the collection stage or a later point of data consolidation or compartmentalization. Documentation of the provenance of these transformed datasets places them in a clear context for later analysis. Like most researchers, my workflow began with the transformation of my dataset into a format that was compatible with my analytical framework. Due to the size of the collection, its manner of storage at the British Library, and the nature of the analysis I intended to undertake – comparing full pages of text against the entire corpus – the dataset could not be accessed or efficiently queried through an online Application Programming Interface (API). Instead, it needed to be transferred and stored on an external hard drive. Compressed in standard zip files, the dataset was roughly 250 GB. Once decompressed, it increased in size to 920 GB. Copies of the compressed and uncompressed data were stored on a single 3 TB external hard drive (allowing its use on several different workstations) while the original compressed data remained on a separate 500 GB hard-drive. The latter allowed me to return to the original data if the larger drive failed or the data became otherwise corrupted – a situation that occurred on two separate occasions owing to mechanical and software errors. The uncompressed data consisted of a collection of eXtensible Markup Language (XML) files, each containing the metadata and text for a single page within the digital newspaper collection. These files were contained in separate directories, derived from individual digitization batches rather than bibliographical details, although the individual filenames did indicate their contents. Because I was comparing documents based on date, this processual division was a hindrance. I merged the files into a single directory through a simple command line function – FOR /R x:\original %G IN (*.xml) DO move %G x:\new – wherein /R searches through the entire tree of subdirectories and %G stands in for the various filenames. Working on the command line was important considering the sheer size of the database; copying or moving these through the graphical user interface (Microsoft Windows 10) often resulted in the system becoming permanently unresponsive (hanging) during the enumeration process, which counted the files before they were moved. Once I had reorganized the data into a single directory, I could reformat them en masse. The XML files, as created through the optical character recognition (OCR) processes, contained a significant amount of metadata that was unnecessary for my analysis. Moreover, the text was formatted in a way that it could not be effectively parsed by standard plagiarism detection software. For example, instead of displaying ‘will take place in the PublicRooms’, the XML displayed the first few words of the page as: will take place in the PublicRooms Excusing the OCR error, which failed to include the space between ‘Public’ and ‘Rooms’, the real difficulty was the inclusion of coordinates for each word. These are used to highlight the text on the corresponding image of the page, but hindered my analysis by breaking up the text I was attempting to compare. Likewise, the metadata preceding each text were important for establishing the provenance of the file but irrelevant to my textual analysis. Therefore, by using eXentsible Style Language (XSL) scripts, I transformed the data from complex, metadata-rich XML into smaller plain-text files.10 These files were given human-readable names – Title_YY_MM_DD_Page.txt – and contained only the raw text. This transformation was done at the command line using the Saxon XSL converter – java net.sf.saxon.Transform -s:WO1_ANJO_1830_01_06-0001.xml -XSL:“x:\stylesheet\BL_TEXT.xsl” -o:“x:\plaintext\”. At this stage, the value of systematic data cleaning comes to the fore. It quickly became apparent that a significant, if seemingly random, number of XML files were corrupted, halting the transformation sequence. There were three key errors: incomplete files, files with corrupted characters, and files that resulted in duplicate human-readable filenames. The last of these was the simplest to correct. These were the result of multiple editions of certain pages. They had been given the prefix S and could be quickly moved into a subdirectory for separate processing. The first two faults, however, were not easily traceable as the errors were not consistent. Each time the process failed, the corrupted file was removed and the process restarted. Fewer than 500 of the two million XML files generated this type of error, but their random dispersal slowed the automation process. Because the errors were idiosyncratic, a list of excluded files has been created but they cannot, as yet, be reintegrated into the wider analysis. Nonetheless, these exclusions are recorded in detail, qualifying the current results and allowing the data to be reintegrated into the set at a later date. Once the all XML data had been transformed into plain-text files containing only the newspaper text, these were inputted into Copyfind, an open-source plagiarism detection software programme.11 Careful documentation of this latter process is required as finding ‘matches’ is a subjective process. What qualifies as a match? A certain number of identical words or characters? A general composition or narrative structure? In addition to making its source code publicly available, Copyfind also allows the user to specify variables such as phrase length, overall minimum word count, mismatch tolerance and the decision to ignore or enforce matching case, punctuation, and non-words. Subtle changes to these can produce very different results. For Scissors and Paste, the variables were set to relatively high matching requirements (at least 20 instances of ten-word phrases), with a very high tolerance for small character differences. This configuration, chosen through trial-and-error, lowered the possibility of false positives, those with phrases or while allowing for errors in the OCR As the size of the dataset required matching to be done in of these and the version of this software, analysis. Likewise, such in and qualifying the results, making clear the assumptions I made when Copyfind provides two of results. These two have two different and are therefore stored The first is a collection of files, and to for of the of any through a list of each within the The is a version of this The files were used to matches as well as to or for efficiently false The number of matches a of data namely a of research with the These researchers accessed the a and recorded their results in a this an of a random of the derived The set of results was the for the list of This single a list of was first transformed with a of the list of filenames into individual of This table was with an open-source for cleaning data, to the newspaper which over time and in different processing This table was through an open-source I created to the This final list was stored on which provides an of the results. It is from this rather than samples and that I my case Once a reprinted is found with at least three manual transcriptions of these articles are These are against the OCR database using Copyfind, new and searches are within other digitized newspaper collections. Once these digital search are a set of transcriptions are made and digitally the individual – errors and the of – are recorded as a data using an open-source Each word or mark is and given a value – or one – whether it appears in that version of the text. This can be used to create of the of these or to detailed evidence for analysis. At this my research into more dissemination and the documentation of my workflow provides a for my final questions about the of my or the of my data can be Although the are not always to my I can for known in a way that I could not for implicit connections between data and my wider Moreover, because I have made this documentation others can build upon or critique my work, in their understanding of how I my in my methodology or dataset can be and rather than my being as to other In the an individual research workflow is a century researcher be of the choices she or he when obtaining, and analysing their sources. Whether information was through an online search or by in a of these and choices allows for not only in conclusions but in the development of new research questions to in the Therefore, critique may digital historians with the most for to wider historiographical as well as being the most way for all historians to with the digital of was by the The research for this article can be accessed at The author would to the of the following researchers at and