"What's at stake when we accept or reject quantitative studies about Shakespeare?" by Gabriel Egan

Quantitative methods were much favoured by the New Bibliographers of the mid-twentieth century. Fredson Bowers and Charlton Hinman, and their followers, thought that by counting features in early Shakespeare editions they could calculate the size of the print run of a book and the order in which its formes were printed, and could distinguish the characteristic handiwork of individual typesetters (Egan 2010, 54-98). They were wrong, as decisively shown by D. F. McKenzie in the 1960s and 1980s and Pervez Rizvi in 2016 (McKenzie 1969; McKenzie 1984; Rizvi 2016). McKenzie and Rizvi did exactly what Bowers and Hinman had done, counting the same phenomena to show that the assumptions their predecessors used to make sense of the numbers were mistaken.

    Such a critique that comes from within is not what brought about the general collapse of the New Bibliography in the 1990s. It was attacked from premises entirely alien to it, and on essentially ideological not empirical grounds (Egan 2010, 129-66). Much of the critique that brought down the New Bibliography was built upon non-quantitative foundations laid by the same D. F. McKenzie in his claim that texts are socialized rather than individuals' constructions, which was most influentially taken up by Jerome J. McGann (McGann 1983; McGann 1985). These non-quantitative ideas appealed to Shakespearians of all kinds, whereas discussions about the mechanics of letter-press printing had a distinctly limited interest.

    In the last 20 years, new quantitative methods have become possible because of the availability of large bodies of digitized early modern text. Much of the new quantitative research on Shakespeare has addressed authorship attribution, but there have also been studies of literary style, genre, influence, and chronology. These topics attract interest from a wide range of Shakespearians and as with the critiques of New Bibliography it is useful to distinguish between those that use essentially the same methods as the studies they critique, and those that come from outside the quantitative subdiscipline. The tension between trends that arise from essentially ideological approaches and those from essentially quantitative ones can be seen in a recent development in theatre history.

    In the last 20 years, theatre history has turned away from the author to follow the direction set by McKenzie and McGann in book history: drama is treated as a collective, socialized form of creativity not an individual, authorial one. There has been a string of books taking the theatre company rather than the author as the focus of attention: Scott McMillin and Sally-Beth MacLean's study of The Queen's Men and their Plays (McMillin & MacLean 1998); Andrew Gurr's studies of the Chamberlain's/King's men (Gurr 2004) and the Admiral's men (Gurr 2009); Lucy Munro's study of the Children of the Queen's Revels (Munro 2005); Helen Ostovich, Holger Schott Syme, and Andrew Griffin's study of the Queen's men (Ostovich, Syme & Griffin 2009); Eva Griffith's study of the Queen's servants at the Red Bull (Griffith 2013); and Lawrence Manley and Sally-Beth MacLean's study of Lord Strange's men (Manley & Maclean 2014). To varying degrees, each of these books goes beyond a merely historiographical narrative of company formation, practice, and demise in order to assert that company identity itself was expressed in the repertory.

    We do not know if this is true. We do not know if the dramatic effects we are interested in--commonalities in writing style, dramatic themes, means of presentation, and so on--cohere around the theatre company rather than the author. We have good reason to think not. Hugh Craig and Brett Greatley-Hirsch's book Style, Computers, and Early Modern Drama tested the claim that individual acting companies had their own distinctive styles. Using 39 plays from across the theatre companies of the 1580s and 1590s, Craig and Greatley-Hirsch showed that computational tools successful in distinguishing authorial style, genres, and chronology can find no evidence for the existence of company style (Craig & Greatley-Hirsch 2017, 164-201).

    The assertion that company style existed is an empirical one, made a posteriori not a priori. It is incumbent upon those who maintain that it existed to show that it did, using only quantitative methods. It is not enough to merely show that certain features are recurrent across plays in one company's repertory, since the assertion is that these features cluster by company more significantly than they cluster by author. Finding that plays in the Chamberlain's/King's men's repertory have above-average occurrences of the words 'gentle', 'answer', and 'beseech', and below-average occurrences of the words 'yes', 'brave', 'sure', and 'hopes' would not count as evidence of company style since these are the known word preferences of the author Shakespeare (Craig & Kinney 2009, 16-18). Nearly half the plays in Roslyn Knutson's reconstructed repertory of the Chamberlain's/King's men, 39 of the 81, are by Shakespeare (Knutson 1991, 179-209), so authorship is potentially a confounding variable in the search for company style.

    I have drawn a distinction between critiques such as McKenzie's and Rizvi's that undermine quantitative analysis from within, by doing the same kind of quantitative work better than their predecessors--what we might call immanent critique--and critiques from without that reject the premises of their predecessors. Two recent review essays about computational approaches to literature have presented themselves as immanent critiques that take the subject on its own empirical, quantitative terms, but in fact both ultimately rest on theoretic, qualitative bases.

    In the first, "A Brief History of Stylometrics" published in English Literary History in 2015 (Kahan 2015), Jeffrey Kahan attempted to wield numbers to confound the numerical work in the studies he critiqued, but repeatedly stumbled in his arithmetic and accused others of miscounting what he had miscounted, as I have shown elsewhere (Egan 2017; Egan 2019). Kahan ultimately asserted that counting itself is the wrong way to go about discovering how collaborative writers put plays together and where the divisions of their labours fall. For Kahan, ". . . as intriguing as such topics are, they lend themselves to historical, not scientific inquiry", and the science of stylometry is "mired in a theory of knowledge designed solely to inquiry [sic] into natural phenomena" (Kahan 2015, 837).

    The second recent critique is Nan Z. Da's "The Computational Case against Computational Literary Studies" published in Critical Inquiry earlier this year. It made a stronger claim to be an insider's approach to the studies it critiques, "taking them on their own terms completely" (Da 2019, 604). These studies attempted to measure literary style by various quantitative means, tracking how it changes over time, how it varies between writers, and how within particular works style changes in step with other aspects of the writing, such as plot. Da's most arresting claim was that in their quantitative work Hugh Craig and his co-author Arthur F. Kinney took the view that Christopher Marlowe wrote the late works of Shakespeare having faked his own death in 1593 (Da 2019, 622n39). No one who has even skimmed Craig and Kinney's book Shakespeare, Computers, and the Mystery of Authorship (Craig & Kinney 2009) could think that, so it seems Da had not read it.

    Da's remarks on the technical details of Computational Literary Studies are full of errors. She wrote that studies in the field "are more or less all organized the same way, detecting patterns based in word count (or 1-, 2-, n-grams, a 1-gram defined as anything separated by two spaces)" (Da 2019, 605). Members of this audience can probably think of several studies not organized that way, and of course "anything separated by two spaces" is not how we define a 1-gram. Da claimed that all studies in this field "make arguments based on the number of times x word or gram appears" (Da 2019, 606). Again, it is easy to think of studies that do something else, including my own that measured the distances between words, not their frequencies (Segarra et al. 2016). Of the attempt to quantify the significance of one's results by distinguishing them from merely random variation, Da wrote that ". . . statistics automatically assumes that 95 percent of the time there is no difference and that only 5 percent of the time there is a difference. That is what it means to look for p-value less than 0.05" (Da 2019, 608). Such a misleading account of the meaning of p-value would earn no credit in an undergraduate essay on statistics.

    Most importantly, Da's use of quantitative methods was, like Kahan's, merely the cover for a different kind of critique. Computational methods, she wrote, are good for certain "industries, sectors, and disciplines" such as law, investment bankers (who data-mine newspaper reports), and email spam detection, but do not work for literary texts (Da 2019, 620). Dismissing an attempt to use topic modelling to reveal the trends in published literary criticism, Da concluded that "If the authors wanted to nonarbitrarily study the change in topics covered in journal articles over time, they could have saved time by just looking at journal abstracts" (Da 2019, 627). In case that sounds like too much reading for anyone to practically undertake, Da rightly pointed out that it also takes a long time to prepare texts for computational analysis. "Looking for, obtaining copyrights to, scraping, and drastically paring down" the texts, she wrote, "takes nearly as much, if not far more, time . . . than actually reading them" (Da 2019, 638). And the results from computational analyses are inferior to those from human readers. Dismissing a study that attempted to spot haikus from their stylistic features, Da asked: "Couldn't someone trained in poetry just find, read, and classify them?" (Da 2019, 637).

    According to Da, the reason that computational methods are good for legal documents, newspaper articles, and email and not for literary texts is that literature is fundamentally a different kind of writing. "Tagging errors and imprecision" in Natural Language Processing, she wrote, "do not sufficiently degrade the extraction of information in many other contexts, but they do for literature" (Da 2019, 636). Consider what Da claimed to have proven: that a set of methods that she agreed are demonstrably insightful in quantitative analysis of various kinds of non-literary writing--court records, journalism, emails, and so on--can be shown, using the very quantitative methods of their investigators, to fail when applied to that special class of writing called literature.

    If she was right, Da had discovered a classifier that can distinguish literary texts from other writings: literature is that class of writing for which the methods she described do not work. If Da was right then by her own logic she was also wrong, since she had shown that computational methods can meaningfully comment on literary texts, at the very least by reliably distinguishing them from other kinds of writing. I suspect that the truth is more mundane and that Da had simply failed to correctly replicate the studies she intended to undermine. The tedious but necessary work for the investigators she critiqued is to critique her critique using the same quantitative principles. Only that way will we find out who is right.

Works Cited

Craig, Hugh and Arthur F. Kinney. 2009. Shakespeare, Computers, and the Mystery of Authorship. Cambridge. Cambridge University Press.

Craig, Hugh and Brett Greatley-Hirsch. 2017. Style, Computers, and Early Modern Drama: Beyond Authorship. Cambridge. Cambridge University Press.

Da, Nan Z. 2019. "The Computational Case against Computational Literary Studies." Critical Inquiry 45. 601-39.

Egan, Gabriel. 2010. The Struggle for Shakespeare's Text: Twentieth-century Editorial Theory and Practice. Cambridge. Cambridge University Press.

Egan, Gabriel. 2017. 'Shakespeare Editions and Textual Studies, 2015': Not the Year's Work in English Studies, a Self-published Annual Review.

Egan, Gabriel. 2019. "Scholarly Method, Truth, and Evidence in Shakespearian Textual Studies." Shakespeare Survey 72. 150-59.

Griffith, Eva. 2013. A Jacobean Company and its Playhouse: The Queen's Servants at the Red Bull Theatre (C. 1605-1619). Cambridge. Cambridge University Press.

Gurr, Andrew. 2004. The Shakespeare Company, 1594-1642. Cambridge. Cambridge University Press.

Gurr, Andrew. 2009. Shakespeare's Opposites: The Admiral's Company 1594-1625. Cambridge. Cambridge University Press.

Kahan, Jeffrey. 2015. "'I'll Tell You What Mine Author Says': A Brief History of Stylometrics." English Literary History 82. 815-44.

Knutson, Roslyn Lander. 1991. The Repertory of Shakespeare's Company 1594-1613. Fayetteville. University of Arkansas Press.

Manley, Lawrence and Sally-Beth Maclean. 2014. Lord Stranges Men and Their Plays. Trans. F. W. Bowers. New Haven CT. Yale University Press.

McGann, Jerome J. 1983. A Critique of Modern Textual Criticism. Chicago. University of Chicago Press.

McGann, Jerome J., ed. 1985. Textual Criticism and Literary Interpretation. Chicago. University of Chicago Press.

McKenzie, D. F. 1969. "Printers of the Mind: Some Notes on Bibliographical Theories and Printing-house Practices." Studies in Bibliography 22. 1-75.

McKenzie, D. F. 1984. "Stretching a Point: Or, the Case of the Spaced-out Comps." Studies in Bibliography 37. 106-21.

McMillin, Scott and Sally-Beth MacLean. 1998. The Queen's Men and Their Plays. Cambridge. Cambridge University Press.

Munro, Lucy. 2005. Children of the Queen's Revels: A Jacobean Theatre Repertory. Cambridge. Cambridge University Press.

Ostovich, Helen, Holger Schott Syme and Andrew Griffin, eds. 2009. Locating the Queen's Men, 1583-1603: Material Practices and Conditions of Playing. Studies in Performance and Early Modem Drama. Aldershot. Ashgate.

Rizvi, Pervez. 2016. "The Use of Spellings for Compositor Attribution in the First Folio." Papers of the Bibliographical Society of America 110. 1-53.

Segarra, Santiago, Mark Eisen, Gabriel Egan and Alejandro Ribeiro. 2016. "Attributing the Authorship of the Henry VI Plays By Word Adjacency." Shakespeare Quarterly 67. 232-56.