Rare N-Grams, Victorian Drama, and Authorship Attribution

Rare N-Grams, Victorian Drama, and Authorship Attribution

Hoover
David L.

New York University, United States of America
david.hoover@nyu.edu

2014-12-19T13:50:00Z

Paul Arthur, University of Western Sidney

Locked Bag 1797
Penrith NSW 2751
Australia
Paul Arthur

Converted from a Word document

DHConvalidator

Paper

Long Paper

n-grams
authorship attribution
Victorian drama

corpora and corpus activities
literary studies
stylistics and stylometry
text analysis
authorship attribution / authority
genre-specific studies: prose
poetry
drama
english studies
English

The use of rare n-grams in authorship attribution has generated considerable recent commentary and polemic. Vickers argues for the expansion of Thomas Kyd’s canon on the basis of 3- to 6-grams not found elsewhere in plays produced during his lifetime but shared by Kyd’s and some anonymous plays (2008). Although Jackson (2008) challenged his method, Vickers has repeated and expanded his claims (2009), arguing that Middleton did not adapt
Macbeth (2010) and that Shakespeare added about 2,600 words to Kyd’s
The Spanish Tragedy (2012). He has also produced a wide-ranging polemical dismissal of computational stylistics in general and the most frequent words in particular (2011). For responses to Vickers’ flawed critique, see Burrows (2012) and Antonia et al. (2014). Here I test his method on Victorian drama.

I have shown that the method is not reliable for similar-sized corpora of late-19th-century novels and modern American poetry (Hoover, 2011; 2012). Vickers has been insufficiently systematic and has not checked his method adequately on texts of known authorship. The huge number of individually rare n-grams ensures that many otherwise unattested n-grams show matches between any authorial corpus and many individual texts, whether those texts are by the same author or not. However, given that authorship attribution methods vary in accuracy with texts of various genres and lengths, it seems important to test Vickers’ method on drama, using a corpus that mimics the shape and size of his corpus of early modern plays. (Plays seem to present special difficulties for authorship attribution because they typically consist almost entirely of the spoken dialogue of multiple characters.)
Vickers emphasizes the unusual nature of the London drama scene in the late 1500s—where plays were being produced very rapidly and often collaboratively by a relatively small number of authors—as a reason to expect repeated n-grams to be especially appropriate as authorship markers (2008). He also usually limits his search for repeated n-grams to a restricted corpus of 55 to 75 plays produced during the productive lifetimes of his candidate authors. This initially seems reasonable, but it also inflates his chances of success and underestimates the likelihood of random matches not resulting from common authorship. Consider that ‘We know the titles of 1,500 plays performed between 1590 and 1642, of which only a few hundred survive . . .’ and that ‘the names of barely a dozen dramatists survive’ (Vickers, 2008, 13). Given these facts, rare n-grams found in one author’s often small and/or problematic corpus and an anonymous play but not in a restricted and incomplete reference corpus seem a dangerous indicator of authorship. We cannot know how many of these n-grams occurred in lost plays. Rare n-gram matches can support attributions or suggest the common authorship of anonymous plays, but they cannot provide the conclusive proof that Vickers seeks, claims, and uses to assert the unreliability of probabilistic methods based on frequent words or n-grams.
The small corpus and the uncertainty of dates, authorship, and collaboration, combined with substantial spelling variation, suggest that many early modern authorship questions are beyond confident solution. As noted above, rare n-grams have not proven effective for 19th-century fiction or modern American poetry, but extending the test to drama will provide a better comparison with Vickers’ method. I created a corpus of Victorian drama that mirrors his 55-play corpus (Vickers, personal communication) but without its drawbacks. I selected authors with enough plays available to match the authorial corpora of the nine known authors in Vickers’ corpus, and tried to match the lengths of the plays as closely as possible. For example, below are the seven plays by Henry Byron I use to match Vickers’ seven plays by Marlowe (sizes approximate):

I also selected nine plays by nine additional authors to match Vickers’ nine anonymous plays. (None of my plays is a collaboration or is of uncertain authorship, as some of Vickers’ unfortunately are, but about half my authors were involved in collaborations.) To overcome the limitations of Vickers’ corpus, I collected 20 more plays by 10 authors (one to three plays each), for testing. I used WordSmith Tools (Scott, 2012) to collect the 3- to 6-grams appearing at least twice in the 75 plays (about 173,000; n-grams appearing only once obviously cannot show matches). For my first test, I removed Buckstone’s six plays from the 55-play corpus, leaving 49 as the reference corpus. I tested a four-play Buckstone corpus against his two other plays and the 20 remaining plays by others by deleting all 3- to 6-grams present in the reference corpus (leaving 37,000), and then deleting all those absent from the Buckstone corpus. I calculated the number of matches between the remaining 6,500 Buckstone n-grams and the 22 test plays. The two remaining Buckstone plays have 363 and 241 n-gram matches with his corpus that are not found in the reference corpus (Vickers’ test). The 20 other plays have 59 to 254 such matches, and two have 241 or more, showing that plays by other authors can equal or outscore those by a candidate author. The results for Taylor (also with a six-play corpus, testing a four-play corpus against the remaining two Taylor plays and the 20 test plays) are worse: nine plays have more matches with Taylor than one of his own plays does. Although Vickers reports raw numbers of matches, as I have just done, the variation in length among his plays and mine makes this seem unwise, so Figures 1 and 2 graph the numbers of matches divided by text length for the Buckstone and Taylor tests above. Two other plays outscore both of Buckstone’s test plays, and three outscore the other (Figure 1). For Taylor, the results are a bit better, with only one play outscoring one of Taylor’s (Figure 2). These results do not, however, suggest a conclusive method.

Figure 1. N-grams absent from the reference corpus with matches between the Buckstone corpus and 22 plays.

Figure 2. N-grams absent from the reference corpus with matches between the Taylor corpus and 22 plays.
I created a larger reference corpus for further testing by collecting 45 more plays, for a total of 100—86 in 14 authorial corpora of two to 17 plays each, and 14 by 14 authors not otherwise represented in the corpus. To mirror the case of the 2,600 words added to
The Spanish Tragedy, I also collected two more full plays and 15 sections of 2,600 words by the 14 authors of the 86 plays (including at least one for each of the 13 authors with at least three plays), and eight sections of 2,600 words by eight additional authors. The goal was to test what might have happened had the corpus of early modern plays been larger and to see how sections by authors from outside the corpus compare to authors’ sections by authors in the set.

I again collected all the 3- to 6-grams appearing twice or more in the entire corpus (206,000), removed the Buckstone authorial corpus, and treated the remaining plays in authorial sets as the reference corpus. I deleted the n-grams found in the reference corpus, then retained only those found in a 6-play Buckstone authorial corpus and checked for matches between Buckstone and the remaining texts. I did the same for Taylor. The results can be seen in Figures 3 and 4.

Figure 3. N-grams absent from the reference corpus with matches between the Buckstone corpus and 39 other texts.

Figure 4. N-grams absent from the reference corpus with matches between the Taylor corpus and 40 other texts.
The Buckstone test succeeds, with all three of the 2,600-word sections of his test play as well as the entire play showing more matches with Buckstone than any of the other texts do, though it does not make a strong case for the certainty Vickers claims. The Taylor test fails badly: the whole Taylor play only barely outscores one other text, and the 2,600-word section trails behind 15 other texts. Tests on two of the other five authors with at least a six-play corpus are clearly successful, and tests on two fail badly. Clearly, in a real authorship test, this method will often produce strong erroneous attributions.
1

Finally, I tested twenty-one 2,600-word sections against all seven authors with at least a six-play corpus and against a composite ‘author’ comprising six plays by six authors (from among the original 20 test plays). Seven sections are by the seven authors and the remaining 14 by other authors; obviously, none is by the composite author. As Figure 5 shows, the tests for Bernard’s and Taylor’s sections fail; Bernard’s corpus is outscored by four other authors, including the composite one, and Taylor’s is outscored by two. The Phillips and Poole tests are very successful, and the correct author has the highest proportion of matches for the remaining three authors, but none of those cases is clearly distinguishable from the results for tests on sections not by any of the seven authors. And it is quite disconcerting that the composite author often scores first or second. Furthermore, these results are artificially successful because I limited each authorial corpus to exactly six plays. When larger authorial corpora are tested, the largest often outscores most or all of the other authors for most sections, showing the importance of the size of each author’s corpus to this method. As welcome as a conclusive method for authorship attribution would be, especially for relatively small amounts of text, the rare n-grams method seems unlikely to provide it.

Figure 5. N-grams absent from the reference corpus with matches between the eight authors and twenty-one 2,600-word sections of plays by 21 authors.
Note
1. Because of the unusual nature of the rare n-grams test, it is difficult to compare results using other methods. However, in a Delta test on authors in Vickers’s 55-text corpus, including only the seven authors with three or more texts and testing 30 texts as if they were unknown (21 whole plays and nine sections of about 2,600 words by one of the authors in the set), 23 texts are correctly attributed to their authors. Among the seven failures are four of the nine short sections. Curiously, there are three errors for Buckstone in the Delta tests, but none for Taylor, reversing the results of the rare n-grams test.

Bibliography

Antonia, A., Craig, H. and Elliott, J. (2014). Language Chunking, Data Sparseness, and the Value of a Long Marker List: Explorations with Word N-grams and Authorial Attribution.
LLC,
29(2): 147–63.

Burrows, J. F. (2012). A Second Opinion on ‘Shakespeare and Authorship Studies in the Twenty-First Century’.
Shakespeare Quarterly,
63(3): 355–92.

Hoover, D. L. (2011). Delta, Zeta, and Iota: An Ngrammatical Investigation.
Language Individuation: A Symposium in Honour of John Burrows, University of Newcastle, Australia, 4–8 July 2011.

Hoover, D. L. (2012). The Rarer They Are, the More There Are, the Less They Matter. Lecture, University of Hamburg, 19 July 2012.

Jackson, M. P. (2008).
Research Opportunities in Medieval and Renaissance,
47: 107–27.

Scott, M. (2012). WordSmith Tools version 6. Liverpool: Lexical Analysis Software.

Vickers, B. (2008). Thomas Kyd, Secret Sharer.
Times Literary Supplement,
18: 13–15.

Vickers, B. (2009). The Marriage of Philology and Informatics.
British Academy Review,
14: 41–44.

Vickers, B. (2010). Disintegrated: Did Thomas Middleton Really Adapt Macbeth?
Times Literary Supplement, 28 May, 13–14.

Vickers, B. (2011). Shakespeare and Authorship Studies in the Twenty-first Century.
Shakespeare Quarterly,
62: 106–42.

Vickers, B. (2012). Identifying Shakespeare’s Additions to
The Spanish Tragedy (1602): A New(er) Approach.
Shakespeare,
8: 13–43.