Concepts Through Time: Tracing Concepts In Dutch Newspapers Discourse (1890-1990) Using Word Embeddings

Concepts Through Time: Tracing Concepts In Dutch Newspapers Discourse (1890-1990) Using Word Embeddings

Wevers
Melvin

University of Utrecht, The Netherlands
m.j.h.f.wevers@uu.nl

Kenter
Tom

University of Amsterdam, The Netherlands
tom.kenter@uva.nl

Huijnen
Pim

University of Utrecht, The Netherlands
p.huijnen@uu.nl

2014-12-19T13:50:00Z

Paul Arthur, University of Western Sidney

Locked Bag 1797
Penrith NSW 2751
Australia
Paul Arthur

Converted from a Word document

DHConvalidator

Paper

Long Paper

word embeddings
semantic spaces
digital cultural history
Americanization
Discourse analysis

corpora and corpus activities
philosophy
semantic analysis
text analysis
content analysis
cultural studies
semantic web
data mining / text mining
English

In this paper, we use a new technique, called Concepts Through Time (CTT), to trace concepts in newspaper discourse. CTT makes use of sequential semantic spaces to follow semantic shifts of concepts diachronically. The semantic spaces are based on the extensive digitized newspaper corpus of the Dutch National Library.
1 As part of our approach, concepts are modeled as words related to each other in a semantic space. A key aspect of CTT is that the vocabulary used in the debate on a concept may change over time. In fact, the set of words used to discuss a particular concept might not show any overlap at all between different periods of time. As such, our method bears a resemblance to the rhizomatic structure as described by philosopher Gilles Deleuze. The conceptual implication of this method approximates the traditional scholarly methods of conceptual history. We test our method through three case studies: consumerism, globalization, and economic models. These domains are inextricably linked to the manifestation of modernization and Americanization within Dutch public discourse throughout the 20th century.
2 Tracing the genealogy of these manifestations helps us rethink the variety of meanings of modernization and Americanization in the Netherlands during this period. Essential to the development and success of this method is that, for every step of the process, decisions have been motivated from both the perspective of computational linguistics and cultural history. This approach makes this research project a genuine collaborative digital humanities effort.
3

Traditionally, the field of conceptual history has tried to combine micro and macro perspectives by studying the emergence and transformation of concepts, ideas, and thoughts over larger periods of time. Historians have increasingly used digital tools for the purposes of conceptual history. However, researchers in these studies regularly employed pre-defined and ahistorical definitions of concepts. Full-text search and n-gram viewers, for example, require a workable definition of a concept or a range of words that cover the subject in order to, subsequently, analyze them within certain contexts and periods.
4 The necessity of pre-defining terms is a serious drawback of working with these tracking tools. The research done in this way runs the risk of ahistoricity. The same goes for top-down approaches, in which a specific language model allows for the recognition of specific semantic information, such as entities via Named Entity Recognition (Grover et al., 2008; van Hooland et al., 2013) or via word classification lists (Klingenstein et al., 2014). Topic modeling approaches partly circumvent this limitation by modeling groups of words rather than single words. Topic modeling tools such as Mallet and TMT allow users to find topics in a corpus of texts without having to provide the algorithm with any prior information.
5 With approaches such as ‘Topics over Time’, researchers could also track topics diachronically (Wang and McCallum, 2006). Other research projects have used topic models to develop contextualized dictionaries that helped, e.g., to define the concept of ‘strikes’ in 20th-century Netherlands (Bogers and Van den Bosch, 2008) or the Chicago School of ‘neoliberalism’ in postwar Germany (Wiedemann et al., 2014).

The methods described above all share the limitation that concepts—whether defined by hand or automatically—are static. Either the user provides a fixed list of words or a fixed list of topics is inferred from the data in these topic-tracking approaches. However, historians commonly use texts to get a grip on the continuities and discontinuities throughout time. This involves the recalibration of their hermeneutical framework as they go along—that is, the list of words they work with. We propose that digital tools should also be able to provide argumentative evidence for such a methodology in constructing historical narratives.
For this purpose, we introduce a new technique, called CTT (Concepts Through Time), that aims to resolve the deficiencies of earlier methods. The technique is explicitly developed to account for historical changes in the semantics of concepts, thereby approximating the way historians traditionally have worked within conceptual history. CCT enables historians to trace concepts over large periods without having to manually select appropriate terms for the entire time span and without being dependent on a fixed set of topics. This allows for a greater sensitivity to semantic changes and an increased interactive heuristic approach to concepts within their discursive context.
6
Methodologically, CTT is based on word embeddings, as inferred by a neural network trained on a large body of data, to monitor semantically related words. These are created using word2vec (Mikolov et al., 2013), a computational technique that does not depart from a top-down model of language. Rather, a semantic space is inferred from the input data. Word2vec uses a continuous bag-of-words or a skip-gram architecture in order to produce a multi-dimensional word-vector space. This space contains semantic and linguistic regularities that can be used for the analysis of discourse (Baroni et al., 2014; Wijaya and Yeniterzi, 20111). A positional shift within the vector space can been established as an indicator for chronological shifts on a semantic and syntactic level—for example, in the use of the words ‘gay’ and ‘cell’ (Kim et al., 2014).
7 This is not done using traditional means of analysis, such as part-of-speech tagging, but by merely observing the geometry in the vector space. However, the authors who have provided the mentioned examples have worked with single words of which the common meaning has shifted almost diametrically. We will be looking at clusters of words and shifts of meaning both large and small.

The data we use to trace concepts are the circa 500,000 newspaper issues between 1890 and 1990 that are available in the Dutch National Library’s digitized newspaper archive. We train subsequent semantic model spaces in which we trace groups of terms, rather than individual words, diachronically and synchronically by keeping track of semantic relations between terms per period. For this approach, no pre-established and fixed topic sets are needed. We will use a single seed set of terms, merely as an entry point into the cluster to then find semantically related words within the semantic model. We will automatically update the set of related words over time, utilizing the subsequent semantic spaces. A key aspect of this procedure is that the original seed words might disappear from the cluster of words over time.
The way our word embedding technique clusters words holds surprising similarities with philosopher Gilles Deleuze’s notion of the rhizomatic structure (Deleuze, 1987).
8 Important features of this structure are that it allows for multiple entry and exit points, and that the semantic meaning of a concept is determined through its ‘relations of exteriority’ (DeLanda, 2006, 10–11). Translated to our approach, the meaning of a term can be considered in terms of the geometric relation in a semantic space.
9 Starting from this theory, we aim to focus on the stabilization and destabilization of relational vectors between words, i.e., the emergence and disappearance of words within a subset, as well as the shifting position of words in relation to one another. The latter could be the effect of forces within the cluster or changing vector relations outside of the cluster. We argue that the semantic space allows us to analyze ‘the processes that historically produce the identity of a given whole, but also the processes [coding and decoding] that maintain that identity through time’ (DeLanda, 2006, 10). Using this approach, we hope to determine words central to a topic and avoid concept drift caused by ambiguous words.

The development of this tool stems from a close collaboration between information retrieval experts and cultural historians. Every single technical or methodological choice has been motivated such that it complied with both sound computational linguistic research and critical historical methods. Consequently, we have high hopes of the applicability of this technique in historical research. We will evaluate our approach through three historical use cases in the context of the historiographical debates on modernization and Americanization in the Netherlands.
10 In the first case, we will trace popular consumer goods, such as cigarettes, alcohol, and fast food. This case study sets out to show which consumer goods appeared in newspaper discourse in specific periods. Moreover, it might show us in what wider discursive contexts these consumer goods were discussed—e.g., as luxury goods or as unhealthy products. The second use case maps out businesses in newspaper discourse. This might shed light on processes of globalization in which local businesses are substituted by multinationals. The final use case deals with the notion of efficiency. The concept was introduced in the Netherlands after the First World War (Bloemen, 1988); we will trace the genealogy of this concept to see whether it was already present in earlier discourse without being denoted as ‘efficiency’. In our final paper, we will show a formalization of the search strategies used to trace concepts through time. Furthermore, we will reflect on the representativeness of the newspaper archive as well as the applicability of the method to other corpora.

Notes
1. This collection includes over half a million newspapers for the 1890–1995 period. See www.delpher.nl.
2. This paper is part of the research project Translantis, which looks into the role of the United States as a reference culture in Dutch public discourse. Therefore, the chosen case studies are related to issues of Americanization within the field of consumer society and economy.
3. Collaboration between computational experts and humanities scholars is an elemental part of the digital humanities. See Burdick et al. (2012), pp. 15–17.
4. This method was adopted to study eugenics by Huijnen et al. (2014).
5. For an explanation of topic modeling, see Blei and Lafferty (2009). For uses of topic modeling in historical research, see Newman and Block (2006); Mimno (2012); Blevins (2010); Wittek and Ravenek (2011).
6. The importance of context for historical research is eloquently expressed in Snickars (2012).
7. In Kim et al.’s paper, the authors show how ‘gay’ has shifted from the connotation with an emotion to sexuality, and ‘cell’ has shifted from the denotation of a prison cell to the cellular phone.
8. For a clear description of rhizomatic thinking and assemblage theory, see ‘Assemblage Theory’ (2010). An interesting implementation of Deleuze’s notion of assemblage and rhizome has been performed by DeLanda (2006).
9. Michel Foucault makes a similar point when he argues that the meaning of an expression cannot be simply deduced from syntactic and semantic meaning. Rather, one should look into the conditions in which expressions appear and change—in his words, ‘by the analysis of the relations between the statement and the spaces of differentiation, in which the statement itself reveals the differences’. See Foucault (2012) .
10. These use cases stem from the Translantis research project. See
www.translantis.nl and van Eijnatten et al., 2013.

Bibliography

‘Assemblage Theory’. (2010). Texas Theory Wiki, Assemblage Theory, http://wikis.la.utexas.edu/theory/page/assemblage-theory.

Baroni, M., Dinu, G. and Kruszewski, G. (2014). Don’t Count, Predict! A Systematic Comparison of Context-Counting vs. Context-Predicting Semantic Vectors. http://anthology.aclweb.org/P/P14/P14-1023.xhtml.

Blei, D. M. and Lafferty, J. D. (2009). Topic Models.
Text Mining: Classification, Clustering, and Applications,
10: 71.

Blevins, C. (2010). Topic Modeling Martha Ballard’s Diary.
Historying, 1 April, http://historying.org/2010/04/01/topic-modeling-martha-ballards-diary/.

Bloemen, E. (1988).
Scientific Management in Nederland, 1900–1930. Doctoral thesis, Leiden.

Bogers, T. and Van den Bosch, A. (2008). Recommending Scientific Articles Using Citeulike. In
Proceedings of the 2008 ACM Conference on Recommender Systems. ACM, pp. 287–90, http://dl.acm.org/citation.cfm?id=1454053.

Burdick, A., et al. (eds). (2012).
Digital Humanities. MIT Press, Cambridge, MA.

DeLanda, M. (2006).
A New Philosophy of Society: Assemblage Theory and Social Complexity. Continuum, London.

Deleuze, G. (1987).
A Thousand Plateaus: Capitalism and Schizophrenia. Minneapolis: University of Minnesota Press, pp. 6–23.

Foucault, M. (2012).
The Archaeology of Knowledge [1974]. Random House, London.

Grover, C., et al. (2008). Named Entity Recognition for Digitised Historical Texts. In
LREC, http://ltg.ed.ac.uk/np/publications/ltg/papers/bopcris-lrec.pdf.

Huijnen, P., et al. (2014). A Digital Humanities Approach to the History of Science. In Nadamoto, A., et al. (eds),
Social Informatics, Lecture Notes in Computer Science 8359. Berlin: Springer, pp. 71–85.

Kim, Y., et al. (2014). Temporal Analysis of Language through Neural Language Models.
arXiv:1405.3515 [cs], 14 May, http://arxiv.org/abs/1405.3515.

Klingenstein, S., Hitchcock, T. and DeDeo, S. (2014). The Civilizing Process in London’s Old Bailey.
Proceedings of the National Academy of Sciences,
111(26) (1 July): 9419–24, doi:10.1073/pnas.1405984111.

Mikolov, T., et al. (2013). Distributed Representations of Words and Phrases and Their Compositionality.
Advances in Neural Information Processing Systems, http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf.

Mimno, D. (2012). Computational Historiography: Data Mining in a Century of Classics Journals.
Journal on Computing and Cultural Heritage,.
5(1) (April): 3:1–3:19.

Newman, D. J. and Block, S. (2006). Probabilistic Topic Decomposition of an Eighteenth-Century American Newspaper.
Journal of the American Society for Information Science and Technology,
57(6): 753–67.

Snickars, P. (2012). If Content Is King, Context Is Its Crown.
VIEW Journal of European Television History and Culture,
1(1) (21 February 21): 34–39.

van Eijnatten, J., Pieters, T. and Verheul, J. (2013). Big Data for Global History: The Transformative Promise of Digital Humanities.
BMGN—Low Countries Historical Review,
128(4) (16 December): 55–77.

van Hooland, S., et al. (2013). Exploring Entity Recognition and Disambiguation for Cultural Heritage Collections.
Literary and Linguistic Computing, 29 November, http://llc.oxfordjournals.org/content/early/2013/11/29/llc.fqt067.

Wang, X. and McCallum, A. (2006). Topics over Time: A Non-Markov Continuous-Time Model of Topical Trends. In
Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, pp. 424–33, http://dl.acm.org.proxy.library.uu.nl/citation.cfm?id=1150450.

Wiedemann, G., Niekler, N., et al. (2014). Document Retrieval for Large-Scale Content Analysis Using Contextualized Dictionaries. In
Terminology and Knowledge Engineering 2014, http://hal.archives-ouvertes.fr/hal-01005879/.

Wijaya, D. T. and Yeniterzi, R. (2011). Understanding Semantic Change of Words over Centuries. In
Proceedings of the 2011 International Workshop on DETecting and Exploiting Cultural diversiTy on the Social Web. ACM: 35–40, http://dl.acm.org/citation.cfm?id=2064475.

Wittek, P. and Ravenek, W. (2011). Supporting the Exploration of a Corpus of 17th-Century Scholarly Correspondences by Topic Modeling.
http://bada.hb.se/handle/2320/9689.