Adapting a system for Named Entity Recognition and Linking for 19th century French Novels

The annotation and linking of Named Entities - be it People, Places, or other proper names - in novels is an important task both for the creation of quality Digital Editions as well as for Digital Literary Stylistics and Spatial Humanities, which largely rely on Distant Reading techniques involving among other things spatialisation (Gregory et al., 2015) and network analysis (Elson et al., 2010).
Automatic NERC (Named Entity Recognition and Classification) algorithms can be crucial in this sense, but they are often developed and tested on contemporary newspaper corpora and need to be adapted. As to NEL (Named Entity Linking), namely the task of referencing entities with links to external Knowledge Bases (KB), the additional issue arises of finding the correct reference base containing knowledge from the past.
Also, the question of formats is crucial; the Text Encoding Initiative (TEI) is the format of choice for most digital editions in the humanities, but the majority of existing Natural Language Processing tools are not suited to support this type of input and require specific formats such as CONLL, which weren’t built to preserve text structure.
In this poster, we describe a pipeline combining two different tools - SEM for NERC (Dupont, 2017) and REDEN for NEL (Brando et al., 2016; Frontini et al., 2016) - and its adaptation for the annotation of 19th century French novels. This work is set within the context of a larger initiative aiming at creating a “Time Machine” project for the city of Paris, following the example of the “Venice Time machine” (di Lenardo and Kaplan, 2015). The input of the pipeline is an XML-TEI edition and the output is an enriched version with tagged and identified Named Entities. REDEN provides also the necessary input for a dynamic cartography, based on information in the KB.

Figure 1 - The overall pipeline.

The corpus consists of the first two chapters of
Le Ventre de Paris

(Zola, 1873) and the first chapter of
Cesar Birotteau

(Balzac, 1837), the size is 30,889 words and both texts were annotated for places and fictional characters by two separate annotators; an inter-annotator agreement of 0,91 (F-measure) indicates a strong overall agreement. An Internationalized Resource Identifier (IRI) was added for those place names for which a reference existed in the KB.

SEM

is a machine learning system based on conditional random fields (CRF) models; its default French model for NER performs poorly on our corpus. We thus re-trained SEM using our gold standard, and evaluated the domain adaptation with two setups:

Setup1: training on the first chapter of Zola and tested on the second one

Setup2: training on the Zola subcorpus and tested on Balzac.

Setup 1
Setup 2

Precision
1
0,7

Recall
0,69
0,26

F-measure
0,82
0,38

Results for Setup 1 are quite encouraging: they show that the model trained on one chapter performs very well on another. This means it could  in principle suffice to manually annotate (or correct) one chapter to obtain a system that can annotate the rest of the novel with an acceptable accuracy. Results for Setup 2 show that even two novels that are relatively close from a temporal, geographical and stylistic point of view are sufficiently different to cause an important drop in performance when the model trained on one is applied on the other, with important consequences for our ongoing work to constitute an adapted NERC model for 19th century French novels.

REDEN

relies on graph-based algorithms and on Semantic Web technologies (see Figure 2), and like many NEL algorithms, is composed of two phases, candidate retrieval and disambiguation. It was designed for DH applications and is adaptable to various domains thanks to the possibility of using different KBs.

Figure 2 - The REDEN model.

For this experiment, REDEN was tested on the task of referencing
placeNames
, using three different Kbs (see table below), a complete description of the evaluation metrics used in this table, and which are commonly used for NEL, can be found in Brando et al., (2016). The best configuration in terms of overall accuracy is the one using Wikidata; however,
BnF

is a more accurate source of information for old place names (e.g. it records the older name “château de Bicêtre” for the “fort de Bicêtre”). Disambiguation accuracy is an interesting measure when there is more than one candidates for a place mention, and for Wikidata the value is strong. Weak values on NIL precision tell us that REDEN, which uses exact string match for retrieving candidates, sometimes misses the correct referent, and needs to be improved in this respect.

KB

Candidate Precision
Candidate recall
NIL precision
NIL recall
Disambiguation accuracy
Overall linking accuracy

DBpedia
1

0,816

0,367

1

none

0,834

BNF
0,76
0,63
0,58
0,97
1
0,7

Wikidata
0,91
0,83
0,44
1
1
0,85

Geo-visualisation

Once the digital edition is enriched with
placeName

tags, REDEN allows for the exploration of the spatial dimension of texts by retrieving structured information about places. For instance, in Wikidata, resources are described according to an ontology which contains properties for coordinate locations <https://www.wikidata.org/wiki/Property:P625>, images <https://www.wikidata.org/wiki/Property:P18>. By dereferencing IRIs, it is possible to access values for the aforementioned properties and use them in the context of a Web mapping application thereby to project places as points onto a map along with media or facts about these places (see Figures below).

Bibliography

Brando, C., Frontini, F. and Ganascia, J.-G.

(2016). REDEN: Named Entity Linking in Digital Literary Editions Using Linked Data Sets.
Complex Systems Informatics and Modeling Quarterly
,
0
(7): 60–80.

Boeglin, N., Depeyre, M., Joliveau, T. and Le Lay, Y.-F.

(2016). Pour une cartographie romanesque de Paris au XIXe siècle. Proposition méthodologique.
Conférence Spatial Analysis and GEOmatics
. (Actes de La Conférence SAGEO’2016 - Spatial Analysis and GEOmatics). Nice, France

(accessed 24 November 2018).

Dupont, Y.

(2017). Exploration de traits pour la reconnaissance d’entités nommées du Français par apprentissage automatiqu. Proceedings of RECITAL.

Elson, D. K., Dames, N. and McKeown, K. R.

(2010). Extracting Social Networks from Literary Fiction.
Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics
. (ACL ’10). Stroudsburg, PA, USA: Association for Computational
Linguistics, pp. 138–147

(accessed 24 November 2018).

Frontini, F., Brando, C. and Ganascia, J. G.

(2016). REDEN ONLINE: Disambiguation, Linking and Visualisation of References in TEI Digital Editions.
Digital Humanities 2016: Conference Abstracts
. Kraków: Jagiellonian University & Pedagogical University, pp. 193–97

.

Gregory, I., Donaldson, C., Murrieta-Flores, P. and Rayson, P.

(2015). Geoparsing, GIS and textual analysis: current developments in spatial humanities research.
International Journal of Humanities and Arts Computing
,
9
(1): 1–14 doi:

10.3366/ijhac.2015.0135

.

Lenardo, I. di and Kaplan, F.

(2015). Venice Time Machine : Recreating the density of the past. https://infoscience.epfl.ch/record/214895 (accessed 25 November 2018).

Piatti, B., Bär, H. R., Reuschel, A.-K., Hurni, L. and Cartwright, W.

(2009). Mapping Literature: Towards a Geography of Fiction.
Cartography and Art
. Berlin, Heidelberg: Springer, pp. 1--16

(accessed 11 January 2016).

Piper, A., Algee-Hewitt, M., Sinha, K., Ruths, D. and Vala, H.

(2017). Studying Literary Characters and Character Networks.
Digital Humanities 2017, DH 2017, Conference
Abstracts, McGill University & Université de Montréal, Montréal, Canada, August 8-11, 2017

.

Vala, H., Jurgens, D., Piper, A. and Ruths, D.

(2015). Mr. Bennet, his coachman, and the Archbishop walk into a bar but only one of them gets recognized: On The Difficulty of Detecting Characters in Literary Texts.
Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing
. Lisbon, Portugal: Association for Computational Linguistics, pp. 769–774

(accessed 24 November 2018).