Beyond Corpora: Elicitation as a tool in second language word formation studies

Beyond Corpora: Elicitation as a tool in second
language word formation studies

Greg
Lessard
French Studies Queen's University
lessardg@qsilver.queensu.ca

Michael
Levison
Computing and Information Service Queen's
University at Kingston
levison@cs.queensu.ca

1999

University of Virginia

Charlottesville, VA

ACH/ALLC 1999

editor

encoder

Sara
A.
Schmidt

Background
In this paper, we will be concerned with the study of second language (L2)
linguistic creativity as it manifests itself on the lexical level,
particularly with respect to word formation. More specifically, we want to
measure the extent to which anglophone learners of French of varying degrees
of experience are capable of judging the relative productivity of different
suffixes, and how this performance compares to that of native speakers.
The measures of productivity of word-formation devices which have been used
to date, such as lexicographical data (Dubois 1962) and corpus data (Baayen
and Renouf 1996, Baayen and Lieber 1991) do not lend themselves well to the
study of L2 productivity. Among other things, L2 corpora show relatively
small amounts of productivity (Lessard, Levison, Maher and Tomek 1994,
Boeder et alii 1993). Despite issues of reliability and interpretation (see
Birdsong 1989, Gass 1994, Coppetiers 1987), elicitation appears to offer a
potentially useful measure.
However, to our knowledge, little research has been done on elicitation to
test lexical productivity in French, particularly among L2 learners. Hawkins
(1985) forms one exception, but he was concerned primarily with the class of
past participle markers, which are at the borderline of affixation and
derivation.
The research described here draws upon and extends previous work on native
speaker judgements of lexical productivity, including Aronoff and
Schvaneveldt 1978, Gorska 1982, Anshen and Aronoff 1991, Levison and
Lessard, 1995a, Fowler and Liberman 1995 and Keane and Costello 1997. It
should be noted however that very little previous work used a computational
environment, whereas this is central to the work presented here, and is
based on the VINCI natural language generation environment.

Experiments and results
The results discussed in the paper represent the third of three stages.
In the first, judgements of acceptability were elicited from native speakers
of French for derived forms in -able, -age, -ment, -tion and -ure. In a
nutshell, a French lexicon sorted by frequency was used to provide verbs of
the first conjugation. Suffixes were added automatically, and at random. A
randomized subset of the resulting derived forms was presented to each
subject, along with the base form of each verb. Subjects were required to
identify which verbs were known to them (almost all, in the case of the
native speakers) and which derived forms they found acceptable. Results
defined a continuum of relative acceptability with -able at the top of the list, followed by -age, -ment, -tion and -ure in that order. It is
important to note that the variable being measured was neither the
correctness of the judgements (whether an existing derived form corresponded
to those seen as acceptable) nor the individual lexical items, but rather
the ranking of suffixes in terms of the number of derived forms they were
seen to be capable of producing.
In the second stage, the same protocol was applied to non-native speakers
ranging from some with high school training in French to some with
significant university level studies in French. Results of these tests
showed that as knowledge of verb bases decreased, non-native speakers showed
increasing discrepancy in their judgements with respect to native speaker
rankings.
The third stage addressed problems found in the second: the absence of an
external measure on which to rank the linguistic skills of the subjects
tested, and the relatively high level of linguistic competence of almost all
the subjects tested. In response to these problems, the experiment was
repeated using the same protocols. Subjects tested were students in oral
French classes at Queen's University. Five levels were represented, ranging
from 016 to 320, where 016 represented those with essentially no previous
knowledge of French, while 320 represented those with near-native
proficiency. Placement in these classes had been done on the basis of an
oral interview.
As an illustration, the results for three of the five suffixes are reproduced
in the following table. In the table, the columns suffix ok and suffix not ok
represent the average of all responses for each class on the basis of 20
questions. In principle, each row should sum to 20, however because some
students responded to less than 20 questions, some small variations are
found. Verb known and verb
not known represent subjects' claimed knowledge of the base
verb.

Suffix
Course
Verb Known

Verb not Known

Suffix ok
Suffix not ok
Suffix ok
Suffix not ok

016
2
2.25
2.75
12.75

017
3.75
1.5
8.75
6

-able
118
6.5
2.5
7
4

219
9.4
3.7
2.4
3.8

320
9.1
4.1
2.1
3.7

016
2
3
1.5
13

017
4.75
3.25
8.5
3.5

-ment
118
5
4.5
9.5
1

219
8.7
4.3
2.7
3.6

320
4.6
8.1
2.1
4.6

016
1.5
4.25
2.5
11.75

017
1.75
4.75
3.5
9.75

-ure
118
2.5
9
2
6

219
1.3
10.9
1.3
5.7

320
2.9
10
1
5.4

The table shows that in all cases, the number of base verbs known rises with
level, suggesting that knowledge of verbal base and placement interview
results are measuring comparable things. As well, bearing in mind that the
acceptance rates by native speakers for derived forms based on -able, -ment and -ure were 77%, 39% and 10% respectively, we see in
the non-native speaker data a gradual convergence on native speaker-like
judgements. Thus, in the case of -able, while
016 students find little to choose between accepting of rejecting derived
forms for verbs they know (2 acceptances and 2.25 rejections) and reject
strongly derived forms for verbs they don't know (2.75 acceptances versus
12.75 rejections), more advanced students tend to accept derived forms for
verbs they know (in the case of 219, 9.4 acceptances versus 4 rejections)
while rejecting somewhat derived forms in -able
for verbs they don't know.

Conclusions and future directions
This data hides considerable variation which will be elaborated during the
presentation. However, it confirms that the measure is robust, even with
students with relatively lower skill levels in French. In the paper, more
detailed analyses will be presented. As well, an extended range of measuring
instruments will be discussed, based on contextualizing examples to be
evaluated in a sentence generated on the fly, other types of measures (see
Feldman 1995 for examples) and scales (see Bard, Robertson and Sorace
1996).

References

F.
Anshen

M.
Aronoff

Morphological Productivity and Phonological
Transparency

Canadian Journal of Linguistics

26
1
63-72,
1981

M.
Aronoff

R.
Schvaneveldt

Testing morphological productivity

Annals of the New York Academy of Sciences

318

106-114
1978

H.
Baayen

R.
Lieber

Productivity and English derivation: a corpus-based
study

Linguistics

29
4
801-844
1991

H.
Baayen

A.
Renouf

Chronicling the Times: Productive Lexical Innovations
in an English Newspaper

Language

72
1
69-96
1996

E.
Bard

D.
Robertson

A.
Sorace

Magnitude Estimation of Linguistic
Acceptability

Language

72
1
32-67
1996

D.
Birdsong

Metalinguistic Performance and Interlinguistic
Competence

Berlin
Springer-Verlag
1989

P.
Broeder

G.
Extra

R.
van Hout

K.
>Voionmaa

Word formation processes in talking about
entities

C.
Perdue

Adult language acquisition: cross-linguistic
perspectives

vol. 2
Cambridge
Cambridge University Press
1993
41-72

R.
Coppetiers

Competence Differences between Native and Near-Native
Speakers

Language

63
3
544-573
1987

J.
Dubois

Ètude sur la dérivation suffixale en francais moderne
et contemporain; essai d'interprétation des mouvements observés dans
le domaine de la morphologie des mots construits

Paris
Larousse
1962

L.
B.
Feldman

Morphological aspects of language processing

Hillsdale, NJ
Lawrence Erlbaum
1995

A.
Fowler

I.
Liberman

The Role of Phonology and Orthography in Morphological
Awareness

L.
B.
Feldman

Morphological aspects of language processing

Hillsdale, NJ
Lawrence Erlbaum
1995
157-188

S.
Gass

The Reliability of Second-Language Grammaticality
Judgements

E.
Tarone

S.
Gass

A.
Cohen

Research Methodology in Second-Language
Acquisition

Hillsdale, NJ
Lawrence Erlbaum
1994
303-322

Elzbieta
Gorska

A way of testing the productivity of word formation
rules (WFRs)?

Studia Anglica Posaniensa

14
1
169-174
1982

R.
Hawkins

Errors in the Use of French Past Participles By Foreign
Speakers and Their Implications for a Model of Morphology

Lingua

67

171-188
1985

M.
Keane

F.
Costello

Where Do "Soccer Moms" Come From? Cognitive Constraints
on Noun-Noun Compounding in English

Proceedings, Computational Models of Creative Cognition
Conference, Dublin

1997

G.
Lessard

M.
Levison

Lexical Creativity in L2 French

Internation Review of Applied Linguistics

(in press)

G.
Lessard

M.
Levison

D.
Maher

I.
Tomek

Modelling Second Language Learner Creativity

Journal of Artificial Intelligence in Education

5
4
455-480
1994

G.
Lessard

M.
Levison

Experiments in Word Creation

ACH/ALLC 95, Santa Barbara Conference Abstracts

1995
74-77