Sound Predicts Meaning: Sound Iconic Relations between Vowels’ Formants and Emotional Tone in German and Japanese

In this research project, we investigate the extent to which the phonetic properties of a text tell us something about the mood expressed in that text. Empirical research on the relationships between sound and meaning in language has provided solid evidence that the articulatory and acoustic properties of phonemes are implicitly related to non-acoustic features such as size, brightness or emotional tone (Reay, 1994; Schmidtke et al., 2014; Sidhu et al., 2018). Moreover, these sound-meaning relationships appear to be broadly universal, i.e., they are found in all languages and language families.
However, while there is a wealth of studies on phonosemantic relationships at the level of individual phonemes or words, few studies have investigated the role of sound-meaning relationships in texts (Aryani et al., 2013; Auracher et al., 2010; Fónagy, 1961; Kraxenberger & Menninghaus, 2016; Whissell, 1999). The aim of this research project is to investigate whether and to what extent the relative frequency of certain phonetic features in texts predicts the emotional mood expressed in that text. That is, we hope to develop a universal (i.e., language-independent) method for automatic sentiment analysis based on the phonetic structure of texts.
To this end, we have compiled corpora of literary texts in various languages and language families, including English, German, Spanish, Chinese, Arabic, Japanese, Thai, and other languages. For the study, these texts are phonetically transcribed (converting graphemes to phonemes) and analyzed for the relative occurrence of certain phonetic features. In experimental studies, readers will then be asked to rate the emotional tone of randomly selected texts using bipolar scales that comprise the three dimensions of evaluation, potency, and activity.
In this paper, we present preliminary results of two experiments in which we compared sound-meaning relationships in German and Japanese poetic texts. In both experiments, we examined the relationship between the formant frequencies of vowels and the emotional tone expressed in the texts. It has been reported that formant dispersion, i.e., the relative distance between first and second formants, is implicitly associated with the notion of size, strength, or dominance (Auracher, 2017; Hoshi, 2019). Our hypothesis, therefore, was that readers would be more likely to find an expression of strength, size, and dominance in texts whose average formant dispersion is relatively low than in texts with relatively high format dispersion.
In a first experiment, 42 native German speakers evaluated the emotional tone of 90 stanzas pseudo-randomly selected from a corpus of about 1400 German poems (8000 stanzas) written between 1720 and 1900. The stanzas were divided into three categories according to whether their average formant dispersion was relatively high, relatively low, or around the general average for all stanzas. Each participant rated 30 stanzas (ten per category) on six items comprising the three bipolar dimensions of Evaluation (pleasant-unpleasant), Potency (dominant-submissive) and Activity (active-passive) (Osgood et al., 1957). Linear-Mixed-Models (LMM) per EPA dimension were run to analyze the data, using stanza category as a fixed factor and participants as a random factor. In the second experiment, we repeated the experimental design with 147 undergraduates from Japan who rated 75 Tanka (a particular form of Japanese poetry) on six bipolar scales. The Tanka were written between 1868 and 1975. Each participant rated between 12 and 21 Tanka.
Our results clearly point to a significant relationship between the phonetic features of the texts on the one hand and their evaluation by the readers on the other. In particular, the ratings of items related to the dimension of Dominance showed a strong and language-independent phonosemantic effect in the predicted direction. In both languages, stanzas with a relatively low formant dispersion were rated as highly significantly more
dominant, more
aggressive or more
powerful compared to stanzas with a relatively high or neutral formant dispersion. Similarly, stanzas with lower formant dispersion were rated as more
lively,
active or
aroused, suggesting that low formant dispersion is associated with Activity or Arousal. In contrast, the ratings of emotional valence (pleasant-unpleasant) expressed in the texts showed no significant relationship with formant dispersion.

In this paper, we will discuss these findings in terms of the potential of sound iconicity in written language for sentiment analysis and outline future plans for extending the research to other languages and genres. We will also present preliminary results from currently ongoing studies in which we re-analyze our data with an extended feature set. Based on these results, we will discuss the potential of using phonetic features as language-independent predictors for sentiment analysis.

Bibliography
Aryani, A., Conrad, M. and Jacobs, A.M. (2013). Extracting salient sublexical units from written texts: “Emophon,” a corpus-based approach to phonological iconicity. Frontiers in Psychology, 4: e654. https://doi.org/10.3389/fpsyg.2013.00654
Auracher, J. (2017). Sound iconicity of abstract concepts: Place of articulation is implicitly associated with abstract concepts of size and social dominance. PLoS One, 12: e0187196. https://doi.org/10.1371/journal.pone.0187196
Auracher, J., Albers, S., Zhai, Y., Gareeva, G. and Stavniychuk, T. (2010). P is for happiness, N is for sadness: Universals in sound iconicity to detect emotions in poetry. Discourse Processes, 48: 1-25. https://doi.org/10.1080/01638531003674894
Fónagy, I. (1961). Communication in Poetry. WORD, 17: 194-218. https://doi.org/10.1080/00437956.1961.11659754
Hoshi H., Kwon, N., Akita, K. and Auracher, J. (2019). Semantic associations dominate over perceptual associations in vowel–size iconicity. i-Perception, 10: e2041669519861981. https://doi.org/10.1177%2F2041669519861981
Kraxenberger, M. and Menninghaus, W. (2016). Mimological reveries? Disconfirming the hypothesis of phono-emotional iconicity in poetry. Frontiers in Psychology, 7: e1779. https://doi.org/10.3389/fpsyg.2016.01779
Osgood, C. E., Suci, G. J. and Tannenbaum, P. H. (1957). The measurement of meaning. Urbana: University of Illinois Press.
Reay, I.E. (1994). Sound symbolism. In Asher, R. E. (ed.), Encyclopedia of language & linguistics. Oxford: Pergamon Press, pp. 4064–4070.
Schmidtke, D.S., Conrad, M., and Jacobs, A.M. (2014). Phonological iconicity. Frontiers in Psychology, 5: e80. https://doi.org/10.3389/fpsyg.2014.00080
Sidhu, D.M. and Pexman, P.M. (2018). Five mechanisms of sound symbolic association. Psychonomic Bulletin & Review, 25: 1619–1643. https://doi.org/10.3758/s13423-017-1361-1
Whissell, C. (1999). Phonosymbolism and the emotional nature of sounds: Evidence of the preferential use of particular phonemes in texts of differing emotional tone. Perceptual and Motor Skills, 89: 19-48. https://doi.org/10.2466%2Fpms.1999.89.1.19