53
Associations between multimodal metaphor functions and signals
in Chinese video ads
Molly Xie Pan, The Hong Kong Polytechnic University
(xiemo.pan@polyu.edu.hk)/Dennis Tay, Nanyang Technological University
(dennis.tayzm@ntu.edu.sg)
Abstract
Most existing studies analyze the manifestation and functions of metaphors in multimodal
video advertisements from a qualitative perspective, focusing on the content of selected
examples and leaving potentially generalizable structural traits underexplored. Addressing
this issue contributes to the systematic and empirical development of multimodal metaphor
studies. This study demonstrates how quantitative analyses can provide insights into the
structural traits of metaphors in video ads by examining the associations among metaphor
signals, functions, and product types in 197 metaphors from 66 metaphorical Chinese video
ads. Reliable procedures for identifying metaphors and coding functions and signals were
applied. Results from a set of categorical data analytics showed i) a significant association
between metaphor functions and product type, ii) a significant association between metaphor
functions and signals, and iii) a non-significant association between product types and signals.
This provides empirical evidence for stable structural traits across large metaphor samples.
Die meisten vorliegenden Studien analysieren die Formen und Funktionen von Metaphern in
multimodaler Videowerbung aus einer qualitativen Perspektive. Dabei konzentrieren sie sich
auf den Inhalt ausgewählter Beispiele und lassen potenziell verallgemeinerbare Strukturmerkmale
unerforscht. Eine Auseinandersetzung mit diesem Thema leistet einen Beitrag zur
systematischen und empirischen Entwicklung der multimodalen Metaphernforschung. Die
vorliegende Studie zeigt, wie quantitative Analysen Einblicke in die strukturellen Merkmale
von Metaphern in Videoanzeigen liefern können, indem sie die Verbindung zwischen
Metaphersignalen, -funktionen und Produkttypen bei 197 Metaphern aus 66 metaphorischen
chinesischen Videoanzeigen untersucht. Dabei kommen verlässliche Verfahren zur Identifikation
von Metaphern und zur Kodierung von Funktionen und Signalen zur Anwendung.
Die Ergebnisse einer Reihe kategorialer Datenanalysen zeigen i) einen signifikanten Zusammenhang
zwischen Metaphernfunktionen und Produkttyp, ii) einen signifikanten
Zusammenhang zwischen Metaphernfunktionen und Signalen und iii) einen nicht signifikanten
Zusammenhang zwischen Produkttypen und Signalen. Dies liefert empirische Belege
für stabile Strukturmerkmale innerhalb großer Metaphernstichproben.
1. Introduction
In the age of digital advancements, videos have become a vital instrument for
advertising. The rise of social media and mobile technology has established
videos as the primary information source for many individuals. The HubSpot
metaphorik.de 35/2024
54
Digital Consumer Trends Report1 shows that over 50% of consumers prefer
acquiring information about a brand or business through videos ads. The use of
metaphors in ads has been demonstrated to enhance engagement, memorability,
and entertainment, thereby significantly amplifying their persuasive
power (Ang/Lim 2006; Burgers et al. 2015; Chang et al. 2018; Delbaere et al.
2011; Pérez-Sobrino et al. 2019). Nevertheless, the investigation of metaphors in
video ads remains relatively limited compared with the print ads. This could be
attributed to the inherent temporal and dynamic complexity of this genre
(Pérez-Sobrino/Littlemore 2017; Pan/Tay 2021).
Current research on characteristics of metaphors in video ads primarily consists
of qualitative analyses focusing on a limited number of cases (Forceville 2007,
2008; Guan/Forceville 2020; Pérez-Sobrino/Littlemore 2017; Urios-Aparisi
2009). These studies offer nuanced insights into the idiosyncratic nature of
metaphorical interpretation within specific topics. However, a comprehensive
understanding of the underlying structural traits is lacking due to the absence
of systematic investigations on a large corpus. To fill this gap, a systematic
exploration of these structural traits can substantiate postulations derived from
case studies, providing empirical evidence for theory development and
application.
Pioneering research on pictorial metaphors (Forceville 1996) serves as the
foundation for exploring multimodal metaphors. Forceville (2008) analyzed
nine Dutch TV ads and revealed distinct difference of metaphors in a service
product ad. Unlike other visualizable products, the service ad does not
necessarily tie the metaphor to the product or a competitor. Instead, the
metaphor appears to establish connections with prospective consumers. While
these analyses provide an overview of how metaphors are used in Dutch TV
ads and highlight potential differences across product types, the descriptive
nature and limited sample size hinder the confirmation of associations between
metaphor usages and product types.
An important challenge in studying metaphor within multimodal discourse is
the absence of explicit cues, such as is or is like in linguistic discourse, to signal
metaphoric relations (Forceville 2009; Müller/Schmitt 2015; Pérez-Sobrino/
Littlemore 2017). Research on visual metaphor typically identifies metaphors
1 https://blog.hubspot.com/marketing/content-trends-global-preferences.
Pan/Tay: Associations between multimodal metaphor functions and signals
55
through visual incongruity (Phillips/McQuarrie 2004; Šorm/Steen 2018).
Nevertheless, the possibility of creating metaphor without visual incongruity
has also been explored (Forceville 1996, 2008, 2016). Overall, there are several
typologies of visual metaphors in print ads (Forceville 2008; Gkiouzepas/Hogg
2011; Phillips/McQuarrie 2004). The techniques of creating these visual metaphors
can potentially serve as important cues for metaphor signaling devices in
video ads (Pan/Tay 2021). The empirical question of whether metaphor signals,
metaphor functions, and product types exhibit systematic correspondences
remains unresolved. Addressing this necessitates a systematic investigation
into the manifestation and functions of metaphors in video ads. Quantitative
analyses of a large sample size are essential to illuminate the underlying
associations.
This paper will report a corpus-driven study on 197 metaphors identified from
66 Chinese metaphorical video ads. This study does not specifically investigate
individual metaphors or conceptual metaphors. Instead, it focuses on analyzing
the frequency of metaphors within real-world video ads and explores the
relationships among metaphor signals, metaphor functions, and product types.
The aim is to uncover stable structural associations across a large sample of
metaphors. The subsequent section will review key issues related to researching
metaphors in video ads, encompassing defining modes and multimodal metaphors,
metaphor identification, metaphor signals, and metaphor functions.
Following this, the method section will introduce details such as corpus
building, metaphor identification procedure, coding schemes of signals and
functions, and the statistical analyses performed. The results will be presented
alongside discussions. The concluding section will summarize the paper by
highlighting major findings and acknowledging limitations.
2. Theoretical background
The following sections introduce a working definition of mode and multimodal
metaphors, discuss the methodological challenge of identifying metaphors in
multimodal discourse, and illustrate metaphor signals and functions through
examples.
metaphorik.de 35/2024
56
2.1 Defining mode and multimodal metaphor
Research on multimodal metaphors proceeds with debates surrounding the
definition of the term mode in multimodality. The term mode has received varied
definitions across different research paradigms. In social semiotics, a mode is
often integrated with a specific meaning influenced by social and ideological
factors. The combination of different modes in meaning-making is described as
“the work of an overarching code whose rules and meanings provide the
multimodal text with the logic of its integration” (Kress/Van Leeuwen 1996:
177). A cognitive semiotic approach also integrates semiotic systems (language,
gesture, and depiction) with sensory modalities (Stampoulidis et al. 2019),
assigning semiotic systems to sensory modalities based on their manifestation:
written language (visual), speech (auditory), gesture (visual), and depiction
(visual). Despite divergent views on what a mode is, we align with Forceville
(2009: 23) that “it is at this stage impossible to give either a satisfactory definition
of ‘mode’, or compile an exhaustive list of modes”. In our study, we identify
distinctive modes in contemporary Conceptual Metaphor Theory-inspired
research on multimodal discourse as potential candidates for modes in video
ads (EI Refaie 2003; Forceville 2017; Pérez-Sobrino 2017; Schilperoord 2018; Tay
2017): (1) visual images; (2) written language; (3) spoken language; (4) gestures;
(5) sounds; (6) music.
A clear definition of multimodal metaphor is crucial for advancing our investigation.
Cognitive linguists conceptualize metaphor as a cognitive process that
involves mapping knowledge and experience from one domain (the source
domain) onto another (the target domain) (Kövecses 2017: 328; Lakoff 1993: 206;
Lakoff/Johnson 1980: 170). Typically, the target domain is more abstract and
less familiar than the source domain. For instance, the sentences like don’t jump
to conclusions and I’m letting my mind drift, words associated with physical
location, such as jump and drift, are used to describe mental activities. This
illustrates the conceptual metaphor IDEAS ARE LOCATIONS at the domain level
and the metaphor THINKING IS MOVING IN THE IDEASCAPE at the frame level,
according to the levels of metaphor (Kövecses 2017). As a conceptual process,
metaphor can manifest itself in any mode of communication, including verbal
language, visual images, gestures, and sounds (Forceville/Urios-Aparisi 2009).
When more than one mode contributes to creating a metaphor, it is termed as a
multimodal metaphor. When a metaphor is formed with only one mode
Pan/Tay: Associations between multimodal metaphor functions and signals
57
contributing to both the target and the source, it is termed as a monomodal
metaphor.
Fig. 1: Inlingua Language School: Restroom2
Figure 1 illustrates a multimodal manifestation of the conceptual metaphor
IDEAS ARE LOCATIONS. This example is drawn from VisMet, wherein the multimodal
metaphor is verbalized as MENTAL ACTIVITIES ARE PHYSICAL ACTIVITIES.
While conveying a meaning akin to THINKING IS MOVING IN THE IDEASCAPE, the
conceptual labels assigned by analysts differ. This highlights a distinction
between metaphor in multimodal discourse and linguistic metaphor,
emphasizing that the verbalization of the former may involve the interpretation
of non-verbal language. Such interpretation may vary, showcasing diversity
and creativity among analysts (Pan/Tay 2023; Pérez-Sobrino/Ford 2023). In this
visual depiction, the brain structures are replaced by spiral stairs leading to the
word restroom. The inception of the staircase is positioned beside the mouth,
accompanied by the written text “Quick! Where’s the…”. This design suggests
that the mind embarks on a journey, searching for the word restroom. The
written text provides a cue for the targeted ideas/mental activities, collaborating
with visuals to construct a multimodal metaphor.
Given the abundant resources for creating metaphors in video ads, a genre
driven by specific goals (i.e. selling the products) (Forceville 2016; Pérez-Sobrino
2017), our study defines metaphor as a noticeable and impressive phenomenon.
2 http://www.vismet.org/VisMet/display.php#filter-Context-ctxt1.
metaphorik.de 35/2024
58
It invites viewers to comprehend and experience one concept in terms of
another, with the overarching goal of facilitating product promotion. While it is
theoretically possible to encounter monomodal metaphors (such as purely
verbal or visual metaphors) in video ads, they are infrequent in our corpus.
Even though we refer to these phenomena as metaphors in the rest of the paper,
their intrinsic nature is multimodal considering their construction. Our
approach is rooted in Conceptual Metaphor Theory (Lakoff 1993) and research
on multimodal metaphors (Forceville 1996; Forceville/Urios-Aparisi 2009),
which diverges from a cognitive semiotic approach that
goes to the opposite extreme of CMT, acknowledging as metaphorical
only fully creative and non-conventional (iconic) processes, while
metaphors in both language and depiction can be more or less
conventionalized (sedimented) in the cultural knowledge of a given
society (Stampoulidis et al. 2019: 6, emphasis in original).
In our study, both conventional and novel metaphors can be crafted as
noticeable, with the potential for systematic mappings across domains.
However, investigating mappings across domains is beyond the scope of this
study.
2.2 Identifying multimodal metaphors from video ads
The identification of multimodal metaphorical phenomena faces methodological
challenges owing to the inherent complexity of video discourse.
Currently, three procedures are proposed for identifying metaphors in TV
commercials: i) Filmic Metaphor Identification Procedure (FILMIP) (Bort-Mir
2021), ii) a three-step procedure developed by Bobrova (2015), and iii) Creative
Metaphor Identification Procedure in Video Ads (C-MIPVA) (Pan/Tay 2021,
2023). All procedures address the challenge of identifying metaphors in the
context of moving images within advertising genres. FILMIP, developed
through the analysis of 11 TV commercials (Bort-Mir 2021), comprises seven
steps. It begins with a comprehensive analysis of every element in a video (Steps
1 and 2), aiming to establish a general understanding of the ad. This involves
drawing on theoretical frameworks related to films and multimodality, such as
the Structured Annotation (Tam/Leung 2001). The procedure then progresses
to detect incongruous filmic components (Step 3), compare meanings of
concepts (Step 4), determine domains of concepts (Step 5), and establish
mappings between concepts (Step 6). The final step involves marking the ad as
Pan/Tay: Associations between multimodal metaphor functions and signals
59
metaphorical if Steps 4, 5, and 6 yield positive results, or marking the ad as nonmetaphorical
if any of the preceding steps is negative. FILMIP offers metaphor
analysts a detailed method for extracting metaphorical elements. However, its
adoption for systematic large-scale studies faces challenges. First, its reliability
necessitates rigorous examination through inter-rater reliability among
annotators (Pan/Tay 2023). Second, the lack of criteria for verbalizing
metaphors identified by FILMIP may influence coding consistency. Third, while
FILMIP provides ample theoretical references for application, it is laborintensive
to analyze even a single ad, which may pose difficulties for large-scale
corpus studies.
The three-step procedure developed by Bobrova (2015) exploits filmic
techniques to identify metaphors in TV commercials. It begins by identifying
potential metaphors through the consideration of filmic techniques (Step 1),
including as compelling context, juxtaposition of objects, and transformation of
an image. Subsequently, it focuses on identifying mapping features (Step 2). The
final step involves pinpointing the source and target domains and formulating
the verbal expression of cross-domain mapping (Step 3). This procedure
addresses three questions for identifying metaphors in multimodal discourse,
as outlined by Forceville (2002: 2):
1. Which are the two terms of the metaphor, and how do we know?
2. Which is the metaphor’s target domain and which is the metaphor’s source
domain, and how do we know?
3. Which features can/should be mapped from the source domain to the target
domain, and how is their selection decided upon?
The inter-rater reliability, however, has not been investigated. Given that each
step involves interpretation, assessing its reliability becomes challenging.
Informed by both this procedure and literature on metaphor analysis in
linguistic discourse, Pan/Tay (2021, 2023) proposed C-MIPVA on a corpus
comprising 20 metaphorical Chinese video ads. The proposed method, CMIPVA
with filmic techniques as metaphor signals, has had its reliability
substantiated through statistical evidence (Pan/Tay 2021; Pan/Tay 2023). A
distinguishing feature of this procedure, in contrast to the two aforementioned
ones, lies in treating metaphor identification as the initial stage of metaphor
analysis. In other words, instead of incorporating interpretation into metaphor
metaphorik.de 35/2024
60
identification, C-MIPVA focuses solely on documenting responses to metaphor
identification. For instance, annotators are only required to indicate with a
“Yes” or “No” whether there is transfer of meaning between two concepts,
rather than providing detailed explanations of the transferred meanings. This
approach acknowledges the diversity of interpretation stemming from individual
differences, cultural influences, and contextual factors. Consequently,
reliability assessment is confined to determining whether a unit is metaphorical
or not. The verbalization of it is in italics and small capitals, such as A IS B or
DOING A IS DOING B. While verbalization may vary across individuals, achieving
evaluation of agreement on whether the targets and the sources from different
raters refer to the same concepts is still possible. To systematically investigate a
large corpus, this study adopts C-MIPVA as the chosen procedure for identifying
metaphors in video ads. This working procedure does not make any
assumption about the cognitive processing of metaphors by viewers.
2.3 Metaphor signals and uses
The essence of C-MIPVA is to exploit concrete filmic techniques to identify
metaphorical components and determine metaphors within the context. The
term filmic techniques is employed broadly and encompasses various methods of
editing and combining visuals, written texts, spoken language, sound and
music to construct a multi-sensory experience in videos. These techniques are
important in directing viewers’ attention and shaping the information flow
(Verstraten 2009). Previous research on metaphors in diverse contexts,
including films (Müller/Schmitt 2015), political video ads (Iversen 2017), and
video ads (Pan/Tay 2021), has underscored the importance of filmic techniques
in signaling metaphors. Pan/Tay (2021) identified five filmic techniques as
metaphor signals through the application of the three-step procedure
developed by Bobrova (2015).
The five filmic techniques are: Transformation of Images, Depicting Non-
Existing Gestalt, Replacement, Juxtaposition, and Simultaneous Cueing of
Different Modes. To illustrate these visual effects, Pan provides several
examples on the Open Science Framework.3 Transformation of Images converts
one or part of an entity into another (Bobrova 2015). This technique typically
3 https://osf.io/egrxh/?view_only=d5a5dcb734be42a09452d8fa91bfaeaa.
Pan/Tay: Associations between multimodal metaphor functions and signals
61
portrays the complete process of transforming A into B. For instance, Figure 2
depicts the progression wherein a projector transforms into the icon of Baidu
Cloud.
a b c
Fig. 2: Screenshots from the Ad of Baidu Cloud4
Depicting Non-Existing Gestalt creates a visual gestalt that is ‘impossible’ in real
life (Forceville 2008; Gkiouzepas/Hogg 2011). This impossible gestalt may
consist of fanciful cartoon elements or a hybrid gestalt combining parts of two
distinct focal objects. Figure 3 illustrates the visual effects of this technique. In
Figure 3 (a), a shadow of a hand and arm is presented, while Figure 3 (b) shows
a hybrid image merging a man with a cartoon depiction of an Automated Teller
Machine (ATM).
a b
Fig. 3: Screenshots from the Ads
(a: Wanmei Eyecream5; b: Jingdong Finance Service6)
Replacement is a common technique in print ads, where one object “has been
replaced by an object foreign to the schema” (Gkiouzepas/Hogg 2011: 105). The
4 https://www.digitaling.com/projects/24017.html.
5 https://v.qq.com/x/page/r0146rntgp7.html.
6 https://v.youku.com/v_show/id_XMTg3NTYxODc2MA==.html?spm=a2h0k.8191407.
0.0&from=s1.8-1-1.2.
metaphorik.de 35/2024
62
visual effect is similar to the example shown by Figure 1, where something
foreign to the context replaces the original item. Figure 1 showcases the use of
replacement in print ad, and this technique is equally applicable in moving
images. In contrast to Transformation of Images, which clearly portrays the
transformation process between two entities presented in complete forms,
replacement directly presents an element alien to the scenario. For example,
Figure 4 depicts a stewardess is helping a package to fasten the seatbelt. The
EMS packages replace the schema of passengers in the airplane, emphasizing
the high quality of the postal service from EMS.
a b
Fig. 4: Screenshots from EMS7
Juxtaposition presents the alignments of one shot or sequence of shots, either
horizontally or vertically (Bobrova 2015). Figure 5 shows this visual effect,
where the product Tommee Tippee is juxtaposed with a guitar to emphasize the
music function embedded within the product.
Fig. 5: A screenshot from the Ad of Tommee Tippee’s Feeding Bottle8
7 https://v.youku.com/v_show/id_XNzc0NzgwNDky.html?spm=a2h0k.8191407.0.0&
from=s1.8-1-1.2.
8 http://www.tvcbook.com/showVideo.html?vid=70386&code=3af5MKotjdgM4uoyPB
4z_qESdgZGArMK4axYhYkgjeqhoA.
Pan/Tay: Associations between multimodal metaphor functions and signals
63
Simultaneous Cueing of Different Modes is only considered when there is no
visual incongruity displayed by the aforementioned four techniques to differentiate
these signals. This signal typically portrays one thing in a mode but
presents another thing in a different mode simultaneously (Forceville 2009). In
this manner, it guides the viewers into connecting the two different things
together. Figure 6 illustrates such a situation where a coat is taken off from a
girl, and the accompanying spoken language states “remove 98% grease off
your body,” inviting viewers to connect the concepts of the coat and grease.
a b c
Fig. 6: A screenshot from the Ad of FOTILE Range Hood9
Research on metaphors in TV commercials indicates that metaphors can carry
major claims about a product and has postulated that metaphors in service ads
exhibit distinct characteristics compared to other types of ads (Forceville 2008).
The examples discussed above highlight the variability in how metaphors
function when applied to different products. For instance, the ad for Inlingua
Language School (Figure 1) employs metaphors related to users, while the ad
for Tommee Tippee’s Feeding Bottle (Figure 5) metaphorically represents its
own product. A study on personification metaphors in video ads proposes three
primary metaphor functions: 1) Features-Highlighting metaphors, emphasizing
the positive attributes of products and involving the product as the target of a
metaphor; 2) Supporting metaphors, coexisting with Features-Highlighting
metaphors and supporting the central metaphorical scenarios by providing
additional relevant mappings; 3) Needs-Highlighting metaphors, creating
scenarios that highlight problems or desirable outcomes for the intended
audience, where the product is introduced as the solution or motivation to
achieve the desirable result (Pan 2022). Applying this categorization, metaphor
in the ad for Inlingua Language School (Figure 1) fall under Needs-highlighting
9 https://v.qq.com/x/page/q0628rrjd6u.html.
metaphorik.de 35/2024
64
metaphors, as it describes challenges faced by second language learners. The
metaphors for the feeding Bottle (Figure 5) TOMMEE TIPPEE FEEDING BOTTLE IS A
GUITAR and Baidu Cloud (Figure 2) BAIDU CLOUD IS A PROJECTOR are classified as
Features-Highlighting metaphors, as they involve the products as the targets,
promoting the attributes of products. The metaphor shown in Figure 6, GREASE
IS COAT, serves as a Supporting metaphor, contributing to the creation of the
central metaphorical scenario where the product is metaphorized as an airplane,
and oil is metaphorized as passengers.
In summary, the current body of literature on metaphors in video ads has laid
a solid theoretical foundation for exploring the interconnections among categorical
variables, such as metaphor signals, metaphor functions, and product
types. Qualitative case studies suggest that metaphors in video ads for different
product types manifest distinct characteristics. However, empirical evidence to
systematically examine the correlations among metaphor signals, metaphor
functions, and product types is currently lacking. This study seeks to address
this gap by investigating the following research questions:
1. Are metaphors prevalent in Chinese public video ads?
2. Is there any association among metaphor signals, metaphor functions, and
product types in Chinese video ads? If yes, what are these associations?
3. Method
This study adopts a corpus-driven approach where the characteristics of metaphors
have been explored through a bottom-up analysis. This section introduces
the corpus, coding schemes for metaphor signals and their uses, and the
statistical analysis performed.
3.1 Corpus building
The corpus comprises 66 metaphorical Chinese video ads, constructed through
a systematic procedure. Initially, stratified random sampling was employed to
form a comprehensive corpus of Chinese video ads, with the goal of ensuring
representation across various product types and popular platforms in the
Chinese mainland. Two distinct product types were identified based on the
tangibility of product attributes within the realm of the Economics of Information
(EOI) (Bloom 1989). Search products, such as tables and cups, possess
Pan/Tay: Associations between multimodal metaphor functions and signals
65
concrete and tangible attributes with predetermined manufacturing standards
(Bloom/Reve 1990; Jiménez/Mendoza 2013). On the other hand, experience
products, like haircuts and travel services, have abstract and intangible attributes,
effective evaluation of which occurs only post-consumption. The strata
were defined along these product types and popular online platforms affiliated
with the BAT companies (Baidu, Alibaba, and Tencent), which dominated the
Chinese digital market from 2016 to 2021, as reported by emarketer. The
prevalence of these platforms among the BAT companies indicates their popularity
among the Chinese audience, resulting in videos from these platforms
have significant exposure to the general public. The keyword used for the search
was ‘广告’ (‘Advertisements’) and the research randomizer7 was employed to
generate random numbers for ad selection, excluding duplicate ads. To ensure
feasibility in data analysis, 10 ads were randomly chosen from each stratum,
amounting to a total collection of 100 Chinese video ads. Data collection was
completed in June 2019, ensuring that the videos in this corpus remained active
within the general public sphere at the time of collection.
Secondly, Pan/Tay applied Creative Metaphor Identification Procedure for
Video Ads (C-MIPVA) (Pan/Tay 2021, 2023) (see Figure 7) to the corpus and
identified 66 ads that contained at least one metaphor. Table 1 displays an
overview of the corpus-building procedure. Inter-rater reliability (IRR) was
conducted among six annotators, including three metaphor analysts and three
novice annotators who did not research metaphor. The procedure involved
calibrating their understanding using examples. Subsequently, the annotators
independently coded 20% of the data to determine whether a unit was metaphorical.
Finally, they engaged in discussions to resolve any disagreements.
Fleiss Kappa was employed to measure the extent to which annotators agreed
that a unit contained a metaphor. Percentage agreement was utilized to assess
the level of agreement between annotators regarding the conceptual labels of
the target and the source. The reliability of metaphor identification was
supported by statistical evidence from Fleiss Kappa (k = .72) in Step 5 of Figure
7 and a percentage agreement of 84% in Step 6. Further details on inter-rater
reliability examinations can be retrieved from Pan/Tay (2023). It is important to
notice that Fleiss Kappa indicates substantial agreement, signifying that
although there is ambiguity in metaphor identification, the agreement on
metaphorical units can reach a statistically supportive level. For percentage
agreement, consensus was reached through discussions on whether the labels
metaphorik.de 35/2024
66
of the target and source represented the same concept. A standardized term was
adopted for the labels representing the same concept. This might compromise
the diversity of conceptual labels from annotators. However, this study
acknowledges the variety and diversity of conceptual labels in interpreting
metaphors. Although statistical evidence for agreement is provided, the study
refrains from claiming a unified interpretation of metaphors. Instead, it reveals
that verbalizing metaphors involves interpretation, posing challenges for interrater
reliability. The current methodological choice is a working procedure for
metaphor verbalization. In addition, the focus of discussion is on assessing the
extent to which the same concept is referred to rather than exploring the idiosyncratic
aspects of interpretation.
Pan/Tay: Associations between multimodal metaphor functions and signals
67
Fig. 7: An overview of the six-step procedure in C-MIPVA (Pan/Tay 2021: 222)
Step 1: Watch through the ad and identify the product.
Step 2: Does the ad contain any
given filmic technique? (Mark all
that are shown in the ad)
A. Transformation of Images
B. Non-Existing Gestalt visually
shown
C. Replacement of objects/scenes
D. Juxtaposition of objects/scenes
Step 2.1: Does the
ad show an object/
scene in one mode
(e.g. visuals) but
name it as another
object/scene in a
different mode
(e.g. verbal language)?
This ad does
not contain
metaphor.
Step 3: Are there different concepts connected by the filmic
technique and/or by the interactions of different modes? What are
these concepts?
Step 4: For each pair of concepts, is there any transfer
of meaning from one concept to the other?
Step 5: Does the transfer of meaning help me look at a
concept in a different way?
Step 6: Verbalize the metaphor as A IS B or DOING A IS DOING B (A/Doing A is the
concept received new perspective in Step 5. Then work on other units, until all
metaphors are identified.
Mark this unit as
non-metaphorical
and work on other
units
No
Yes
No
Yes
Yes No
Yes
No
No
Yes
metaphorik.de 35/2024
68
Tab. 1: An overview of corpus building
3.2 Coding metaphor signals and functions
In C-MIPVA, Step 2 involves the segmentation of the video by identifying
segments containing filmic techniques with the potential to construct a
metaphor. These filmic techniques are considered metaphor signals in video ads
(Pan/Tay 2021). A video may contain more than one metaphorical segment. The
coding of metaphor signals and functions was conducted for each segment
determined to be metaphorical. There was a total of 197 metaphors identified in
the corpus.
3.2.1 Identifying metaphor signals
The five filmic techniques were derived from existing literature on multimodal
metaphor and analysis of 15 Chinese video ads randomly selected from the
corpus. The use of the term filmic technique is broad, encompassing various
methods of conveying multisensory information. The analysis process is iterative,
involving the application of theoretically interested filmic techniques
(Bobrova 2015) and refinements with reference to the data. Eventually, we
identified four filmic techniques that create visual incongruity, a prominent cue
for multimodal metaphor (Šorm/Steen 2018), and one filmic technique that
does not create visual incongruity. While it is also theoretically possible to have
a monomodal visual metaphor without any verbal or technical cue or incongruity
to prompt metaphorical thinking, such cases were not found in our data.
This absence may be attributed to the genre of commercial advertising, which
has a clear goal of selling a product/service within a limited time frame but with
a substantial budget to convey information (Forceville 2016; Pérez-Sobrino 2017;
Pérez-Sobrino/Littlemore/Ford 2021). Therefore, we chose to focus on the
100 Chinese Video
Ads
34 Non-Metaphorical
Video Ads
66 Metaphorical
Video Ads
C-MIPVA
Pan/Tay: Associations between multimodal metaphor functions and signals
69
prevalent ways of signaling metaphors in our corpus. Pan/Tay developed a
scheme to code each filmic technique (see Table 2). Another annotator, a native
Chinese speaker, underwent the inter-rater reliability process together with
Pan/Tay. The IRR process involved two raters going through a process of
calibrating their understanding of the scheme, independent coding, and
discussions to resolve disagreements (Cameron/Maslen 2010). The two raters
independently annotated 20 video ads, accounting for 20% of the entire corpus.
Results from Krippendorff’s Alpha (α = .78) indicated substantial agreement
(Krippendorff 2004).
Phase 1 Watch through the entire ad
Phase 2 Scrutinise the ad and mark down every segment that contains any
of the five filmic techniques and the timespan for each, according to
the following descriptions.
A. Transformation
of Images
The timespan starts with visuals of the first entity or feature and
ends at visuals of the entire new entity or feature. The segment
involves the old and new entities and the process of transformation.
B. Depicting Non-
Existing Gestalt
The timespan starts with visuals that the non-existing gestalt shows
and ends at the scene where this gestalt disappears. The segment
clearly shows the whole entity of this gestalt and its show-time in
the scene(s). If it shows more than once, record each segment
containing it.
C. Replacement The timespan starts when the visuals of replacement show and ends
with their disappearance. The segment clearly shows the whole
duration of this replacement.
D. Juxtaposition If the juxtaposition shows in one shot, record the timespan of this
shot. If not, the timespan starts with the beginning of the first shot
and ends at the end of the second shot. It might be possible to have
shots that are not placed together. In this case, each timespan for the
shots that build up juxtaposition is recorded. The segment clearly
shows the shots that are juxtaposed either in one shot or several
shots. When there is more than one juxtaposition, the cohesion of
the ad is helpful to do the segmentation.
E. Simultaneous
Cueing of Different
Modes
If a video contains more than one mode, consider this possibility by
providing a separate segment that marks the entire ad.
Tab. 2: A scheme to detect the five filmic techniques (Pan/Tay 2021: 224)
metaphorik.de 35/2024
70
3.2.2 Identifying metaphor functions
Each metaphor was further coded for its function, following the scheme
outlined in Table 3 (Pan 2022: 177-178). Fourteen video ads, containing a total
of 65 metaphors and accounting for 36% of the overall metaphors, underwent
an inter-rater reliability examination procedure. Pan, working with a doctoral
student who is a native Chinese speaker, independently coded the functions of
these 65 metaphors and discussed ambiguous cases after calculating the agreement.
Results from Cohen’s Kappa (k = .74) indicated a substantial agreement
(Cohen 1968; Pan 2022).
Steps Descriptions
Step 1 Watch the whole ad and get to know the context of each metaphor.
For each metaphor,
Step 2 Judge whether the metaphor’s target or source is the product. If yes, mark it
as Features-Highlighting Metaphor. If no, go to step 3.
Step 3 Judge the relationship between the claim of this metaphor (conveyed by two
terms and mapping) and the ad’s claim for the product.
A. If the ad contains a central metaphor, consider whether this metaphor
provides sub-mappings to support the metaphorical scenario lead by the
Features-Highlighting Metaphor.
B. If yes, mark it as Supporting Metaphor.
If no, consider the following possibilities.
C. The metaphor’s claim is in alignment with the ad’s claim for the product.
It provides more details of the product’s traits.
Mark it as Supporting Metaphor.
D. The metaphor contributes to a problematic scenario, which needs a
solution. The product can be a solution to the problem. Mark it as Needs-
Highlighting Metaphor (D).
E. The metaphor contributes to a beneficial scenario where the product can
be a platform/motive to achieve the benefits. Mark it as Needs-
Highlighting Metaphor (E).
F. There is no clear relationship between the metaphor and the product. But
the metaphor is eye-catching/entertaining/fancy. Mark it as Needs-
Highlighting Metaphor (F).
Tab. 3: A coding scheme for metaphor function (Pan 2022: 177−178)
Pan/Tay: Associations between multimodal metaphor functions and signals
71
3.3 Analysis
Statistical analyses were carried out to address research questions. First, to
compare the frequency of metaphorical ads and literal ads, a Chi-Square Goodness
of Fit was performed (Tay 2017). Then, a set of categorical data analytics,
including Log-Linear Analysis with Chi-Squared Decomposition and Multiple
Correspondence Analysis (MCA), was performed to uncover latent patterns in
the associations between metaphor signals, functions, and types of products.
Log-Linear Analysis provides statistical evidence for associations among more
than two categorical variables by examining the extent to which these variables
fit a linear model (Christensen 1990; Pan 2022; Tay 2018). MCA involves visualizing
the associations between more than two categorical variables as geometrical
distance, illustrating how each category contributed to the overall
variance (Le Roux/Rouanet 2010; Pan 2022; Tay 2020). This study will use MCA
as a visual aid to present associations among variables, providing viewers with
a general picture. The statistical details of associations were further illustrated
by the Chi-square Test of Independence (Pan 2022; Tay 2017). These statistical
tests were performed with R.10 Programming codes are provided in the
appendix.
4. Results and discussion
This section presents the general findings regarding the frequency of metaphorical
video ads in the corpus. Subsequently, it delves into the associations
between metaphor signals, functions, and product types from overall relationships
to binary associations. The latent patterns underlying the unstructured
multimodal data were revealed through statistical analyses, which included
Log-Linear Analysis with Chi-Squared decomposition and MCA.
4.1 Frequency of metaphorical ads
We employed the Chi-Square Goodness of Fit Test using the chisq.test()
function to compare the frequency of metaphorical videos (N = 66) with the nonmetaphorical
video (N = 34) in contemporary mass media. The results show a
significant difference in the adoption of metaphors within the initial corpus of
10 https://www.r-project.org/.
metaphorik.de 35/2024
72
100 ads, X2(1, 34) = 10.24, p = 0.001, indicating a notable prevalence of metaphors
in general video ads. This addresses the first research question.
4.2 Overall associations
The overall associations among three categorical variables, i.e. metaphor signals
(S), metaphor functions (U), and product types (PT) was first explored by Log-
Linear Analysis. The analyses used the LOGLINEAR function in the
Crosstabs.Loglinear package11. The Log-Linear analysis identifies significant
associations among three variables through a process of ‘backward elimination’.
Initially, all variables are assumed to be associated, and non-significant
associations are eliminated in a stepwise fashion until a final list of surviving
associations is obtained which best captures the relationships in the data. The
analysis begins with Step 0, which examines the three-way interaction among
PRODUCT TYPE * FUNCTIONS * SIGNALS. The results indicate that this
three-way interaction is not significant (p = .07), suggesting that removing it
does not significantly affect the fit between observed frequencies and the remaining
effects. Moving on to Step 1, the analysis explores the impact of deleting
bivariate associations. The results show significant bivariate interactions
PRODUCT TYPE * FUNCTIONS (p <.001), PRODUCT TYPE * SIGNALS
(p = .003), and USES * SIGNALS (p <.001). These findings suggest that the
bivariate associations play a significant role in the model fit. The absence of a
three-way interaction suggests that the associations between two variables
remain consistent regardless of the third variable (Field 2018). The likelihood
ratio showed a good fit of this model, X2 (8) = 14.08, p = .08. Table 4 shows the
crosstabulation for the three variables. Table 5 shows backward elimination
statistics.
PRODUCT
TYPE (PT)
FUNCTIONS
(U)
SIGNALS
Transformation
Replacement
Juxtaposition
Simultaneous
Cueing
Non-
Existing
Gestalt
Experience Features-U 7 2 5 5 1
Supporting 3 9 5 8 0
Needs-U 0 0 14 12 13
11 https://oconnor-psych.ok.ubc.ca/loglinear/LOGLINEAR_vignettes.html.
Pan/Tay: Associations between multimodal metaphor functions and signals
73
Search Features-U 1 9 21 9 5
Supporting 1 7 4 11 4
Needs-U 0 12 6 8 11
Tab. 4: Crosstabulation for PRODUCT TYPE, FUNCTION and SIGNALS
Stepa GenDel Effects LR_Chi_Square df p
0 Generating
Class
Product Types:
Functions: Signals
0 0 1
Deleted Effect Product Types:
Functions: Signals
14.082 8 0.07965
1 Generating
Class
All of these terms: 14.082 8 0.07965
Product Types
Functions
Signals
Product Types: Functions
Product Types: Signals
Functions: Signals
Deleted Effect
Test
Product Types: Functions 16.708 2 0.00024
Deleted Effect
Test
Product Types: Signals 15.67 4 0.0035
Deleted Effect
Test
Functions: Signals 50.036 8 0
Deleted On This
Step
None deleted
Tab. 5: Backward Elimination Statistics
a. The hierarchical backward elimination procedure begins with all possible terms in the
model and then removes, one at a time, terms that do not satisfy the criteria for remaining
in the model. A term is dropped only when it is determined that removing the term does
not result in a reduction in model fit AND if the term is not involved in any higher order
interaction. On each step above, the focus is on the term that results in the least-significant
change in the likelihood ratio chi-square if removed. If the change is not significant, then
the term is removed.
MCA is helpful in visualizing the underlying structure of overall associations
between more than two categorical variables, even in the absence of a three-way
interaction. The purpose of conducting MCA in this study is to capture a general
metaphorik.de 35/2024
74
picture of the associations before delving into the interpretation of statistical
results. Plots from MCA visualize associations as geometrical distance. MCA
describes patterns by locating each analytical unit as a point in a lowdimensional
space and grouping variable categories based on their distribution
(Le Roux/Rouanet 2010). Figure 8 is the MCA plot generated by packages of
FactoMineR and factoextra (Husson et al. 2017). The Figure displays the overall
associations between metaphor functions, signals and product types. The first
two dimensions typically reveal the most important underlying dimensions that
best capture the variance in the data (Tay 2020). In this case, the first dimension
accounts for 19.5% of the variance and the second dimension accounts for 18.6%,
capturing a total variance of 38.1%.
Fig. 8: MCA Map for Variable Categories
Exploring relations between categorical variables involves considering the
degree of closeness, angles from the origin, and locations in quadrant (Higgs
1991). In Figure 8, Search Product is clustered with Features-Highlighting use
and the signal of Juxtaposition, whereas Experience Product is closer to signals
of Replacement and Simultaneous Cueing. A far distance from the origin
suggests a big deviation from the expected proportions (Nenadic/Greenacre
2007). To further assist in result interpretation, Figure 8’s map visualizes the
Pan/Tay: Associations between multimodal metaphor functions and signals
75
variable contributions through color. The lesser the contribution of a variable is,
the colder the color. Likewise, a warm color of a variable indicates a strong
contribution to the variance. Notably, metaphor signals of Juxtaposition,
Transformation, and Non-Existing Gestalt are represented in warm colors and
their distinctive positioning in the quadrants sets them apart from other
variables. This suggests that they make substantial contributions to underlying
associations. Metaphor function variables also exhibit warm colors and are
positioned far from dimension 1, suggesting a potential strong contribution to
associations. In contrast, both types of products (Search Products and
Experience Products) are depicted in colder colors and located close to the
origin, signaling a limited contribution of product type to the underlying
associations.
4.3 Bivariate associations
While MCA produced a reader-friendly visualization of overall associations, a
closer examination of bivariate associations requires the application of the Chi-
Square Test of Independence. The absence of a three-way interaction in the
results of the Log-Linear Analysis indicates that bivariate associations remain
consistent regardless of changes in another variable (Tay 2018). The bivariate
associations were examined by Chi-Square decomposition, using the package
lsr (Navarro 2015).
4.3.1 The relationship between metaphor signals and functions
A Chi-Square Test of Independence was performed to compare the frequency
of five metaphor signals across three metaphor functions. The results show a
significant association between these two variables, X2 = (8, N = 197) = 38.59,
p < .001, Cramer’s V =.31, indicating that the occurrence of signals for different
functions was unlikely to be random. In addition to the p value, interpreting
effect size (Cohen 1988) is crucial. Effect size measures the magnitude of an
observed difference or relationship, providing a standardized measure of
practical significance regardless of sample size. In this case, the effect size
indicates a medium to large association (Cohen 1988).
Table 6 presents the counts and the adjusted residual for each variable. The
adjusted residual indicates the extent to which the occurrences deviated from
expectations. When the absolute value of the adjusted residual is above 2, the
metaphorik.de 35/2024
76
frequency contributes to significance. Green cells indicate pairs showing more
frequency than expected at a statistical level. Red cells show the opposite
condition. It can be observed that the signal Transformation of Images was used
more frequently than expected for Features-Highlighting metaphors but were
less frequently to signal Needs-Highlighting metaphors. Juxtaposition tends to
signal Features-Highlighting metaphors but not Supporting metaphors. Simultaneous
Cueing of Different Modes occurs more frequently to signal Supporting
metaphors. Depicting Non-Existing Gestalt tends to signal Needs-Highlighting
metaphors but is avoided when signaling Features-Highlighting metaphors.
There is no strong tendency to exploit the signal Replacement for different
metaphor functions.
Signals
Total
Transformation
Replacement
Juxtaposition
Non-
Existing
Gestalt
Simultaneous
Cueing
Functions
Features-U Count 8 11 26 6 14 65
Adjusted
Residual
***2.6 -1.7 ***3.3 ***-2.1 -1.2
Supporting Count 4 16 4 4 19 47
Adjusted
Residual
.8 1.8 ***-3.0 -1.8 ***2.4
Needs-U Count 0 21 20 24 20 85
Adjusted
Residual
***-3.1 .1 -.5 ***3.6 -.9
Total Count 12 48 50 34 53 197
Tab. 6: Signals * Functions Crosstabulation
The raw frequencies show that Simultaneous Cueing of Different Modes has the
highest counts, whereas Transformation of Images has the fewest occurrences.
These differences reach statistical significance, as confirmed by the Chi-Square
Goodness of Fit test, X2(4, 197) = 29.22, p < .001. The discrepancy may be
attributed to the inherent difficulty in creating such effects. Simultaneous
Cueing of Different Modes does not display visual incongruity, enhancing scene
coherence and naturalness. For instance, in the ad for FOTILE Range Hood,12 a
12 https://v.qq.com/x/page/q0628rrjd6u.html.
Pan/Tay: Associations between multimodal metaphor functions and signals
77
popular kitchen hood brand in China, visuals depict airport passengers waiting
for boarding. The spoken language states that these passengers are cooking
fumes and their luggage and coats are oil and grease. Without the spoken
language, the ad would be challenging to comprehend. The simultaneous
cueing of visuals and spoken language creates many Supporting metaphors,
such as COOKING FUMES ARE PASSENGERS, GREASE IS A COAT. These Supporting
metaphors guide the viewers in making sense of the final Features-Highlighting
metaphor FOTILE RANGE HOOD IS AN AIRPLANE. The target and source of a
Supporting metaphor, like cooking fumes and passengers are respectively related
to the target and source of the Features-Highlighting metaphor, i.e. FOTILE
Range Hood and an airplane. This function of metaphor may be less prevalent and
less salient in print ads due to space limitations. While visual incongruity aids
in detecting metaphors in multimodal discourse (Šorm/Steen 2018), especially
in static contexts like print ads and street art, in the case of video ads, the flexible
interactions among visuals, written text, spoken language, and music increase
the opportunities for metaphor creation without visual incongruity. Our
analyses not only showed the high frequency of such metaphors in video ads,
but also pointed out that they are likely to be employed to support a metaphorical
plot for the product.
Juxtaposition and Replacement were employed with similar frequency. No
strong tendency has been found for Replacement, suggesting that this technique
is utilized to create metaphors for different functions with an equal chance. The
signal Juxtaposition is frequently used to convey Features-Highlighting metaphors
but is avoided for Supporting metaphors. Juxtaposition presents two
entities in alignment, either within a single shot or across sequences, encouraging
viewers to compare these two things and draw connections between
them. Features-Highlighting metaphors typically involve products as targets,
conveying central claims about the product and seeking salience and attention
from the intended audience to prevent oversight. For instance, the juxtaposition
of a guitar and a feeding bottle in Figure 5 serves to highlight the similarities
between the guitar and the product, emphasizing its function of playing music.
The visual effects achieved through Juxtaposition contribute to explicitly displaying
metaphors about the product, whereas supporting metaphors may lean
towards natural and coherent visual effects to construct a cohesive scenario.
metaphorik.de 35/2024
78
It was observed that the signal Depicting Non-Existing Gestalt captures
viewers’ attention through its elaborate visual effects. This signal was frequently
employed to convey Needs-highlighting metaphors, either by accentuating
issues faced by viewers or by introducing entertaining elements. In the
realm of print ads, a hybrid gestalt might be portrayed (Van Mulken et al. 2010;
Gkiouzepas/Hogg 2011). In video ads, the availability of more sophisticated
filmic techniques allows for a dynamic presentation of non-existing gestalts.
Our corpus analysis showed that elaborate cartoon effects were commonly
utilized under this signal, as exemplified in Figure 9.
Fig. 9: A screenshot from the Ad Eleme13
The Transformation of Images occurred with the least frequency, likely
attributed to the high level of difficulty in achieving this effect. The visuals typically
depict the entire process wherein one entity (A) undergoes a transformation
into a different one (B). It was found that this signal usually is used to
convey Features-Highlighting metaphors, presenting the target and another
entity visually. In this context, entity (A) often serves as the source in the central
metaphor, while the product is designed as the target, entity (B). The punchline
usually follows this technique, bring the metaphor to its culmination. For
instance, Figure 2 illustrates how the central element in the metaphorical
scenario (the projector) undergoes a transformation into the product, Baidu
Cloud. Subsequently, the ad reaches its conclusion.
13 https://www.digitaling.com/projects/27286.html.
Pan/Tay: Associations between multimodal metaphor functions and signals
79
4.3.2 The relationship between product types and metaphor functions
A Chi-Square Test of Independence shows a significant association between
product types and metaphor functions, X2 = (2, N = 197) = 9.96, p = .007,
Cramer’s V = .23. The effect size, as indicated by Cohen (1988), demonstrates a
moderate tendency. In Table 7, it is evident that Needs-Highlighting metaphors
occur more frequent than expectation in ads for Experience products, while they
display less frequency in ads for Search products. This suggests that ads for
Experience products tend to leverage metaphors to create problematic scenarios
or employ fancy appeals to entertain viewers. On the other hand, ads for Search
products use metaphors comparatively less for these purposes. Features-
Highlighting metaphors are frequently shown in ads for Search products but
are avoided in ads for Experience products. This implies that ads for Search
products tend to design metaphors that involve the product as targets to
highlight product attributes, whereas ads for Experience products avoid to
present products in this way. No strong tendency is observed for Supporting
metaphors, indicating no preference for supporting metaphors in either type of
products. Our discussion, supported by ad examples, highlights that both types
of products utilize metaphors for all three functions. The statistical evidence
confirms latent associations, such as the occurrence of Features-Highlighting
metaphors being more frequent in ads for search products and Needs-
Highlighting metaphors being more frequent in ads for experience products.
These trends might not be easily discerned by qualitative analysis of a few cases.
Functions
Features-U Supporting Needs-U Total
Product
Type
Experience Count 20 20 48 88
Adjusted Residual **-2.7 -.3 **2.9
Search Count 45 27 37 109
Adjusted Residual **2.7 .3 **-2.9
Total Count 65 47 85 197
Tab. 7: Product Type * Functions Crosstabulation
metaphorik.de 35/2024
80
4.3.3 The relationship between metaphor signals and product types
A Chi-Square Test of Independence was conducted to examine the occurrence
of five metaphor signals in ads for Experience Products and Search Products.
The results indicate no statistically significant difference at the .05 significant
level, X2 = (4, N = 197) = 8.63, p = .07 Cramer’s V = .21. Since the p value achieves
the marginal significant level (p <.1), associations between pairs are discernible
through Table 8. Notably, the Transformation of Images signal exhibits a
tendency to occur more frequently in ads for experience products, while its
occurrence is notably lacking in ads for search products, as indicated by an
adjusted residual exceeding the absolute value of 2.0. No obvious trend is
observed for the other metaphor signals. The associations between the signal
Transformation of Images and the two types of products contributed to the
overall findings. However, these associations lacked robust support from
statistical evidence, suggesting that the design of metaphor signals is unlikely
to be influenced by the types of products. This aligns with the temporal and
dynamic nature of video ads, where metaphors have ample opportunities to be
utilized for various purposes.
Signals
Total
Transformation
Replacement
Juxtaposition
Non-
Existing
Gestalt
Simultaneous
Cueing
Product
Type
Experience Count 10 20 19 14 25 88
Adjusted
Residual
*2.8 -.5 -1.1 -.5 .4
Search Count 2 28 31 20 28 109
Adjusted
Residual
*-2.8 .5 1.1 .5 -.4
Total Count 12 48 50 34 53 197
Tab. 8: Product Type * Signals Crosstabulation
Pan/Tay: Associations between multimodal metaphor functions and signals
81
5. Conclusion
This study conducted a corpus-driven investigation into the structural traits of
multimodal metaphors in 66 metaphorical video ads. It employed systematic
methods to identify metaphors, signals, and functions, supported by inter-rater
reliability examinations. Categorical data analytics techniques, including Log-
Linear Analysis with Chi-Squared Decomposition, uncovered the latent associations
among product types, metaphor functions, and metaphor signals.
Multiple Correspondence Analysis visualized the associations in a readerfriendly
map. The response to the first research question indicates a significant
prevalence of metaphors in Chinese video ads. Regarding the second research
question, significant associations were observed between metaphor signals and
metaphor functions, product types and metaphor functions, but not between
product types and metaphor signals.
The analysis shows that the signal Simultaneous Cueing of Different Modes has
the highest raw frequency and is likely to convey Supporting metaphors. This
may be attributed to the absence of visual incongruity displayed by this signal,
making it an easily exploitable technique. Transformation of Images and Juxtaposition
tend to convey Features-Highlighting metaphors, while Depicting
Non-Existing Gestalt occurs more frequently when convey Needs-Highlighting
metaphors. Additionally, it was observed that search products are more inclined
to design Features-Highlighting metaphors, whereas experience products
tend to design Needs-Highlighting metaphors. This correlation may be
linked to the tangibility of products, where search products are visually
representable, while experience products are inherently abstract. However,
different types of products do not exhibit a preference for a specific metaphor
signal, possibly due to the ample resources and opportunities to craft a metaphor
in videos. This also aligns with our finding that the Supporting metaphor,
important for constructing metaphorical scenarios in ad plot, does not exhibit a
strong tendency for either type of product.
These findings make a valuable contribution to the manifestation and
characteristics of multimodal metaphor in video ads from an empirical
perspective. The study demonstrates how multimodal metaphors in video ads
can be systematically analyzed from a quantitative perspective, addressing
methodological challenges highlighted in previous works (Bateman/Hiippala
2020; Pérez-Sobrino 2017). Utilizing concrete filmic techniques as metaphor
metaphorik.de 35/2024
82
signals, the study showcases how these signals, in conjunction with associated
metaphor functions, convey key messages about the products. The identified
design patterns, emerging from this analysis, serve as a practical reference for
video ad designers in crafting multimodal metaphors for video ads. However,
the effectiveness of these design patterns in persuasion remains a subject for
further empirical examination. Additionally, the study’s findings offer
statistical evidence that aligns with earlier postulations from case studies,
confirming that metaphors for service products ‘behave’ differently (Forceville
2008). Specifically, products with abstract attributes are shown to exploit
metaphors in different functions compared to those with concrete attributes.
The limitations of this study are recognized. Given that the development of CMIPVA
and the analyses were conducted exclusively on Chinese video ads, it
is important to note that metaphor signals and the findings may not be
applicable to video ads in other countries. Culture, recognized as an important
factor, can exert a substantial influence on the design of metaphors and ads
(Forceville 2017; Pérez Sobrino/Littlemore/Samantha 2021). Future studies
could further explore this issue by studying a large corpus of video ads in a
different language.
6. References
Aaker, Jennifer L. (1997): “Dimensions of brand personality”, in: Journal of
Marketing Research 34 (3), 347−356.
Bateman, John/Hiippala, Tuomo (2020): “Statistics for Multimodality: Why,
When, How – an invitation”, SocArXiv January 18.
(https://doi.org/10.31235/osf.io/7j3np).
Bateman, John/Wildfeuer, Janina/Hiippala, Tuomo (2017): Multimodality:
Foundations, Research and Analysis – A Problem-Oriented Introduction, Berlin:
De Gruyter Mouton.
Beard, Elizabeth/Henninger, Nicole M./Venkatraman, Vinod (2022): “Making
Ads stick: Role of metaphors in improving advertising memory”, in:
Journal of Advertising 53 (1), 86−103.
Bloom, Paul N. (1989): “A decision model for prioritizing and addressing
consumer information problems”, in: Journal of Public Policy & Marketing
8 (1), 161−180.
Bloom, Paul N./Reve, Torger (1990): “Transmitting signals to consumers for
competitive advantage”, in: Business Horizons 33, 4, 58−66.
Pan/Tay: Associations between multimodal metaphor functions and signals
83
Bobrova, Larysa (2015): “A Procedure for Identifying Potential Multimodal
Metaphors in TV Commercials”, in: Multimodal Communication 4 (2),
113−131.
Bort-Mir, Lorena (2021): “Identifying metaphors in TV commercials with
FILMIP: The filmic metaphor identification procedure”, in: Journal of
English Studies 19, 23−46.
Burgers, Christian/Konijn, Elly A./Steen, Gerard J./Iepsma, Marlies A.R.
(2015): “Making ads less complex, yet more creative and persuasive: The
effects of conventional metaphors and irony in print advertising”, in:
International Journal of Advertising 34, 3, 515−532.
Cameron, Lynne/Maslen, Robert (2010): “Identifying metaphors in discourse
data”, in: Cameron, Lynne/Maslen, Robert (eds.): Metaphor analysis:
Research practice in applied linguistics, social sciences and the humanities,
London: Equinox, 97−115.
Chang, Chun-Tuan/Wu, Yuan-Ciao/Lee, Yu-Kang/Chu, Xing-Yu (2018):
“Right metaphor, right place: choosing a visual metaphor based on
product type and consumer differences”, in: International Journal of
Advertising 37 (2), 309−336.
Christensen, Ronald (1990): Log-linear Models, New York: Springer.
Delbaere, Marjorie/McQuarrie, Edward F./Phillips, Barbara J. (2011): “Personification
in advertising”, in: Journal of Advertising 40 (1), 121−130.
El Refaie, Elisabeth (2003): “Understanding visual metaphor: The example of
newspaper cartoons”, in: Visual Communication 2 (1), 75−95.
Fahlenbrach, Kathrin (2016a): “Audiovisual Metaphors as Embodied Narratives
in Moving Images”, in: Fahlenbrach, Kathrin (ed.): Embodied Metaphors in
Film, Television, and Video Games, London: Routledge, 33−50.
Forceville, Charles (1996): Pictorial Metaphor in Advertising, London: Routledge.
Forceville, Charles (1999): “The metaphor ‚COLIN IS A CHILD’ in Ian
McEwan’s, Harold Pinter’s, and Paul Schrader’s The Comfort of Strangers”,
in: Metaphor and Symbol 14 (3), 179−198.
Forceville, Charles (2007): “Multimodal metaphor in ten Dutch TV
commercials”, in: Public Journal of Semiotics 1 (1), 15−34.
Forceville, Charles (2008): “Pictorial and multimodal metaphor in
commercials”, in: McQuarrie, Edward F./Phillips, Barbara J. (eds.): Go
figure! New directions in advertising rhetoric, New York/London: ME Sharpe,
178−204.
metaphorik.de 35/2024
84
Forceville, Charles (2009): “Non-verbal and multimodal metaphor in a
cognitivist framework: Agendas for research”, in: Forceville, Charles/
Urios-Aparisi, Eduardo (eds.): Multimodal metaphor, Berlin/New York:
Mouton de Gruyter, 19−44.
Forceville, Charles (2016): “Pictorial and Multimodal Metaphor”, in: Klug,
Nina/Stöckl, Hartmut (eds.): Handbuch Sprache im multimodalen Kontext,
Berlin: Mouton de Gruyter, 241−261.
Forceville, Charles (2017): “Visual and multimodal metaphor in advertising:
Cultural perspectives”, in: Styles of Communication 9, 2, 26−41.
Forceville, Charles (2020): Visual and multimodal communication: Applying the
relevance principle, Oxford: Oxford University Press.
Forceville, Charles/Urios-Aparisi, Eduardo (eds.) (2009): Multimodal metaphor,
Berlin/New York: Walter de Gruyter.
Gkiouzepas, Lampros/Hogg, Margaret K. (2011): “Articulating a new
framework for visual metaphors in advertising”, in: Journal of Advertising
40 (1), 103−120.
Higgs, NT (1991): “Practical and innovative uses of correspondence analysis”,
in: Journal of the Royal Statistical Society: Series D (The Statistician) 40 (2),
183−194.
https://blog.hubspot.com/marketing/content-trends-global-preferences
(08/12/2023).
https://oconnor-psych.ok.ubc.ca/loglinear/LOGLINEAR_vignettes.html
(08/12/2023).
https://www.r-project.org/ (08/12/2023).
Husson, François/Le, Sébastien/Pagès, Jérôme (22017): Exploratory Multivariate
Analysis by Example Using R, New York: Chapman and Hall/CRC.
Iversen, Magnus Hoem (2017): “Employing film form and style in the
argumentative analysis of political advertising”, in: Tseronis,
Assimakis/Forceville, Charles (eds.): Multimodal Argumentation and
Rhetoric in Media Genres, Amsterdam: John Benjamins, 217−238.
Jiménez, Fernando R./Mendoza, Norma A. (2013): “Too popular to ignore: The
influence of online reviews on purchase intentions of search and
experience products”, in: Journal of Interactive Marketing 27 (3), 226−235.
Kövecses, Zoltán (2017): “Levels of metaphor”, in: Cognitive Linguistics 28 (2),
321−347.
Kress, Gunther R./Van Leeuwen, Theo (1996): Reading images: The grammar of
visual design, London: Routledge.
Pan/Tay: Associations between multimodal metaphor functions and signals
85
Krippendorff, Klaus (2004): “Reliability in content analysis: Some common
misconceptions and recommendations”, in: Human Communication
Research 30 (3), 411−433.
Lakoff, George (1993): “The contemporary theory of metaphor”, in: Ortony,
Andrew (ed): Metaphor and thought, Cambridge: Cambridge University
Press, 202−250.
Lakoff, George/Johnson, Mark (1980): Metaphors we live by, Chicago: Chicago
University Press.
Le Roux, Brigitte/Rouanet, Henry (2010): Multiple correspondence analysis,
Thousand Oaks: Sage.
Müller, Cornelia/Schmitt, Christina (2015): “Audio-visual metaphors of the
financial crisis: meaning making and the flow of experience”, in: Revista
Brasileira de Linguística Aplicada 15 (2), 311−342.
Navarro, Daniel (2013): Learning Statistics with R: A Tutorial for Psychology
Students and Other Beginners, Adelaide: University of Adelaide.
Nenadic, Oleg/Greenacre, Michael (2007): “Correspondence analysis in R, with
two-and three-dimensional graphics: The ca package”, in: Journal of
statistical software 20, 1−13.
Pan, Xie (2022): “Personification metaphors in Chinese video ads: Insights from
a data analytics approach”, in: Tay, Dennis/Pan, Xie (eds.): Data Analytics
in Cognitive Linguistics: Methods and Insights, Berlin/Boston: Mouton de
Gruyter, 169−196.
Pan, Xie/Chen, Zhuoan (2022): “When wars are good: Emotional unpacking
anti-coronavirus measures through metaphors in HK press conferences”,
in: Musolff, Andreas/Breeze, Ruth/Kondo, Kayo/Vilar-Lluch, Sara (eds.):
Pandemic and crisis discourse. Communicating COVID-19, London et al.:
Bloomsbury, 225−240.
Pan, Xie/Tay, Dennis (2021): “Identifying creative metaphor in video ads”, in:
Lin, Ling/Mwinlaaru, Issac/Tay, Dennis (eds.): Approaches to specialized
genres: In memory of Stephen Evans, New York: Routledge, 216−240.
Pan, Xie/Tay, Dennis (2023): “Individual differences in identifying creative
metaphors from video ads”, in: Metaphor and the Social World 13 (2),
221−247. (https://doi.org/https://doi.org/10.1075/msw.20016.pan).
Pérez-Sobrino, Paula (2017): Multimodal metaphor and metonymy in advertising,
Amsterdam: John Benjamins.
metaphorik.de 35/2024
86
Pérez-Sobrino, Paula/Littlemore, Jeannette (2017): “Facing methodological
challenges in multimodal metaphor research”, in: Baicchi, Annalisa/
Pinelli, Erica (eds.): Cognitive modeling in language and discourse across
cultures, Cambridge: Cambridge Scholars, 383−399.
Pérez Sobrino, Paula/Littlemore, Jeannette/Ford, Samantha (2021): Unpacking
Creativity: The Power of Figurative Communication in Advertising, Cambridge:
Cambridge University Press.
Pérez-Sobrino, Paula/Littlemore, Jeannette/Houghton, David (2019): “The role
of figurative complexity in the comprehension and appreciation of
advertisements”, in: Applied Linguistics 40 (6), 957−991.
Pérez Sobrino, Paula/Samantha, Ford (2023): “What counts as a multimodal
metaphor and metonymy? Evolution of inter-rater reliability across rounds
of annotation”, in: Language and Cognition 15 (4), 786−814.
Schilperoord, Joost (2018): “Ways with pictures: Visual incongruities and
metaphor”, in: Steen, Gerard (ed): Visual metaphor: Structure and process,
Amsterdam: John Benjamins, 11−46.
Šorm, Ester/Steen, Gerard J. (2018): “VISMIP: Towards a method for visual
metaphor identification”, in: Steen, Gerard (ed.): Visual metaphor: Structure
and process, Amsterdam: John Benjamins, 47−87.
Stampoulidis, Georgios/Bolognesi, Marianna/Zlatev, Jordan (2019): “A cognitive
semiotic exploration of metaphors in Greek street art”, in: Cognitive
Semiotics 12, 1, 20192008. (https://doi.org/10.1515/cogsem-2019-2008).
Tam, Audrey M/Leung, Clement H. C. (2001): “Structured natural-language
descriptions for semantic content retrieval of visual materials”, in: Journal
of the Association for Information Science and Technology 52, 930−937.
Tay, Dennis (2017): “Metaphor construction in online motivational posters”, in:
Journal of Pragmatics 112, 97−112.
Tay, Dennis (2018): “Metaphors of movement in psychotherapy talk”, in: Journal
of Pragmatics 125, 1−12.
Tay, Dennis (2020): “Co-constructing crisis with metaphor: A quantitative
approach to metaphor use in psychotherapy talk”, in: Huang, Mimi/
Holmgreen, Lise-Lotte (eds.): The language of crisis: Metaphors, frames, and
discourse, Amsterdam: John Benjamins, 231−255.
Van Mulken, Margot/Le Pair, Rob/Forceville, Charles (2010): “The impact of
perceived complexity, deviation and comprehension on the appreciation
of visual metaphor in advertising across three European countries”, in:
Journal of Pragmatics 42 (12), 3418−3430.
Pan/Tay: Associations between multimodal metaphor functions and signals
87
Verstraten, Peter (2009): Film narratology, Toronto/Buffalo/London: University
of Toronto Press.
William, Jennifer Marston (2017): Cognitive Approaches to German Historical Film:
Seeing is Not Believing, New York: Palgrave Macmillan.
Zlatev, Jordan (2015): “Cognitive semiotics”, in: Trifonas, Peter Pericles (ed.):
International Handbook of Semiotics, Berlin: Springer, 1043−1068.
7. Sources
http://www.tvcbook.com/showVideo.html?vid=70386&code=3af5MKotjdgM
4uoyPB4z_qESdgZGArMK4axYhYkgjeqhoA (08/12/2023).
http://www.vismet.org/VisMet/display.php#filter-Context-ctxt1
(08/12/2023).
https://osf.io/egrxh/?view_only=d5a5dcb734be42a09452d8fa91bfaeaa
(08/12/2023).
https://v.qq.com/x/page/q0628rrjd6u.html (08/12/2023).
https://v.qq.com/x/page/r0146rntgp7.html (08/12/2023).
https://v.youku.com/v_show/id_XMTg3NTYxODc2MA==.html?spm=a2h0k
.8191407.0.0&from=s1.8-1-1.2 (08/12/2023).
https://v.youku.com/v_show/id_XNzc0NzgwNDky.html?spm=a2h0k.81914
07.0.0&from=s1.8-1-1.2 (08/12/2023).
https://www.digitaling.com/projects/24017.html (08/12/2023).
https://www.digitaling.com/projects/27286.html (08/12/2023).
Appendix: Programming codes in R
Library (readxl)
#Bring in data
t <- read_excel(“y.xlsx”)
View(t)
#Codes for Log-Linear Analysis
library(Crosstabs.Loglinear)
LOGLINEAR(data = t,
data_type = 'counts',
variables=c('Products', 'Uses', 'Signals'),
Freq = NULL )
metaphorik.de 35/2024
88
#Codes for Chi-Square Test of Independence
install.packages(“lsr”)
library(gmodels)
library(lsr)
#Make a contingency table of two variables
pb = table (p$Products, p$Uses)
#chi-square test of independence
chisq.test(pb)
#effect size
cramersV(pb)
#View the adjusted residuals
CrossTable(p$Products,p$Uses, asresid = TRUE, format = “SPSS”)
#Codes for MCA
install.packages(c(“FactoMineR”, “factoextra”))
library(“FactoMineR”)
library(“factoextra”)
res.mca <- MCA(p, graph = FALSE)
print(res.mca)
get_eigenvalue(res.mca)
fviz_mca_var(res.mca, col.var = “contrib”,
gradient.cols = c(“#00AFBB”, “#E7B800”, “#FC4E07”),
repel = TRUE, # avoid text overlapping (slow)
ggtheme = theme_minimal())