A Database of North American Multiple Modals from YouTube

Steven Coats
English, University of Oulu, Finland
steven.coats@oulu.fi

8th CMC-Corpora Conference
Radboud University, Nijmegen, 29 October 2021

Outline

1. Participatory framework of YouTube: Multimodal and multilevel possibilities for CMC corpora
2. Corpus of North American Spoken English (CoNASE)
3. Double modals: Identification and manual inspection/annotation
4. Preliminary linguistic results
5. Corpus-based study of self-repair: Table of North American Double Modals and Self-repairs
6. Summary, caveats, future outlook
   
   
   
   

Slides for the presentation are on my homepage at https://cc.oulu.fi/~scoats

A double modal in a YouTube video

Fairhope, Alabama
City of Fairhope City Directors' Roundtable September 10, 2018

Participatory framework of YouTube

• YouTube as a platform for mediated quasi-interaction with three communicative levels (Bou-Franch et al. 2012, Dynel 2014)
• First level: Face-to-face spoken interaction (cf. Jakobson 1960, Goffman 1981, Levinson 1988)
• Second level: Corresponds to classifications of mass media (cf. Bell 1991, Hutchby 2006)
• Third level: Additional modalities/affordances of platform (e.g. commenting)
• Multi-authored, multi-semiotic, dialogic and dynamic (Androutsopoulos & Tereick 2016)

	Primary	Secondary	Tertiary
Location	Local council room	Council room/Internet	Internet
Participants	Council members, members of the public	YouTube viewers	YouTube viewers
Medium	Speech	Audiovisual recording	Text, audio-visual content
Producers/speakers	Councilmembers, members of the public	Councilmembers, members of the public (governmental organization)	YouTube viewers
Listeners/hearers	Councilmembers, members of the public	YouTube viewers	YouTube viewers

Traditional corpora of CMC

• Primarily written genres such as email; SMS messages; chats; forum, bulletin board, Reddit, or instant messaging posts; wiki discussion pages, blogs, Twitter and other social media (cf. Beißwenger et al. 2012, Herring 2007)
• Often analyzed using the tools of corpus linguistics

Multimodal and multilevel corpora of CMC

• In addition to texts/transcripts, may contain static images, audio and video recordings including live streams, video/webcam chats, Zoom/Teams meetings
• May allow analysis of features of speech, physical presence, and gesture
• Analysis of acoustic properties (vowel quality/formant extraction, VOT, articulation rate and other prosodic features), visual aspects (facial expressions, posture, gesture, etc.)
• Analysis of multilevel communicative phenomena such as comment threads, like/dislike ratios, view counts, subscriptions, interaction with streaming viewers ("write in the chat")

Corpus of North American Spoken English (CoNASE)

• 1,252,066,371 words
• 301,847 Automatic Speech Recognition (ASR) transcripts from 2,572 YouTube channels, corresponding to 154,041 hours of video
• Transcripts from all 50 US states, D.C., all 13 Canadian provinces/territories (Coats 2019, 2020)
• Primarily recordings of meetings of local government bodies
• Also interviews, informational videos, news reports, community announcements, recordings of sports events, etc.
• Includes transcripts of live-streamed meetings and multilevel interactive phenomena
• Starting point for multimodal corpus

WebVTT file

CoNASE data collection and geolocation

• Channel selection with YouTube API, transcripts with Youtube-DL
• Channel selection by scraping municipal websites (Coats 2021a)
• Use of Tor for http requests to YouTube search and for YouTube-DL
• Pass channel name + state/province/territory name to geolocator

CoNASE channel locations

• After additional filtering steps (only transcripts with word timing, only transcripts ≥ 100 words, only ASR transcripts, no translations)

Excerpt from corpus table

df_pos_t

	country	state	channel_title	video_title	video_id	name	type	channel_id	channel_username	video_length	location	address	nr_words	text_pos	latlong
0	USA	Alabama	Baldwin County Alabama	Loxley School	KAFtEkKE_ik	COUNTY OF BALDWIN	general purpose	UCVEowdBqDvlT_TJDjf8YuyA	BCCommission	1651.38	Baldwin County, AL, USA	Baldwin County, AL, USA	3513	the_DT_25.619 Baldwin_NNP_26.619 County_NNP_27...	(30.6010744, -87.77633329999999)
1	USA	Alabama	Baldwin County Alabama	2019 Baldwin County Sewer Utilities Informatio...	6KvmxXRoMGA	COUNTY OF BALDWIN	general purpose	UCVEowdBqDvlT_TJDjf8YuyA	BCCommission	6110.09	Baldwin County, AL, USA	Baldwin County, AL, USA	16090	welcome_UH_1.37 everybody_NN_2.37 glad_JJ_3.89...	(30.6010744, -87.77633329999999)
2	USA	Alabama	Baldwin County Alabama	Heritage Museum of Baldwin County Alabama	bGL_FlSulZ4	COUNTY OF BALDWIN	general purpose	UCVEowdBqDvlT_TJDjf8YuyA	BCCommission	1255.31	Baldwin County, AL, USA	Baldwin County, AL, USA	2318	located_VBN_22.039 in_IN_23.039 the_DT_23.13 t...	(30.6010744, -87.77633329999999)
3	USA	Alabama	Baldwin County Alabama	AL State Veterans Memorial Cemetery at Spanish...	BX18sxhdL5Y	COUNTY OF BALDWIN	general purpose	UCVEowdBqDvlT_TJDjf8YuyA	BCCommission	4376.31	Baldwin County, AL, USA	Baldwin County, AL, USA	7991	good_JJ_15.94 morning_NN_16.94 everyone_NN_17....	(30.6010744, -87.77633329999999)
4	USA	Alabama	Baldwin County Alabama	USO Canteen Dance Re-enactment	VqCvFSku9Es	COUNTY OF BALDWIN	general purpose	UCVEowdBqDvlT_TJDjf8YuyA	BCCommission	3893.26	Baldwin County, AL, USA	Baldwin County, AL, USA	6846	[Music]_XX_5.1 [Music]_XX_14.86 [Music]_XX_104...	(30.6010744, -87.77633329999999)

Multiple modals in North America

Non-standard feature of spoken language in the Southern United States (also in Scotland/N. England, Caribbean Englishes/creoles)

• Y'all think you might could help me?
• I may can help you

Studied in the context of

• Dialectology and sociolinguistics (Feagin 1979; Labov 1972, Mishoe & Montgomery 1994, Montgomery 1998)
• Syntax (Batistella 1995; Di Paolo 1989; Hasty 2012a, 2012b, 2014; Nagle 2003)
• Language history (Fennell & Butters 1996, Montgomery & Nagle 1994)
• English varietal typology (Zullo et al. 2021)

and from other perspectives

Previous research

For most studies, data is not naturalistic and limited in geographical scope

• Acceptability judgment surveys, sentence completion tasks (e.g. Butters 1973; Di Paolo 1989; Hasty 2011, 2012a, 2012b, 2014; Williamson 2018)
• Regional linguistic atlases (e.g. Montgomery 1989, 1998; Mishoe & Montgomery 1994)
• Observational studies (e.g. Labov 1972, Feagin 1979, Mishoe & Montgomery 1994)

"Double modals occur with very low frequency in real-life utterances, and they are quite difficult to elicit in sufficient quantities and in a reliable fashion" (Fennell & Butters 1996: 265)

Identifying double modals

Corpus search → Creation of a table of occurrences with URLs: Every instance can be checked

13 modals/semi-modals:

can, could, may, might, must, shall, should, will, would, ought to, oughta, 'll, used to

= 156 possible double modals (excluding repititions)

Regular expression

we_PRP_10.0 might_MD_10.2 could_MD_10.7

Regex for timed-token corpus:

"\\s+(mightn?_\\w+_\\S+\\s+(?:i?_\\w+_\\S+\\s+|we?_\\w+_\\S+\\s+|you?_\\w+_\\S+\\s+|he?_\\w+_\\S+\\s+|she?_\\w+_\\S+\\s+|it?_\\w+_\\S+\\s+|they?_\\w+_\\S+\\s+|haven?_\\w+_\\S+\\s+|'ve_\\w+_\\S+\\s+|'t_\\w+_\\S+\\s+|not_\\w+_\\S+\\s+){0,3}couldn?_\\w+_\\S+\\s+(?:haven?_\\w+_\\S+\\s+|'ve_\\w+_\\S+ |not_\\w+_\\S+\\s+|'t_\\w+_\\S+\\s+)?)"

This matches

• might could, including negations, perfect constructions, questions/inverted constructions with personal pronouns might not could, might have could, mightn't she could, etc.

False positives

Many two-modal sequences are not actually "true" double modals

• "Food from a can would not taste as good" (homonym)
• "Will can help you with that" (personal name/homonym)
• "If I may, could I remark..." (clause/phrase overlap)
• "We will, if we can." "Would it cost a lot?" (speaker overlap)
• ASR errors ("Mike" = "might", "we'll" = "will", many others)
• "That woul- could be expensive" (self-repair, disfluency)
• "If you will... um... can help me" (self-repair, filled pause)
• Audio errors (video cuts off)
• Video unavailable

Manual verification

• To filter out false positives, manually annotate a subset of the regex search hits
• Disfluencies often marked by discrepancies in word timing or prosody (Levelt 1983, Postma 2000, Lickley 2015)
• If there are no disfluencies and the speech is semantically and logically coherent and consistent, it is a "true" double modal t
• Otherwise, annotate as sr self-repair, o overlap (phrase/clause boundary or speaker), a ASR error, u audio unintelligible, i video/transcript error

Double modals: Ocurrence data frame

• Editable html page with link to video of each DM instance found in the corpus at time of utterance
• Every occurrence is available for instant inspection

state	channel	group	dm	url	type	notes
Alabama	City of Fairhope	may can	may can	https://youtu.be/CgDl04nUGO0?t=1068	t
Alabama	City of Fairhope	may can	may can have	https://youtu.be/wjzNjPSI8es?t=1587	t	"we may, we may can have some"
Alabama	City of Mobile, AL	may can	may not can	https://youtu.be/aUamq_HG8Ak?t=3103	t	emphatic (repeated) AA
Alabama	BrewtonTV	could will	could will	https://youtu.be/MSQo1kOZoSM?t=526	sr d	"and the experts say it cou- will continue"

A few preliminary linguistic results (Coats 2021b)

• Double modals are mostly Southern, but can be found everywhere in North America, including in places not previously attested (West, North, New England, Canada)
• There are more double modal combinations than has been previously thought
• DMs result from changing semantics of modals in N. America? (Coates 1983, Biber et al. 1999, Facchinetti et al. 2003, Gresset 2003, Leech 2003, Leech et al. 2009, Myhill 1995)

→ "Careful epistemicity" with double forms?

Might can and might could, local autocorrelation measure

Normalized frequencies for manually-verified occurrences of might could and might can used to calculate Getis-Ord G^*_i values based on a 100-nearest-neighbor binary weights matrix

These characteristic double modals are concentrated in the US Southeast

Corrections/self-repairs

In the Conversation Analysis tradition (Schegloff et al. 1977, Schegloff 1987, Jefferson 2018): Self-repair considered in descriptive studies of detailed individual transcripts in which the analytical emphasis is often on the "work" accomplished by the repair move

Table of North American Double Modals and Self-repairs

• 1,056 "true" double modals, 1,016 self-repairs

A repaired modal

Groton, Connecticut
Groton Town Council C.O.W. - 11/14/17

Corpus-based study of self-repair

• Discourse and semantic context of utterance
• Careful negotiation, hypothetical situations, epistemic modality

Double modals as fossilized self-repairs?

Summary, caveats, outlook

• Double modals and self-repairs were extracted from CoNASE using regex and manual annotation
• Many possibilities for corpus-based study of spoken English, including conversational phenomena and multilevel CMC features
• Method gets aggregate transcripts (no diarization)
• Word error rates can be high, ASR algorithm is a "black box" and changes over time (Coats 2021a)
• False negatives , manual judgments of one annotator
• Disfluencies are not always marked prosodically: How do we distinguish between covert repairs and fluent production of utterances generated by a speaker using a non-standard grammar?
• Analysis of self-repairs: Discourse and semantic contexts
• Automatic extraction and analysis of vowels in North American English
• CoNASE can be used to quickly find naturalistic recordings of rare speech phenomena
• Customized corpora for more specific tasks ("multilevel" phenomena)

Thanks for your attention!

References

Androutsopoulos, J. & Tereick, J. (2016). YouTube: Language and discourse practices in participatory culture. In Georgakopoulou, A. & Spilioti, T. (eds.), The Routledge handbook of language and digital communication, 354–370. Routledge.

Battistella, E. (1995). The syntax of the double modal construction. Linguistica Atlantica 17, 19–44.

Beißwenger, M., Ermakova, M., Geyken, A., Lemnitzer, L. & Storrer, A. (2012). A TEI Schema for the Representation of Computer-mediated Communication. Journal of the Text Encoding Initiative 3. https://doi.org/10.4000/jtei.476

Bell, A. (1991). The language of news media. Blackwell.

Biber, D., Johansson, S., Leech, G., Conrad, S. & Finegan, E. (1999). Longman Grammar of Spoken and Written English. Pearson Education.

Bou-Franch, P., Lorenzo-Dus, N. & Garcés-Conejos Blitvich, P. (2012). Social interaction in YouTube text-based polylogues: A study of coherence. Journal of Computer-mediated Communication 17, 501–521.

Butters, R. (1973). Acceptability judgments for double modals inSouthern dialects. In Bailey, R. & Shuy, R. (eds.), New ways of analyzing variation in linguistics, 276–286. Georgetown University Press.

Coates, J. (1983). The semantics of the modal auxiliaries. Croom Helm.

Coats, S. (2019). A corpus of regional American language from YouTube. In Navarretta, C. et al. (eds.), Proceedings of the 4th Digital Humanities in the Nordic Countries Conference, Copenhagen, Denmark, March 6–8, 2019 (pp. 79–91). CEUR.

Coats, S. (2020). Articulation rate in American English in a corpus of YouTube videos. Language and Speech 63(4), 799–831. https://doi.org/10.1177/0023830919894720

Coats, S. (2021a). Dialect corpora from YouTube. Manuscript submitted for publication.

Coats, S. (2021b). Naturalistic double modals in North America. Manuscript submitted for publication.

Di Paolo, M. (1989). Double modals as single lexical items. American Speech 64(3), 195–224.

Dynel, M. (2014). Participation framework underlying YouTube interaction. Journal of Pragmatics 73, 37–52. https://doi.org/10.1016/j.pragma.2014.04.001.

Feagin, C. (1979). Variation and change in Alabama English: A sociolinguistic study of the White community. Georgetown University Press.

Goffman, E. (1981). Forms of talk. Blackwell.

Gresset, S. (2003). Towards a contextual micro-analysis of the non-equivalence of might and could. In Facchinetti, R., Palmer, F. & Krug, M. (eds.), Modality in contemporary English, 81–102. De Gruyter Mouton.

Hasty, J. D. (2011). I might not would say that: A sociolinguistic study of double modal acceptance. University of Pennsylvania Working Papers in Linguistics 17(2), 91–98.

Hasty, J. D. (2012a). This might could help Us better understand syntactic variation: The double modal construction in Tennessee English.PhD diss., Michigan State University.

Hasty, J. D. (2012b). We might should oughta take a second look at this: A syntactic re-analysis of double modals in Southern United States English. Lingua 122(14), 1716–1738.

Hasty, J. D. (2014). We might should be thinking this way: Theory and practice in the study of syntactic variation. In Zanuttini, R. & Horn, L. R. (eds.), Micro-syntactic variation in North American English, 269–293. Oxford University Press.

Herring, S. (2007). A faceted classification scheme for computer-mediated discourse. Language@Internet 4. https://www.languageatinternet.org/articles/2007/761/

Hutchby, I. (2006). Media talk: Conversation analysis and the study of broadcasting. Open University Press.

Jakobson, R. (1960). Closing statements: Linguistics and Poetics. In Sebok, T. A. (ed.), Style in language, 350–377. Wiley

Labov, W. (1972). Language in the inner city: Studies in the Black English vernacular. University of Pennsylvania Press.

Leech, G. (2003). Modality on the move: The English modal auxiliaries 1961–1992. In Facchinetti, R., Palmer, F. & Krug, M. (eds.), Modality in contemporary English, 223–240. De Gruyter Mouton.

Leech, G., Hundt, M., Mair, C. & Smith, N. (2009). Change in contemporary English: A grammatical study. Cambridge University Press.

Levelt, W. J. M. (1983). Monitoring and self-repair in speech. Cognition 14(4), 41–104.

Levinson, S. (1988). Putting linguistics on a proper footing: Explorations in Goffman's participation framework. In Drew, P., Wootton, A. (eds.), Erving Goffman: Exploring the Interaction Order, 161–227. Polity Press.

Lickley, R. J. (2015). Fluency and disfluency. In M. Redford (ed.), The handbook of speech production, 445–469. Wiley-Blackwell.

Mishoe, M. & Montgomery, M. (1994). The pragmatics of multiple modal variation in North and South Carolina. American Speech 69(1), 3–29.

Montgomery, M. (1989). Exploring the roots of Appalachian English. English World-Wide 10(2), 227–278.

Montgomery, M. (1998). Multiple modals in LAGS and LAMSAS. In Montgomery, M. & Nunnally, T. E. (eds.), From the Gulf States and beyond: The legacy of Lee Pederson and LAGS, 90–122. University of Alabama Press.

Montgomery, M. & Nagle, S. J. (1994). Double modals in Scotland and the Southern United States: Trans-Atlantic inheritance or independent development? Folia Linguistica Historica, 14(1-2), 91–108.

Myhill, John. (1995). Change and continuity in the function of the American English modals. Linguistics 33, 157–211.

Nagle, S. J. (2003). Double modals in the Southern United States: Syntactic structure or syntactic structures? In In Facchinetti, R., Palmer, F. & Krug, M. (eds.), Modality in contemporary English, 349–371. De Gruyter Mouton

Postma, A. (2000). Detection of errors during speech production: a review of speech monitoring models. Cognition 77, 97–131.

Schegloff E. A., Jefferson G. & Sacks H. (1977). The preference for self-correction in the organization of repair in conversation. Language 53, 361–382.