• Tweets with populated place attributes from 55 European countries/territories collected using Tweepy (Roesslein 2015) from the Twitter Streaming API November 2016 – June 2017 (‘seed’ data, 153.2m tweets, 2.9m users)

• User timelines downloaded of users who are bi- or multilingual and can be reliably associated with a particular European country

  • tweet_source more likely to be human i.e. one of 10 widely-used clients (filter out Foursquare, Sandaysoft Cumulus, etc.)
  • At least 100 tweets in the ‘seed’ data
  • \(\geq\) 90% of tweets in the ‘seed’ data from one of the 55 countries/territories of interest

  • \(\geq\) 10% of tweets in one language for at least two languages

  • Once the user timlines were downloaded, retain only users for whom \(\geq\) 50% are from one country/territory
  • Retained timelines of 50,268 unique authors who published on Twitter between 2007 and 2017

Tweets identified as Albanian by cld2

##  [1] "rahhhhhh brenda didn't win"       
##  [2] "mdrrr j'en ai marre de lui"       
##  [3] "j'en ai marre je te jure mddr"    
##  [4] "ptdrrrr j'en ai marre de toi"     
##  [5] "j'en ai marre des gens cons ptn"  
##  [6] "j'en ai marre de mes tâches"      
##  [7] "j'en ai marre de ce gamin️"        
##  [8] "1010 j'en ai marre d'ce gamine"   
##  [9] "mdrrrr la flm j'te jure ."        
## [10] "robson il force à pas sortir kane"

• Use threshhold of minimum 10 word tokens

##  [1] "kosova maqedonia unë të dua unë do të jetë i pranishëm në tiranë në prill"                                         
##  [2] "keni dokumente të leshuara nga institucione të huaja për t i legalizuar aplikoni online nëpërmjet platformës"      
##  [3] "mj takon ministrin rritja e b p ekonomik përforcimi i b p ndërkufitar në foku"                                     
##  [4] "mj siguron i ofron shqipërisë çdo asistencë të mundshme për emergjencën e zjarreve"                                
##  [5] "regjistrohu në app vetëm duke përdorur të dhënat emër mbiemër dhe adresën e postës elektronike"                    
##  [6] "mbahet samiti i kartës së adriatikut me zëvëndës presidentin amerikan mike pence"                                  
##  [7] "ministri mirëpret nënshkrimin e traktatit të miqësisë mes maqedonisë dhe bullgarisë"                               
##  [8] "a e dini se dje ishte dita e fundit e edicionit po që u grumbulluan 22k faleminderit"                              
##  [9] " kryesia bullgare do t ju mbështesë në marrjen e vendimit politik për hapjen e procesit të negociatave zv km mpj"  
## [10] " shpresojmë në 2018 të kemi një b p edhe më të fokusuar duke konsideruar prioritetin e zgjerimit të be të kryesisë"

• Now almost no false positives in the data (49,961 users, 27,052,395 tweets)

• Introduced by Shannon (1948) to quantify the information content in character sequences
• Can be used to express the uniformity (non-uniformity) of a distribution (i.e. diversity)

\[ H_{user}^{\prime} = -\sum _{i=1}^{n}{p_{i}\log_{2} ({p_{i}})} \]

• For each user:

• \(p_{i}\) is the proportion of tweets in language \(i\)

• \(n\) is the number of languages that user tweets in

• The value ranges between 0 (no variation; user tweets in only one language) and \(\log_2{n}\) (user posts an equal number of tweets in all \(n\) languages)

• \(H^{\prime}=1\): User tweets an equal amount in 2 languages

• Entropy was calculated for the ~50k users in the data set

• Mean entropy values were caluclated by considering all users from a particular country or all users of a particular language (for languages used by at least 100 users)

\[ H_{country/language}^{\prime} = \frac{1}{N}\sum_{j=1}^N {p_{j}} \]

• For each country or language:

• \(p_{j}\) is the entropy of a user who tweets from that country or in that language (min 100 users)

• \(N\) is the number of users from that country / of that language

• The value ranges between 0 (all users are monolingual) to \(max(H_{user}^{\prime})\) (users from the country/of the language posts an equal number of tweets in all \(n\) languages)

• From this table a correlation coefficient \(\phi\) can be calculated:

\[ \phi_{ij}= \frac{(O_{11}O_{22}-O_{12}O_{21})}{\sqrt{R_{1}R_{2}C_{1}C_{2}}} \]

• \(\phi\) ranges in value from \(-1\) to \(1\). A t-statistic and p-values were calculated for the \(\phi\) scores for all language pairs