Social identity correlates of social media engagement before and after the 2022 Russian invasion of Ukraine

Study 1: news sources before the invasion

Study 1 is a generalization of Rathje et al.⁷ to the context of pro-Ukrainian and pro-Russian news sources in Ukraine before the 2022 invasion. This study aims to test whether the correlates of engagement on US social media identified by Rathje et al.⁷ (i.e., mentions of ingroup and outgroup identity) hold up in a very different affectively polarized national context. Like Rathje et al.⁷, we collected data from Facebook and Twitter. Facebook is the most popular social media platform in Ukraine, while Twitter ranks fifth, behind YouTube, Instagram, and Telegram⁴⁸. We collected posts by the most popular pro-Ukrainian and pro-Russian news sources in Ukraine posted between 12 July 2021, and 24 February 2022, in Ukrainian or Russian (see Methods and Supplementary Information for details and the temporal distribution of the data as not all Twitter posts were collected). Following the original study, we classified a post as pro-Ukrainian (or pro-Russian) if it came from a pro-Ukrainian (or pro-Russian) news source and counted how many words in each post referred to Ukrainian and Russian identity and used negative, positive, or moral-emotional language⁶. We created dictionaries with mentions of Ukrainian and Russian identity that contained matter-of-fact references to the two nations: country name, capital, currency, decision-making center (i.e., Bankova and Kremlin), the 10 largest cities and ten most well-known politicians, and their language-specific morphological derivations. We used previously validated affective dictionaries (negative and positive language^49,50 and an existing dictionary of moral emotions⁶, which we translated into Ukrainian and Russian). Finally, we fit mixed effects linear regressions predicting log-transformed engagement (i.e., the log of the sum of all platform-specific reactions + 1) based on mentions of the ingroup and outgroup and negative, positive, and moral-emotional language, with news source as the random effect. We controlled for the follower count, URL, and media attachments, word count, and whether a tweet was a retweet. Like the original study, we hypothesized that mentions of the outgroup would have the strongest association with engagement across platforms (N = 468,310 and N = 114,557 posts, and N = 182,053 and N = 46,705 posts for pro-Ukrainian and pro-Russian Facebook, and pro-Ukrainian and pro-Russian Twitter, respectively). Across the three studies, statistical tests are two-tailed and the p-values and Cohen’s d are calculated using Satterthwaite d.f. Our results are robust to dictionary variations (Supplementary Table 23).

Similar to Rathje et al.⁷, the use of outgroup mentions in a social media post was the strongest predictor of engagement compared to ingroup mentions and emotional language in all data sets, except for pro-Russian Twitter. Controlling for all other factors, each additional outgroup word increased engagement by 4% to 23%. Ingroup language was associated with a 4% to 16% increase in engagement. Outgroup mentions predicted more engagement than ingroup mentions in three out of four datasets, with the exception of pro-Russian Twitter. Also replicating previous work^6,7,13, moral-emotional language significantly increased engagement by 3% to 6% across data sets. Positive and negative language predicted a 2% to 5% increase in engagement in all datasets, with the exception of negative language on pro-Russian Twitter, where we observed no effect. Our findings are broadly in line with the results reported by Rathje et al.⁷: using terms referring to the outgroup was strongly predictive of engagement on Ukrainian Facebook and Twitter, more so than ingroup terms and emotional language categories, before the 2022 invasion. A visual representation of the results can be found in Fig. 2, and complete regression tables are available in Supplementary Table 1.

Before Russia’s 2022 invasion of Ukraine, mentions of the outgroup were the strongest predictor of engagement on Ukrainian social media, as compared to ingroup mentions and negative, positive, and moral emotional words. The only exception was pro-Russian Twitter, where outgroup mentions had the same effect as ingroup language. a Predictors of engagement with posts by pro-Ukrainian news sources for each additional word in the corresponding dictionary (N = 468,310 Facebook posts; N = 182,053 tweets). b Predictors of engagement with posts by pro-Russian news sources for each additional word in the corresponding dictionary (N = 114,557 Facebook posts; N = 46,705 tweets). Data presented as exp(β) estimates with error bars representing 95% CIs. See Results Study 1 and Supporting Table 1 for full information.

Study 2: pro-Ukrainian news sources after the invasion

In Study 2, we explored how the correlates of social media engagement vary with time and during critical events – in this case, the outbreak of a full-scale war. Specifically, we investigated (1) whether the correlates of engagement identified by Rathje et al.⁷, i.e., ingroup and outgroup mentions, become more or less strongly associated and (2) whether ingroup solidarity or outgroup hostility³⁵ was associated with more engagement after the 2022 invasion (25 February 2022 – 13 September 2022; overall, N = 1,011,171 posts and N = 399,555 posts for Facebook and Twitter, respectively). We only studied pro-Ukrainian data, as Facebook and Twitter were banned in Russia shortly after the invasion, making the data limited and unlikely to be representative of authentic engagement patterns⁵¹.

After the start of the 2022 invasion, Ukrainian and Russian identity mentions became less strongly associated with engagement. Controlling for all else, outgroup mentions dropped from predicting a 16% increase, exp(β)  = 1.16, t(466964) = 63.89, p d = 0.19, 95% CI=[1.15, 1.17], to 7%, exp(β)  = 1.07, t(535719) = 41.71, p d = 0.11, 95% CI = [1.07, 1.08], on Facebook and from a 23% increase, exp(β)  = 1.23, t(182021) = 44.93, p d = 0.21, CI=[1.22, 1.24]), to 6%, exp(β)  = 1.056, t(217172) = 17.36, p d = 0.08, 95% CI = [1.05, 1.06], on Twitter (per each additional word and controlling for all other variables). Similarly, the effects of ingroup terms dropped slightly from an 11% increase, exp(β)  = 1.11, t(468248) = 62.27, p d = 0.18, 95% CI = [1.11, 1.12], to 4%, exp(β)  = 1.04, t(535714) = 26.27, p d = 0.07, 95% CI = [1.04, 1.05], on Facebook and from a 16% increase, exp(β)  = 1.16, t(182032) = 46.66, p d = 0.22, 95% CI = [1.15, 1.16], to 11%, exp(β)  = 1.11, t(217180) = 35.99, p d = 0.15, 95% CI=[1.11, 1.12], on Twitter. Facebook and Twitter posts were likely to gain 4–5% more engagement for each additional moral emotional word and 3-7% for each additional positive word. Negative words, however, had no significant effect on both social media platforms after the invasion. See Fig. 3a for a visual depiction and Supplementary Table 2 for all coefficients.

a Ingroup and outgroup mentions and negative, positive, and moral emotional language as predictors of engagement after the invasion on pro-Ukrainian Facebook and Twitter. b Regression coefficients for each of the binary categories of group identity language predicting the corresponding platform-specific reaction after the invasion on pro-Ukrainian Facebook and Twitter. Data presented as exp(β) estimates with error bars representing 95% CIs; both figures are based on N = 535,797 Facebook posts and N = 217,245 tweets. See Results Study 2 and Supporting Tables 1, 23, and 25 for full information.

Next, we explored what may have replaced the group identity mentions as a predictor of engagement after the invasion. We first manually inspected a series of Facebook posts with high amounts of engagement to see whether there were any obvious patterns in terms of what went viral. Based on this and drawing on recommendations from intergroup emotions theory to conceptually and empirically distinguish ingroup love from outgroup hate³⁵, we created two classifiers of emotionally-charged group identity: ingroup solidarity and outgroup hostility (note that these two are not mutually exclusive with group mentions)^32,35,52.

We trained two binary classifiers of ingroup solidarity and outgroup hostility (see Table 1 for post examples)³⁵. We operationalized ingroup solidarity in this context as expressing solidarity, liking, or unity of Ukraine or Ukrainians; for example, if a post praises Ukraine, mentions Ukrainians as competent, good people, or Ukraine as a great, strong, or united nation. Outgroup hostility was operationalized as expressing hostility, derogation, or dislike of Russia or Russians; for example, if a post criticizes Russia, mentions Russians as incompetent, immoral people, or the Russian Federation as a bad, weak, or failing nation. To create the models, the first author manually labeled 2000 posts from the datasets and used 1600 of them for training (see Supplementary Information for the code book). We then fine-tuned ingroup solidarity and outgroup hostility multilingual Natural Language Inference DeBERTa models (BERT-NLI models) following Laurer et al.⁵³, who showed that inference models require much less labeled data to achieve good classification results compared to traditional fine-tuning of BERT-style models and other supervised machine learning. Another advantage of this method is that we were able to provide the models with explicit definitions of ingroup solidarity and outgroup hostility. Both classifiers attained an accuracy of ~0.87 and an F1-macro of 0.80. We then tested how expressions of intergroup emotions and the binarized ingroup and outgroup mentions predicted engagement. We also conducted the same analysis with dictionaries instead of classifiers, yielding very similar results (see Methods and Supplementary Figs. 6 and 7 and Table 29). Our ingroup solidarity dictionary contained words that positively reference the ingroup (e.g., defenders, heroes), while the outgroup hostility dictionary had words that negatively refer to the outgroup (e.g., occupiers, invaders). We investigated the change in engagement patterns overall and using a sliding window time series approach. For each 14-day interval in the pro-Ukrainian data starting on the first day in our sample (i.e., between 12 July 2021 and 13 September 2022), we fit a mixed-effects linear regression predicting log-transformed engagement based on the binary categories of expressing ingroup solidarity, expressing outgroup hostility, binarized mentions of Ukrainian and Russian identity (i.e., binary indicators of whether a post contained a word referring to a group from Study 1), and negative, positive, and moral emotional word counts and control variables. We plotted the resulting time series of estimates, shifting it by half the window length (seven days) to accurately capture the timing of the effects.

Before the invasion, ingroup solidarity was associated with more engagement than outgroup hostility (Fig. 4). On both Facebook and Twitter, outgroup hostility briefly increased right before the invasion and dropped back down afterward. Ingroup solidarity, however, remained a strong predictor after the start of the invasion, hovering around a 95% increase on Facebook and a 65% increase on Twitter (Supplementary Fig. 2). Fitting one regression model to all of the pro-Ukrainian data from after the invasion, we find that, controlling for all else, posts that were classified as ingroup solidarity were likely to gain 92% more engagement, exp(β)  = 1.92, t(533141) = 151.95, p d = 0.42, 95% CI = [1.91, 1.94], on Facebook and 68% more engagement, exp(β) = 1.68, t(214803) = 96.31, p d = 0.42, 95% CI = [1.67, 1.70]) on Twitter overall (see Supplementary Table 3; Fig. 3b depicts how these language categories are associated with specific reactions). Meanwhile, outgroup hostility was likely to increase engagement by just 1%, exp(β) = 1.01, t(533140) = 3.32, p = 0.001, d = 0.01, 95% CI = [1.01, 1.02]), on Facebook and had no statistically significant effect on Twitter, exp(β) = 0.99, t(214803) = −1.11, p = 0.265, d = 0.01, 95% CI=[0.98, 1.01]). In other words, ingroup solidarity became the strongest correlate of social media engagement with news-related content on both Facebook and Twitter after the invasion and remained so for at least half a year. Descriptively, ingroup solidarity and outgroup hostility made up around 15% and 10% of content, respectively, on Facebook and 9% and 6% on Twitter before the invasion. After the invasion, the proportions jumped to 31% and 35% on Facebook and 21% and 25% on Twitter, respectively (Supplementary Tables 5 and 6).

Outgroup (Russian) identity mentions consistently predicted more engagement than ingroup (Ukrainian) mentions in our dataset of pro-Ukrainian news media accounts on Facebook before the 2022 invasion. However, after 24 February 2022, outgroup and ingroup mentions started predicting less engagement. Instead, ingroup solidarity emerged as the strongest predictor of engagement by a margin of more than 50%. A similar pattern was seen on Twitter (Supplementary Fig. 2). The dashed vertical line represents the invasion date (24 February 2022), and the content from that day is present in the gray window around it. The peak in ingroup solidarity coefficients around the end of August 2021 coincides with the celebrations of the Ukrainian Independence Day on August 24 – a highly identity-salient event supposed to elicit solidarity. Data presented as time series of exp(β) estimates (central lines) with 95% CIs (shaded regions around the central line); N = 1,011,171 Facebook posts.

Study 3: geolocated pro-Ukrainian Twitter after the invasion

In Study 3, we tested whether the post-invasion engagement patterns from Study 2 generalize to non-news social media data. We collected a dataset of original, non-replies tweets geolocated to Ukraine spanning July 2021 to September 2022. Unlike the previous two studies, where we used the orientation of the account of origin to classify posts into pro-Ukrainian or pro-Russian, in this study we used a fine-tuned RoBERTa model trained to classify posts into pro-Ukrainian or pro-Russian point of view about the war on 30 thousand manually labeled examples from seven social media platforms, including Twitter, achieving an accuracy of 0.86 and F1-macro of 0.81. Here, pro-Ukrainian does not mean positive about Ukraine or Ukrainians, but rather can mean any expression in support of Ukraine or opposition of Russia, including expressions that are negative about Russia or Russians (e.g., a sentence such as I hate Russia, for instance, would also be classified as pro-Ukrainian by this particular classifier). We then only analyzed the posts identified as pro-Ukrainian and posted after the invasion (N = 148,959 tweets). As the RoBERTa classifier was trained and evaluated exclusively on the data from after the invasion and thus does not necessarily generalize to the prior period, we cannot make any claims for the period before the invasion, but we provide this data for completeness (Supplementary Tables 4 and 8). We further fine-tuned our ingroup solidarity and outgroup hostility Study 2 BERT-NLI models on 1000 posts from pro-Ukrainian geolocated Twitter, achieving an F1-macro of 0.75 and 0.82, respectively. We fit a mixed effects linear regression predicting log-transformed engagement based on the same variables as in Study 2 (excluding whether a post had media attachments, which was not available) and whether the user was verified. Follower count and verified information were missing for some of the accounts in our data, but subsetting to only the data where we have full user information produces largely the same results (see Supplementary Table 22). This study is not intended to replicate the findings regarding changes from pre-invasion to post-invasion, but only the engagement patterns post-invasion.

We find that the post-invasion results from Study 2 are replicated in a non-news-specific dataset: after the start of the invasion, posts containing ingroup solidarity were likely to get 14%, exp(β)  = 1.14, t(147613) = 14.85, p d = 0.08, 95% CI = [1.12, 1.16], more engagement, as compared to 7%, exp(β) = 1.07, t(145862) = 8.01, p d = 0.04, 95% CI = [1.05, 1.09], for outgroup hostility, and 4%, exp(β) = 1.04, t(146191) = 4.87, p d = 0.03, 95% CI = [1.03, 1.06]), and 7%, exp(β)  = 1.07, t(147015) = 11.00, p d = 0.06, 95% CI = [1.06, 1.09], for the outgroup and ingroup mentions, respectively (see Supplementary Table 4 for all regression coefficients). These results suggest that our findings from Study 2 generalize beyond news content.