The recent proliferation of online platforms has not only revolutionized the way people communicate and acquire information but has also led to propagation of malicious information (e.g., online human trafficking, spread of misinformation, etc.). Propagation of such information occurs at unprecedented scale that could ultimately pose imminent societal-significant threats to the public. To better understand the behavior and impact of the malicious actors and counter their activity, social media authorities need to deploy certain capabilities to reduce their threats. Due to the large volume of this data and limited manpower, the burden usually falls to automatic approaches to identify these malicious activities. However, this is a subtle task facing online platforms due to several challenges: (1) malicious users have strong incentives to disguise themselves as normal users (e.g., intentional misspellings, camouflaging, etc.), (2) malicious users are high likely to be key users in making harmful messages go viral and thus need to be detected at their early life span to stop their threats from reaching a vast audience, and (3) available data for training automatic approaches for detecting malicious users, are usually either highly imbalanced (i.e., higher number of normal users than malicious users) or comprise insufficient labeled data.

To address the above mentioned challenges, in this dissertation I investigate the propagation of online malicious information from two broad perspectives: (1) content posted by users and (2) information cascades formed by resharing mechanisms in social media. More specifically, first, non-parametric and semi-supervised learning algorithms are introduced to discern potential patterns of human trafficking activities that are of high interest to law enforcement. Second, a time-decay causality-based framework is introduced for early detection of “Pathogenic Social Media (PSM)” accounts (e.g., terrorist supporters). Third, due to the lack of sufficient annotated data for training PSM detection approaches, a semi-supervised causal framework is proposed that utilizes causal-related attributes from unlabeled instances to compensate for the lack of enough labeled data. Fourth, a feature-driven approach for PSM detection is introduced that leverages different sets of attributes from users’ causal activities, account-level and content-related information as well as those from URLs shared by users.

The United States is attempting to find the most efficient ways of responding to the threat of terrorist recruitment within its borders. ISIS has effectively recruited individuals from around the world on a large scale and specifically targets citizens of Western countries with high-quality, cinematic, English-language recruitment material. In the following analysis, we propose an additional approach to understanding ISIS recruitment appeal by comparing the content of recruitment messaging from a militaristic (but value-oriented) organization that is familiar to the authors of this thesis (the United States Marine Corps) with the militaristic but value-oriented unfamiliar (ISIS). Through this analysis, we seek to understand ISIS recruitment not from a theological basis but from a communications framework: narrative analysis. We identified narratives in each organization's recruitment materials and, by comparing larger themes that appeared across materials, determined the overarching narrative arc for each organization (into which the many smaller individual narratives were tied). We found that the narratives of the organizations are similar and different in many ways, but most significantly, they articulate fundamentally different resolutions: ISIS is driving towards a defined narrative resolution (which results in the end of the modern world) while the USMC recruitment materials depict no concrete resolution, as the organizational arc is depicted as continuing throughout time. Our discussion of narrative trajectory and defined resolutions directly supports existing scholarly literature linking the need for cognitive closure with extremist views: providing certainty and assurance about the future to potential extremist recruits. As demonstrated in our analysis, the narratives produced by ISIS for the purpose of recruitment depict a definite and conclusive resolution to both individual and organizational narratives, removing ambiguity (of actions, of antagonists, and of resolutions) and the anxiety associated with chance from the lives of the potential recruits. We believe ISIS's removal of uncertainty and provided template for how individuals should conduct their lives is an important part of the appeal its recruitment material has for Western recruits. Our suggestions for real-world use of our findings apply the immediacy and defined resolution found in ISIS recruitment narratives to counter ISIS-recruitment strategies.