The omnipresent data, growing number of network devices, and evolving attack techniques have been challenging organizations’ security defenses over the past decade. With humongous volumes of logs generated by those network devices, looking for patterns of malicious activities and identifying them in time is growing beyond the capabilities of their defense systems. Deep Learning, a subset of Machine Learning (ML) and Artificial Intelligence (AI), fills in this gapwith its ability to learn from huge amounts of data, and improve its performance as the data it learns from increases. In this dissertation, I bring forward security issues pertaining to two top threats that most organizations fear, Advanced Persistent Threat (APT), and Distributed Denial of Service (DDoS), along with deep learning models built towards addressing those security issues. First, I present a deep learning model, APT Detection, capable of detecting anomalous activities in a system. Evaluation of this model demonstrates how it can contribute to early detection of an APT attack with an Area Under the Curve (AUC) of up to 91% on a Receiver Operating Characteristic (ROC) curve. Second, I present DAPT2020, a first of its kind dataset capturing an APT attack exploiting web and system vulnerabilities in an emulated organization’s production network. Evaluation of the dataset using well known machine learning models demonstrates the need for better deep learning models to detect APT attacks. I then present DAPT2021, a semi-synthetic dataset capturing an APT attackexploiting human vulnerabilities, alongside 2 less skilled attacks. By emulating the normal behavior of the employees in a set target organization, DAPT2021 has been created to enable researchers study the causations and correlations among the captured data, a much-needed information to detect an underlying threat early. Finally, I present a distributed defense framework, SmartDefense, that can detect and mitigate over 90% of DDoS traffic at the source and over 97.5% of the remaining DDoS traffic at the Internet Service Provider’s (ISP’s) edge network. Evaluation of this work shows how by using attributes sent by customer edge network, SmartDefense can further help ISPs prevent up to 51.95% of the DDoS traffic from going to the destination.

With the rapid development of both hardware and software, mobile devices with their advantages in mobility, interactivity, and privacy have enabled various applications, including social networking, mixed reality, entertainment, authentication, and etc.In diverse forms such as smartphones, glasses, and watches, the number of mobile devices is expected to increase by 1 billion per year in the future. These devices not only generate and exchange small data such as GPS data, but also large data including videos and point clouds. Such massive visual data presents many challenges for processing on mobile devices. First, continuously capturing and processing high resolution visual data is energy-intensive, which can drain the battery of a mobile device very quickly. Second, data offloading for edge or cloud computing is helpful, but users are afraid that their privacy can be exposed to malicious developers. Third, interactivity and user experience is degraded if mobile devices cannot process large scale visual data in real-time such as off-device high precision point clouds. To deal with these challenges, this work presents three solutions towards fine-grained control of visual data in mobile systems, revolving around two core ideas, enabling resolution-based tradeoffs and adopting split-process to protect visual data.In particular, this work introduces: (1) Banner media framework to remove resolution reconfiguration latency in the operating system for enabling seamless dynamic resolution-based tradeoffs; (2) LesnCap split-process application development framework to protect user's visual privacy against malicious data collection in cloud-based Augmented Reality (AR) applications by isolating the visual processing in a distinct process; (3) A novel voxel grid schema to enable adaptive sampling at the edge device that can sample point clouds flexibly for interactive 3D vision use cases across mobile devices and mobile networks. The evaluation in several mobile environments demonstrates that, by controlling visual data at a fine granularity, energy efficiency can be improved by 49% switching between resolutions, visual privacy can be protected through split-process with negligible overhead, and point clouds can be delivered at a high throughput meeting various requirements.Thus, this work can enable more continuous mobile vision applications for the future of a new reality.

Studies on underground forums can significantly advance the understanding of cybercrime workflow and underground economies. However, research on underground forums has concentrated on public information with little attention paid to users’ private interactions. Since detailed information will be discussed privately, the failure to investigate private interactions may miss critical intelligence and even misunderstand the entire underground economy. Furthermore, underground forums have evolved into criminal freelance markets where criminals trade illicit products and cybercrime services, allowing unsophisticated people to launch sophisticated cyber attacks. However, current research rarely examines and explores how criminals interact with each other, which makes researchers miss the opportunities to detect new cybercrime patterns proactively. Moreover, in clearnet, criminals are active in exploiting human vulnerabilities to conduct various attacks, and the phishing attack is one of the most prevalent types of cybercrime. Phishing awareness training has been proven to decrease the rate of clicking phishing emails. However, the rate of reporting phishing attacks is unexpectedly low based on recent studies, leaving phishing websites with hours of additional active time before being detected. In this dissertation, I first present an analysis of private interactions in underground forums and introduce machine learning-based approaches to detect hidden connections between users. Secondly, I analyze how criminals collaborate with each other in an emerging scam service in underground forums that exploits the return policies of merchants to get a refund or a replacement without returning the purchased products. Finally, I conduct a comprehensive evaluation of the phishing reporting ecosystem to identify the critical challenges while reporting phishing attacks to enable people to fight against phishers proactively.

Despite extensive research by the security community, cyberattacks such as phishing and Internet of Things (IoT) attacks remain profitable to criminals and continue to cause substantial damage not only to the victim users that they target, but also the organizations they impersonate. In recent years, phishing websites have taken the place of malware websites as the most prevalent web-based threat. Even though technical countermeasures effectively mitigate web-based malware, phishing websites continue to grow in sophistication and successfully slip past modern defenses. Phishing attack and its countermeasure have entered into a new era, where one side has upgraded their weapon, attempting to conquer the other. In addition, the amount and usage of IoT devices increases rapidly because of the development and deployment of 5G network. Although researchers have proposed secure execution environment, attacks targeting those devices can often succeed. Therefore, the security community desperately needs detection and prevention methodologies to fight against phishing and IoT attacks. In this dissertation, I design a framework, named CrawlPhish, to understand the prevalence and nature of such sophistications, including cloaking, in phishing attacks, which evade detections from the anti-phishing ecosystem by distinguishing the traffic between a crawler and a real Internet user and hence maximize the return-on-investment from phishing attacks. CrawlPhish also detects and categorizes client-side cloaking techniques in phishing with scalability and automation. Furthermore, I focus on the analysis redirection abuse in advanced phishing websites and hence propose mitigations to classify malicious redirection use via machine learning algorithms. Based on the observations from previous work, from the perspective of prevention, I design a novel anti-phishing system called Spartacus that can be deployed from the user end to completely neutralize phishing attacks. Lastly, inspired by Spartacus, I propose iCore, which proactively monitors the operations in the trusted execution environment to identify any maliciousness.

Many residences from student apartment units to family homes use a range of smart devices to make the day-to-day lives of the residents safer and more convenient. The ability to remotely access these devices has further increased their convenience, but it comes with the increased risk of vulnerable devices being exploited to achieve unauthorized access or to conduct surveillance on the users. This highlights the need for an access control system to securely restrict home device access to authorized users only. Existing approaches for securing smart homes use less secure authentication methods, do not allow for data ownership or fine-grained access control, and do not reliably store credential modification records, access records, or access policy modification records. These records can be a valuable resource to have available in the case of a security incident.In this thesis, a secure and efficient remote mutual authentication system with fine-grained access control integrating blockchain and digital signatures to authenticate users, authenticate the home gateway, and provide reliable auditing of the credential modifications, access history, and access policy modifications of the devices is presented. The immutability and verifiability properties of blockchain make it useful for securely storing these records. In this approach, a smart contract is created in the blockchain to keep track of authorized users, manage the access policy, and record requests for access or control of the home devices. A private blockchain is used to provide trust and privacy, which is necessary for a smart home system. Elliptic curve digital signatures are used to verify identities because the shorter key sizes and signature times are more adapted to Internet of Things contexts. The approach presented in this thesis is better than existing approaches because it provides fine-grained access control, and reliably stores credential modification records, access records, and access policy modification records. The approach was implemented and evaluated using Hyperledger, a private open-source blockchain, and the results show that this approach has significant additional security benefits with negligible additional overhead cost.

Cyberspace has become a field where the competitive arms race between defenders and adversaries play out. Adaptive, intelligent adversaries are crafting new responses to the advanced defenses even though the arms race has resulted in a gradual improvement of the security posture. This dissertation aims to assess the evolving threat landscape and enhance state-of-the-art defenses by exploiting and mitigating two different types of emerging security vulnerabilities. I first design a new cache attack method named Prime+Count which features low noise and no shared memory needed.I use the method to construct fast data covert channels. Then, I propose a novel software-based approach, SmokeBomb, to prevent cache side-channel attacks for inclusive and non-inclusive caches based on the creation of a private space in the L1 cache. I demonstrate the effectiveness of SmokeBomb by applying it to two different ARM processors with different instruction set versions and cache models and carry out an in-depth evaluation. Next, I introduce an automated approach that exploits a stack-based information leak vulnerability in operating system kernels to obtain sensitive data. Also, I propose a lightweight and widely applicable runtime defense, ViK, for preventing temporal memory safety violations which can lead attackers to have arbitrary code execution or privilege escalation together with information leak vulnerabilities. The security impact of temporal memory safety vulnerabilities is critical, but,they are difficult to identify because of the complexity of real-world software and the spatial separation of allocation and deallocation code. Therefore, I focus on preventing not the vulnerabilities themselves, but their exploitation. ViK can effectively protect operating system kernels and user-space programs from temporal memory safety violations, imposing low runtime and memory overhead.

The rampant occurrence of spam telephone calls shows a clear weakness of authentication and security in our telephone systems. The onset of cheap and effective voice over Internet Protocol (VoIP) technology is a major factor in this as our existing telephone ecosystem is virtually defenseless by many features of this technology. Our telephone systems have also suffered tremendously from a lack of a proper Caller ID verification system. Phone call spammers are able to mask their identities with relative ease by quickly editing their Caller ID. It will take a combination of unique innovations in implementing new authentication mechanisms in the telephone ecosystem, novel government regulation, and understanding how the people behind the spam phone calls themselves operate.<br/><br/>This study dives into the robocall ecosystem to find more about the humans behind spam telephone calls and the economic models they use. Understanding how the people behind robocalls work within their environments will allow for more insight into how the ecosystem works. The study looks at the human component of robocalls: what ways they benefit from conducting spam phone calls, patterns in how they identify which phone number to call, and how these people interact with each other within the telephone spam ecosystem. This information will be pivotal to educate consumers on how they should mitigate spam as well as for creating defensive systems. In this qualitative study, we have conducted numerous interviews with call center employees, have had participants fill out surveys, and garnered data through our CallFire integrated voice broadcast system. While the research is still ongoing, initial conclusions in my pilot study interview data point to promising transparency in how the voices behind these calls operate on both a small and large scale.

The use of reactive security mechanisms in enterprise networks can, at times, provide an asymmetric advantage to the attacker. Similarly, the use of a proactive security mechanism like Moving Target Defense (MTD), if performed without analyzing the effects of security countermeasures, can lead to security policy and service level agreement violations. In this thesis, I explore the research questions 1) how to model attacker-defender interactions for multi-stage attacks? 2) how to efficiently deploy proactive (MTD) security countermeasures in a software-defined environment for single and multi-stage attacks? 3) how to verify the effects of security and management policies on the network and take corrective actions?

I propose a Software-defined Situation-aware Cloud Security framework, that, 1) analyzes the attacker-defender interactions using an Software-defined Networking (SDN) based scalable attack graph. This research investigates Advanced Persistent Threat (APT) attacks using a scalable attack graph. The framework utilizes a parallel graph partitioning algorithm to generate an attack graph quickly and efficiently. 2) models single-stage and multi-stage attacks (APTs) using the game-theoretic model and provides SDN-based MTD countermeasures. I propose a Markov Game for modeling multi-stage attacks. 3) introduces a multi-stage policy conflict checking framework at the SDN network's application plane. I present INTPOL, a new intent-driven security policy enforcement solution. INTPOL provides a unified language and INTPOL grammar that abstracts the network administrator from the underlying network controller's lexical rules. INTPOL develops a bounded formal model for network service compliance checking, which significantly reduces the number of countermeasures that needs to be deployed. Once the application-layer policy conflicts are resolved, I utilize an Object-Oriented Policy Conflict checking (OOPC) framework that identifies and resolves rule-order dependencies and conflicts between security policies.

Due to the increase in computer and database dependency, the damage caused by malicious codes increases. Moreover, gravity and the magnitude of malicious attacks by hackers grow at an unprecedented rate. A key challenge lies on detecting such malicious attacks and codes in real-time by the use of existing methods, such as a signature-based detection approach. To this end, computer scientists have attempted to classify heterogeneous types of malware on the basis of their observable characteristics. Existing literature focuses on classifying binary codes, due to the greater accessibility of malware binary than source code. Also, for the improved speed and scalability, machine learning-based approaches are widely used. Despite such merits, the machine learning-based approach critically lacks the interpretability of its outcome, thus restricts understandings of why a given code belongs to a particular type of malicious malware and, importantly, why some portions of a code are reused very often by hackers. In this light, this study aims to enhance understanding of malware by directly investigating reused codes and uncovering their characteristics.

To examine reused codes in malware, both malware with source code and malware with binary code are considered in this thesis. For malware with source code, reused code chunks in the Mirai botnet. This study lists frequently reused code chunks and analyzes the characteristics and location of the code. For malware with binary code, this study performs reverse engineering on the binary code for human readers to comprehend, visually inspects reused codes in binary ransomware code, and illustrates the functionality of the reused codes on the basis of similar behaviors and tactics.

This study makes a novel contribution to the literature by directly investigating the characteristics of reused code in malware. The findings of the study can help cybersecurity practitioners and scholars increase the performance of malware classification.

The Java programing language was implemented in such a way as to limit the amount of possible ways that a program written in Java could be exploited. Unfortunately, all of the protections and safeguards put in place for Java can be circumvented if a program created in Java utilizes internal or external libraries that were created in a separate, insecure language such as C or C++. A secure Java program can then be made insecure and susceptible to even classic vulnerabilities such as stack overflows, string format attacks, and heap overflows and corruption. Through the internal or external libraries included in the Java program, an attacker could potentially hijack the execution flow of the program. Once the Attacker has control of where and how the program executes, the attacker can spread their influence to the rest of the system.

However, since these classic vulnerabilities are known weaknesses, special types of protections have been added to the compilers which create the executable code and the systems that run them. The most common forms of protection include Address SpaceLayout Randomization (ASLR), Non-eXecutable stack (NX Stack), and stack cookies or canaries. Of course, these protections and their implementations vary depending on the system. I intend to look specifically at the Android operating system which is used in the daily lives of a significant portion of the planet. Most Android applications execute in a Java context and leave little room for exploitability, however, there are also many applications which utilize external libraries to handle more computationally intensive tasks.

The goal of this thesis is to take a closer look at such applications and the protections surrounding them, especially how the default system protections as mentioned above are implemented and applied to the vulnerable external libraries. However, this is only half of the problem. The attacker must get their payload inside of the application in the first place. Since it is necessary to understand how this is occurring, I will also be exploring how the Android operating system gives outside information to applications and how developers have chosen to use that information.