BotCatch: A Behavior and Signature Correlated Bot
Detection Approach
Yuede Ji and Yukun He
College of Computer Science and Technology,
Jilin University,
Changchun Jilin, China,
Email: {jiyd12, heyk12}@mails.jlu.edu.cn
Qiang Li and Dong Guo
College of Computer Science and Technology,
Jilin University,
Changchun Jilin, China,
Email: {li qiang, guodong}@jlu.edu.cn
Abstract—Botnet has become one of the most serious threats
to Internet security. Compared with network-based bot detection
approaches, host-based approaches can discover more insights
of unknown bots, and we may completely eliminate bots if we
can successfully detect them on end-host. Host-based approaches
mainly include signature and behavior-based detection approach-
es. In this paper we propose a behavior and signature correlated
bot detection approach, BotCatch. Firstly, we present the design
of BotCatch. There are four components in BotCatch: analysis
engine, signature analysis engine, behavior analysis engine, and
correlation engine. The analysis engine assigns the suspicious
sample to signature analysis engine and behavior analysis engine.
These two engines analyze the sample and generate signature and
behavior analysis result. Then correlation engine correlates these
two analysis results to generate the final detection result. There
is also a feedback mechanism which presents the correlation
result to behavior analysis engine to guide its learning procedure.
Secondly, we analyze the effectiveness of our correlation approach
compared with signature-based and behavior-based bot detection
approach. The analysis indicate that our correlation approach can
effectively improve the detection accuracy. Thirdly, we evaluate
our approach through experiments and the result indicate that
our approach can detect bots effectively.
Keywords—bot detection; host based; behavior and signature;
correlation; feedback;
I. INTRODUCTION
Botnet has become one of the most serious threats to
Internet security [1]. A bot is a host compromised by malwares
under the control of the botmaster through Command and
Control (C&C) channel (i.e., IRC, HTTP, P2P, etc.). A large
scale of bots form a botnet. The botmaster can utilize botnet
to conduct various cyber crimes such as spreading malwares,
DDoS attacks, spamming, phishing, and other cyber crimes.
Botnet has become the major platform for most online criminal
activities.
In order to defend bots, a large number of researches have
been carried on regarding the detection of bot and botnet. Ac-
cording to the execution location, existing approaches can be
divided into three categories: host-based approaches, network-
based approaches, and host and network correlation approach-
es. (1) Host-based approaches mainly include signature- and
behavior-based detection approaches [2]. Signature-based de-
tection approaches mainly extract the feature information of
the suspicious program to match with a knowledge database
of existing bot, such as Rishi [3]. Behavior-based detection
approaches monitor the abnormal behaviors on hosts to de-
termine whether the host is infected, such as the status of
operating system, the running status of suspicious programs,
access to suspicious Registries, files, system call sequences,
etc. [4], [5], [6]. (2) Network-based approaches mainly analyze
the network traffic to filter out the host which has abnormal
traffics [7]. Network-based approaches will be ineffective
with traffic encrypted, C&C protocols changed, etc. These
approaches are not able to identify the malicious process.
(3) Host and network correlation approaches correlate host
information with network traffic to detect bots [8], [9]. This is
a new detection mechanism at the exploration stage because
existing approaches still have the problems of both host- and
network- based approaches.
Compared with network-based approaches, host-based ap-
proaches can discover more insights of unknown bots, and
more importantly, we may completely eliminate the bot if we
can successfully detect it on end-host. In signature-based bot
detection approaches, the known bot programs and suspicous
programs do not need to run. Thus signature-based approaches
have low risks. While behavior-based approaches need to run
all of them, thus they have high risks. Signature-based ap-
proaches use the match method to generate the detection result,
if the match result reaches a level the program is regarded as
bot. Thus it can only detect known bots, to be more specific,
it can only detect the bots in the knowledge database. While
behavior-based approaches monitors the real execution behav-
iors to determine whether the programs are malicious. Thus
they can detect unkown bots. Most signature-based approaches
have low false positives because they are based on the strict
match result. They have different false negatives according
to the difference between experiment data and knowledge
database. While in behavior detection approaches, up-to-date
bot programs use many hidden mechanisms and act like benign
programs. Thus behavior-based approaches have low detection
accuracy. From the perspective of overhead, signature-based
approaches only analyze the suspicious binary to extract its
signature, while behavior-based approaches need to monitor
all the system calls. Thus signature-based approaches have low
overhead and behavior-based approaches have high overhead.
Up-to-date bots use many obfuscation techniques to evade
the detection, and these techniques can evade signature-based
approaches while well-designed behavior-based approaches
are still able to detect them. The comparison of these two
approaches are summarized in Table I.
2013 IEEE International Conference on High Performance Computing and Communications & 2013 IEEE International Conference
on Embedded and Ubiquitous Computing
978-0-7695-5088-6/13 $31.00 © 2013 IEEE
DOI 10.1109/HPCC.and.EUC.2013.230
1634