Run sysdig as a Non-root User
For security, it's best to have a non-root user to run sysdig. Create a custom group for sysdig:
sudo groupadd sysdig
Add one or more users to the group. In our example, we'll add the user
sammy.
sudo usermod -aG sysdig sammy
Locate the binary file for sysdig:
which sysdig
You might receive a response like
/usr/bin/sysdig
Give all members of the
sysdig group privileges to run the
sysdig executable (and that binary only). Edit
/etc/sudoers with:
sudo visudo
Add the following lines for the
sysdig group in the groups section. Adding the new lines after the
%wheel section is fine. Replace the path with sysdig's location on your system:
## Same thing without a password
# %wheel ALL=(ALL) NOPASSWD: ALL
## sysdig
%sysdig ALL= /usr/bin/sysdig
If you need further clarifications on editing the
/etc/sudoers file, it is recommended that you take a look at
this article.
Running sysdig
You can run sysdig in two modes.
You can view the the real-time stream of server activity live, or you
can save records of system operations to a file for later offline
analysis.
Since you will most likely want to use the second option, that is
what we will cover here. Note that when saving system activity to a
file, sysdig takes a full snapshot of the operating system, so that
everything that happens on your server during that interval of time will
be available for offline analysis.
Note: When you run sysdig commands, please make sure
that each option is preceded by a single short dash. Copying and
pasting may cause an issue where a single dash is pasted as a long dash
and therefore not recognized by the program.
Let's run a basic sysdig command to capture 1000 lines of server activity.
To capture system activity to a file named
act1.scap, and limit the output to 1000 events, run the following command (omit the
-n 1000 part if you want to run sysdig for an unspecified period of time). The
-z switch is used to enable compression of the trace file.
sudo sysdig -w act1.scap.gz -n 1000 -z
Note: If you omitted the -n switch in the last step, you can interrupt the execution of sysdig by pressing the CTRL + C key combination.
Chisels — An Overview of sysdig Scripts
Chisels are sysdig scripts. To display a list of the available chisels, we need to run the following command:
sudo sysdig -cl
In order to audit the network traffic on our CentOS 7 LAMP server
using the trace file created by sysdig, we will use the chisels
available under the Net category:
Category: Net
-------------
iobytes_net Show total network I/O bytes
spy_ip Show the data exchanged with the given IP address
spy_port Show the data exchanged using the given IP port number
topconns top network connections by total bytes
topports_server Top TCP/UDP server ports by R+W bytes
topprocs_net Top processes by network I/O
Further description of a specific chisel, along with instructions for its use, can be viewed with:
sudo sysdig -i chisel name
For example:
sudo sysdig -i spy_ip
This outputs:
Category: Net
-------------
spy_ip Show the data exchanged with the given IP address
shows the network payloads exchanged with an IP endpoint. You can combine this chisel with the -x, -X or -A sysdig command line switches to customize the screen output
Args:
[ipv4] host_ip - the remote host IP address
[string] disable_color - Set to 'disable_colors' if you want to disable color output
The
Args section indicates whether you need to pass an argument to the chisel or not. In the case of
spy_ip, you need to pass an IP address as an argument to the chisel.
Auditing Network Traffic (Practical Example)
Let's walk through a practical example of how to use sysdig to
analyze bandwidth use and see detailed information about network
traffic.
To get the best results from this test, you will need to set up a
dummy web form on your server so appropriate traffic is generated. If
this is a server with a fresh LAMP installation, you can make this form
at
/var/www/html/index.php.
<body>
<form id="loginForm" name="loginForm" method="post" action="login.php">
<table width="300" border="0" align="center" cellpadding="2" cellspacing="0">
<tr>
<td width="112"><b>Username:</b></td>
<td width="188"><input name="login" type="text" class="textfield" id="login" /></td>
</tr>
<tr>
<td><b>Password:</b></td>
<td><input name="pass" type="password" class="textfield" id="pass" /></td>
</tr>
<tr>
<td> </td>
<td><br />
<input type="submit" name="Submit" value="Login" /></td></tr>
</table>
</form>
</body>
This isn't required, but to make everything tidy, you can also create the
/var/www/html/login.php page:
<body>
<p>Form submitted.</p>
</body>
Warning: Please delete this form when you are done testing!
Step 1 — Saving Live Data for Offline Analysis
We will starting capturing our log collection of data by issuing the following command:
sudo sysdig -w act1.scap.gz -z -s 4096
Leave sysdig running for a reasonable amount of time. Your command prompt will hang while sysdig runs.
Now, visit your server's domain or IP address in your web browser.
You can visit both existing and non-existing pages to generate some
traffic. If you want this specific example to work, you should visit the
home page, fill out the login information with anything you like, and
submit the login form a few times. In addition, feel free to run queries to your MySQL/MariaDB database as well.
Once you've generated some traffic, press CTRL + C to stop sysdig.
Then you will be ready to run the analysis queries that we will discuss
later in this tutorial.
In a production environment, you could start the sysdig data collection during a busy time on your server.
Understanding Filters: Classes and Fields
Before we get into sorting the sysdig data, let's explain some basic sysdig command elements.
Sysdig provides classes and fields as filters. You can think of
classes as objects and fields as properties, following an analogy based
on object-oriented programming theory.
You can display the complete list of classes and fields with:
sudo sysdig -l
We will use classes and fields to filter output when analyzing a trace file.
Since we want to audit the network traffic to and from our LAMP server, we will load the trace file
act1.scap.gz and perform the following tests with sysdig:
Displaying the list of top processes using network bandwidth
sudo sysdig -r act1.scap.gz -c topprocs_net
You should see output somewhat like this:
Bytes Process
------------------------------
331.68KB httpd
24.14KB sshd
4.48KB mysqld
Here you can see that Apache is using the most bandwidth (the
httpd process).
Based on this output, you can make an informed and supported judgment
call to decide whether you need to increase your available bandwidth in
order to serve your current and future estimated requests. Otherwise,
you may want to place appropriate restrictions on the maximum rate of
already available bandwidth that can be used by a process.
Displaying network usage by process
We may also want to know which IPs are using the network bandwidth consumed by
httpd, as shown in the previous example.
To that purpose, we will use the
topconns chisel (which shows the top network connections by total bytes) and add a filter formed with the class
proc and field
name to filter results to show only
http connections. In other words, the following command:
sudo sysdig -r act1.scap.gz -c topconns proc.name=httpd
This will return the top network connections to your server, including the source, where the process serving the request is
httpd.
Bytes Proto Conn
------------------------------
56.24KB tcp 111.111.111.111:12574->your_server_ip:80
51.94KB tcp 111.111.111.111:15249->your_server_ip:80
51.57KB tcp 111.111.111.111:27832->your_server_ip:80
51.26KB tcp 111.111.222.222:42487->your_server_ip:80
48.20KB tcp 111.111.222.222:42483->your_server_ip:80
48.20KB tcp 111.111.222.222:42493->your_server_ip:80
4.17KB tcp 111.111.111.111:13879->your_server_ip:80
3.14KB tcp 111.111.111.111:27873->your_server_ip:80
3.06KB tcp 111.111.222.222:42484->your_server_ip:80
3.06KB tcp 111.111.222.222:42494->your_server_ip:80
Note that the original source and destination IP addresses have been obscured for privacy reasons.
This type of query can help you find top bandwidth users that are sending traffic to your server.
After looking at the output above you may be thinking that the
numbers after the source IP addresses represent ports. However, that is
not the case. Those numbers indicate the event numbers as recorded by
sysdig.
Step 3 — Analyzing Data Exchanged Between a Specific IP and Apache
Now we'll examine the connections between a specific IP address and Apache in more detail.
The
echo_fds chisel allows us to display the data that
was read and written by processes. When combined with a specific process
name and a client IP (such as
proc.name=httpd and fd.cip=111.111.111.111 in this case), this chisel will show the data that was exchanged between our LAMP server and that client IP address.
In addition, using the following switches helps us to show results in a more friendly and accurate way:
-s 4096: For each event, read up to 4096 bytes from
its buffer (this flag can also be used to specify how many bytes of each
data buffer should be saved to disk when saving live data to a trace
file for offline analysis) -A: Print only the text portion of data buffers, and echo end-of-lines (we want to only display human-readable data)
Here's the command. Be sure to replace
111.111.111.111 with a client IP address from the previous output.
sudo sysdig -r act1.scap.gz -s 4096 -A -c echo_fds fd.cip=111.111.111.111 and proc.name=httpd
You should see quite a bit of output, depending on the number of
connections made by that IP address. Here's an example showing a 404
error:
GET /hi HTTP/1.1
Host: your_server_ip
Connection: keep-alive
Cache-Control: m
------ Write 426B to 111.111.111.111:39003->your_server_ip:80
HTTP/1.1 404 Not Found
Date: Tue, 02 Dec 2014 19:38:16 GMT
Server: Apache/2.4.6 (CentOS) PHP/5.4.16
Content-Length: 200
Keep-Alive: timeout=5, max=99
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL /hi was not found on this server.</p>
</body></html>
This type of query can help you figure out exactly what kinds of
connections were made by a top bandwidth-using IP address. For example,
if you found that the IP address was reaching a certain page very
frequently, you could make that page's assets as small as possible, to
reduce bandwidth use. Or, if the traffic doesn't seem to be legitimate,
you could create a new
firewall rule to block bandwidth-hogging IPs.
Step 4 — Examining Data Exchanged with an IP Address by Keyword
Depending on the server activity during the capture interval, the
trace file may contain quite a lot of events and information. Thus,
going through the results of the command in the previous section by hand
could take an impractical amount of time. For that reason, we can look
for specific words in event buffers.
Suppose we have a set of web applications running on our web server,
and we want to make sure that login credentials are not being passed as
plain text through forms.
Let’s add a few flags to the command used in the previous example:
sudo sysdig -r act1.scap.gz -A -c echo_fds fd.ip=111.111.111.111 and proc.name=httpd and evt.is_io_read=true and evt.buffer contains form
Here the class
evt, along with field
is_io_read, allow us to examine only read events (from the server’s point of view). In addition,
evt.buffer allows us to search for a specific word inside the event buffer (the word is
form in this case). You can change the search keyword to one that make sense for your own applications.
The following output shows that a username and password are being
passed from the client to the server in plain text (thus becoming
readable to anyone with enough expertise):
------ Read 551B from 111.111.111.111:41135->your_server_ip:80
POST /login.php HTTP/1.1
Host: your_server_ip
Connection: keep-alive
Content-Length: 35
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://104.236.40.111
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: http://104.236.40.111/
Accept-Encoding: gzip,deflate
Accept-Language: en-US,en;q=0.8
login=sammy&pass=password&Submit=Login
Should you find a similar security hole, notify your developer team immediately.
Conclusion
What you can accomplish with sysdig in auditing network traffic on a
LAMP server is mostly limited by one’s imagination and application
requests. We've seen how to find top bandwidth users, examine traffic
from specific IPs, and sort connections by keywords based on requests
from your applications.
Should you have any further questions about the present article, or
would like suggestions on how to work with sysdig in your current LAMP
environment, feel free to submit your comment using the form below.
25 Hardening Security Tips for Linux ServersHow-to-set-up-dnssec-on-an-nsd-nameserver-on-ubuntu-14-04
How-to-configure-webdav-access-with-apache-on-ubuntu-14-04
/
Créer un disque internet avec accès webDAV /Window$
How-to-install-discourse-on-ubuntu-14-04
Elasticsearch-fluentd-and-kibana-open-source-log-search-and-visualization
Héberger son serveur avec OpenBSD
Maltrail - Malicious Traffic Detection System
Mod_security-devant-un-serveur-web-apache
This is a easy install Apache server made to just plug and play.
XP Win 7 version
Apache 2.2.29 - Php 5.39 - MySQL 5.622 - PhpMyAdmin 4.32
VC9 Libraries
Win 7 OS2 Version
Apache 2.4.10 - Php 5.6.4 - MySQL 5.6.22 - PhpMyAdmin 4.32
VC11 Libraries
Neard sur window$
Sauvegarder ses emails sur Centos
Pour les blogs et CMS :
