24.2.1 Catching One in the Act

The easiest way to catch an intruder is by looking for events that are out of the ordinary. For example:

• A user who is logged in more than once. (Many window systems register a separate login for each window that is opened by a user, but it is usually considered odd for the same user to be logged in on two separate dial-in lines at the same time.)

• A user who is not a programmer but who is nevertheless running a compiler or debugger.

• A user who is making heavy and uncharacteristic use of the network.

• A user who is initiating many dialout calls.

• A user who does not own a modem logged into the computer over a dial-in line.

• A person who is executing commands as the superuser.

• Network connections from previously unknown machines, or from sites that shouldn't be contacting yours for any reason.

• A user who is logged in while on vacation or outside of normal working hours (e.g., a secretary dialed in by phone at 1:00 a.m. or a computer science graduate student working at 9:00 a.m.).

UNIX provides a number of commands to help you figure out who is doing what on your system. The finger, users, whodo, w, and who commands all display lists of the users who are currently logged in. The ps and w commands help you determine what any user is doing at any given time; ps displays a more comprehensive report, and w displays an easy-to-read summary. The netstat command can be used to check on current network connections and activity.

If you are a system administrator, you should be in the habit of issuing these commands frequently to monitor user activity. After a while, you will begin to associate certain users with certain commands. Then, when something out of the ordinary happens, you will have cause to take a closer look.

Be aware, however, that all of these commands can be "fooled" by computer intruders with sufficient expertise. For example, w, users, and finger all check the /etc/utmp file to determine who is currently logged in to the computer. If an intruder erases or changes his entry in this file, these commands will not report the intruder's presence. Also, some systems fail to update this file, and some window systems do not properly update it with new entries, so even when the file is protected, it may not have accurate information.

As the ps command actually examines the kernel's process table, it is more resistant to subversion than the commands that examine the /etc/utmp file. However, an intruder who also has attained superuser access on your system can modify the ps command or the code in the system calls it uses so that it won't print his or her processes. Furthermore, any process can modify its argv arguments, allowing it to display whatever it wishes in the output of the ps command. If you don't believe what these commands are printing, you might be right!

24.2.2 What to Do When You Catch Somebody

You have a number of choices when you discover an intruder on your system:

1. Ignore them.

2. Try to contact them with write or talk, and ask them what they want.

3. Try to trace the connection and identify the intruder.

4. Break their connection by killing their processes, unplugging the modem or network, or turning off your computer.

5. Contact a FIRST (Forum of Incident Response and Security Teams) team (e.g., the CERT-CC[2]) to notify them of the attack.

   [2] See Appendix F, Organizations, for a complete list of FIRST teams.

What you do is your choice. If you are most inclined towards option #1, you probably should reread Chapter 1, Introduction, and Chapter 2, Policies and Guidelines, . Ignoring an intruder who is on your system essentially gives him or her free reign to do harm to you, your users, or others on the network.

If you choose option #2, keep a log of everything the intruder sends back to you, then decide whether to pursue one of the other options. Options #3 and #4 are discussed in the sections "Tracing a Connection" and "Getting Rid of the Intruder" that follow.

NOTE: Some intruders are either malicious in intent, or extremely paranoid about being caught. If you contact them, they may react by trying to delete everything on disk so as to hide their activities. If you try to contact an intruder who turns out to be one of these types, be certain you have a set of extremely current backups!

If the intruder is logged into your computer over the network, you may wish to trace the connection first, because it is usually much easier to trace an active network connection than one that has been disconnected.

After the trace, you may wish to try to communicate with the intruder using the write or talk programs. If the intruder is connected to your computer by a physical terminal, you may wish to walk over to that terminal and confront the person directly (then again, you might not!).

NOTE: Avoid using mail or talk to contact the remote site if you are trying to trace the connection without tipping off the intruder, because the remote root account on the remote site may be compromised. Try to use the telephone, if at all possible, before sending email. If you must send a message, send something innocuous, such as "could you please give me a call: we are having problems with your mailer." Of course, if somebody calls you, verify who they are.

24.2.3 Monitoring the Intruder

You may wish to monitor the intruder's actions to figure out what he is doing. This will give you an idea if he is modifying your accounting database, or simply rummaging around through your users' email.

There are a variety of means that you can use for monitoring the intruder's actions. The simplest way is to use programs such as ps or lastcomm to see which processes the intruder is using.

Depending on your operating system, you may be able to monitor the intruder's keystrokes using programs such as ttywatch or snoop. These commands can give you a detailed, packet-by-packet account of information sent over a network. They can also give you a detailed view of what an intruder is doing. For example:

# snoop
asy8.vineyard.net -> next         SMTP C port=1974 
asy8.vineyard.net -> next         SMTP C port=1974 MAIL FROM:<dfddf@vin
        next -> asy8.vineyard.net SMTP R port=1974 250 <dfddf@vineyard.
asy8.vineyard.net -> next         SMTP C port=1974 
asy8.vineyard.net -> next         SMTP C port=1974 RCPT TO:<vdsalaw@ix.
        next -> asy8.vineyard.net SMTP R port=1974 250 <vdsalaw@ix.netc
asy8.vineyard.net -> next         SMTP C port=1974 
asy8.vineyard.net -> next         SMTP C port=1974 DATA\r\n
        next -> asy8.vineyard.net SMTP R port=1974 354 Enter mail, end 

In this case, an email message was intercepted as it was sent from asy8.vineyard.net to the computer next. As the above example shows, these utilities will give you a detailed view of what people on your system are doing, and they have a great potential for abuse.

You should be careful with the tools that you install on your system, as these tools can be used against you, to monitor your monitoring. Also, consider using tools such as snoop on another machine (not the one that has been compromised). Doing so lessens the chance of being discovered by the intruder.

24.2.4 Tracing a Connection

The ps, w, and who commands all report the terminals to which each user (or each process) is attached. Terminal names like /dev/tty01 may be abbreviated to tty01 or even to 01. Generally, names like tty01, ttya, or tty4a represent physical serial lines, while names that contain the letters p, q, or r (such as ttyp1) refer to network connections (virtual ttys, also called pseudo-terminals or ptys).

If the intruder has called your computer by telephone, you may be out of luck. In general, telephone calls can be traced only by prior arrangement with the telephone company. However, many telephone companies offer special features such as CALL*TRACE and CALLER*ID (CNID), which can be used with modem calls as easily as with voice calls. If you have already set up the service and installed the appropriate hardware, all you need to do is activate it. Then you can read the results.

If the intruder is logged in over the network, you can use the who command to determine quickly the name of the computer that the person may have used to originate the connection. Simply type who:

% who 
orpheus  console  Jul 16 16:01 
root     tty01    Jul 15 20:32 
jason    ttyp1    Jul 16 18:43 (robot.ocp.com) 
devon    ttyp2    Jul 16 04:33 (next.cambridge.m) 
%

In this example, the user orpheus is logged in at the console, user root is logged on at tty01 (a terminal connected by a serial line), and jason and devon are both logged in over the network: jason from robot.ocp.com, and devon from next.cambridge.ma.us.

Some versions of the who command display only the first 16 letters of the hostname of the computer that originated the connection. (The machine name is stored in a 16-byte field in /etc/utmp; some versions of UNIX store more letters.) To see the complete hostname, you may need to use the netstat command (described in Chapter 16, TCP/IP Networks). You will also have to use netstat if the intruder has deleted or modified the /etc/utmp file to hide his presence. Unfortunately, netstat does not reveal which network connection is associated with which user. (Of course, if you have the first 16 characters of the hostname, you should be able to figure out which is which, even if /etc/utmp has been deleted. You can still use netstat and look for connections from unfamiliar machines.) Luckily, most modern versions of UNIX, including SVR4, report the entire machine name.

Let's say that in this example we suspect Jason is an intruder, because we know that the real Jason is at a yoga retreat in Tibet (with no terminals around). Using who and netstat, we determine that the intruder who has appropriated Jason's account is logged in remotely from the computer robot.ocp.com. We can now use the finger command to see which users are logged onto that remote computer:

% finger @robot.ocp.com 
[robot.ocp.com] 
Login     Name                  TTY Idle       When 
olivia    Dr. Olivia Layson      co  12d   Sun 11:59 
wonder    Wonder Hacker          p1        Sun 14:33  
%

Of course, this method doesn't pin the attacker down, because the intruder may be using the remote machine only as a relay point. Indeed, in the above example, Wonder Hacker is logged into ttyp1, which is another virtual terminal. He's probably coming from another machine, and simply using robot.ocp.com as a relay point. You would probably not see a username like Wonder Hacker. More likely, you would only see an assorted list of apparently legitimate users and have to guess who the attacker is. Even if you did see a listing such as that, you can't assume anything about who is involved. For instance, Dr. Layson could be conducting industrial espionage on your system, using a virtual terminal (e.g., xterm) that is not listed as a logged in session!

If you have an account on the remote computer, log into it and find out who is running the rlogin or telnet command that is coming into your computer. In any event, consider contacting the system administrator of that remote computer and alert him or her to the problem.

24.2.5 Getting Rid of the Intruder

Killing your computer's power - turning it off - is the very quickest way to get an intruder off your computer and prevent him from doing anything else - including possibly further damage. Unfortunately, this is a drastic action. Not only does it stop the intruder, but it also interrupts the work of all of your legitimate users. It may also delete evidence you night need in court some day, delete necessary evidence of the break-in, such as running processes (e.g., mailrace), and cause the system to be damaged when you reboot because of the Trojaned startup scripts. In addition, the UNIX filesystem does not deal with sudden power loss very gracefully: pulling the plug might do significantly more damage than the intruder might ever do.

In some cases, you can get rid of an intruder by politely asking him or her to leave. Inform the person that breaking into your computer is both antisocial and illegal. Some computer trespassers have the motivation of a child sneaking across private property; they often do not stop to think about the full impact of their actions. However, don't bet on your intruder being so simplistic, even if he acts that way. (And keep in mind our warning earlier in this chapter.)

If the person refuses to leave, you can forcibly kill his or her processes with the kill command. Use the ps command to get a list of all of the user's process numbers, change the password of the penetrated account, and finally kill all of the attacker's processes with a single kill command. For example:

# ps -aux 
USER  PID  %CPU  %MEM   VSIZE   RSIZE  TT  STAT   TIME  COMMAND 
root 1434  20.1   1.4    968K   224K   01    R    0:00  ps aux 
nasty 147   1.1   1.9   1.02M   304K   p3    S    0:07  - (csh) 
nasty 321  10.0   8.7    104K   104K   p3    S    0:09  cat /etc/passwd 
nasty 339   8.0   3.7   2.05M   456K   p3    S    0:09  rogue 
... 
# passwd nasty 
Changing password for nasty. 
New password: rogue32
Retype new password: rogue32 
# kill -9 147 321 339 

You are well-advised to change the password on the account before you kill the processes - especially if the intruder is logged in as root. If the intruder is a faster typist than you are, you might find yourself forced off before you know it! Also bear in mind that most intruders will install a back door into the system. Thus, even if you change the password, that may not be sufficient to keep them off: you may need to take the system to single-user mode and check the system out, first.

As a last resort, you can physically break the connection. If the intruder has dialed in over a telephone line, you can turn off the modem - or unplug it from the back of the computer. If the intruder is connected through the network, you can unplug the network connector - although this will also interrupt service for all legitimate users. Once the intruder is off your machine, try to determine the extent of the damage done (if any), and seal the holes that let the intruder get in. You also should check for any new holes that the intruder may have created. This is an important reason for creating and maintaining the checklists described in Chapter 9, Integrity Management.

24.2.6 Anatomy of a Break-in

The following story is true. The names and a few details have been changed to protect people's jobs.

Late one night in November 1995, a part-time computer consultant at a Seattle-based firm logged into one of the computers that he occasionally used. The system seemed sluggish, so he ran the top command to get an idea of what was slowing down the system. The consultant noticed that a program called vs was consuming a large amount of system resources. The program was running as superuser.

Something didn't look right. To get more information, the programmer ran the ps command. That's when things got stranger still - the mysterious program didn't appear when ps was run. So the occasional system manager used the top command again, and, sure enough, the vs program was still running.

The programmer suspected a break-in. He started looking around the filesystem using the Emacs dired command and found the vs program in a directory called /var/.e. That certainly didn't look right. So the programmer went to his shell window, did a chdir() to the /var directory, and then did an ls -a. But the ls program didn't show the directory /var/.e. Nevertheless, the program was definitely there: it was still visible from the Emacs dired command.

The programmer was now pretty sure that somebody had broken into the computer. And the attack seemed sophisticated, because system commands appeared to have been altered to hide evidence of the break-in. Not wanting to let the break-in proceed further, the operator wanted to shut down the computer. But he was afraid that the attacker might have booby-trapped the /etc/halt command to destroy traces of the break-in. So before the programmer shut down the system, he used the tar command to make a copy of the directory /var/.e, as well as the directories /bin and /etc. As soon as the tar file was made, he copied it to another computer and halted the system.

The following morning, the programmer made the following observations from the tar file:

• Somebody had broken into the system.

• The program /bin/login had been modified so that anybody on the Internet could log into the root account by trying a special password.

• The /var/.e/vs program that had been left running was a password sniffing program. It listened on the company's local area network for users typing their passwords; these passwords were then sent to another computer elsewhere on the Internet.

• The program /bin/ls and /bin/ps had been modified so that they would not display the directory /var/.e.

• The inode creation dates and the modification times on the files /bin/ls, /bin/ps and /bin/login had been reset to their original dates before the modifications took place. The checksums for the modified commands (as computed with the sum command) matched those of the original, unmodified versions. But a comparison of the programs with a backup made the previous month revealed that the programs had been changed.

It was 10:00 p.m. at night when the break-in was discovered. Nevertheless, the consultant telephoned the system manager at home. When he did, he discovered something else:

• The computer's system manager had known about the break-in for three days, but had not done anything about it. The reason: she feared that the intruder had created numerous holes in their system's security, and was afraid that if she angered the intruder, that person might take revenge by deleting important files or shutting down the system.

In retrospect, this was rather stupid behavior. Allowing the intruder to stay on the system let him collect more passwords from users of the system. The delay also allowed for plenty of time to make yet further modifications to the system. If it was compromised before, it was certainly compromised now!

Leaving the intruder alone also left the company in a precarious legal position. If the intruder used the system to break in anywhere else, the company might be held partially liable in a lawsuit because they left the intruder with free run of the compromised system.

So, what should the system manager have done when she first discovered the break-in? Basically, the same thing as what the outside consultant did: take a snapshot of the system to tape or another disk, isolate the system, and then investigate. If the staff was worried about some significant files being damaged, they should have done a complete backup right away to preserve whatever they could. If the system had been booby-trapped and a power failure occurred, they would have lost everything as surely as if they had shut down the system themselves.

The case above is typical of many break-ins that have occurred in 1994 and 1995. The attackers have access to one of many "toolkits" used to break into systems, install password sniffers, and alter system programs to hide their presence. Many of the users of these toolkits are quite ignorant of how they work. Some are even unfamiliar with UNIX: we have heard many stories of monitored systems compromised with these sophisticated toolkits, only to result in the intruders attempting to use DOS commands to look at files!