Code Red round two

Tue Jul 31 18:17:10 UTC 2001

At 10:00 AM -0400 7/31/01, Dave Stewart wrote:
>At 09:49 AM 7/31/2001, Jeff Ogden wrote:
>>So what, if anything, are people planning to do differently as 8 pm 
>>EDT today and the possibility of a new round of Code Red Worm 
>>activity approaches? Are there things that we as network operators 
>>can and should be doing beyond encouraging end users to patch their 
>>vulnerable systems?
>
>You can scan your network(s) for machines that are vulnerable, and 
>patch them.  Or contact the end users and require that they patch 
>them.... if they aren't patched by 7:45pm or so, you can block port 
>80 access to those machines until they are patched.

OK, but even if we get every one of the vulnerable systems on our own 
and our customer's networks patched, we will still be subject to 
probes from infected systems elsewhere. In the last go round ten or 
eleven days ago it was the probes of unused IP addresses more than 
infected systems on our network that seemed to cause problems. So 
while we will continue to be good network citizens and work to get 
systems on our network patched, we will continue to see problems as 
long as there are "enough" unpatched systems out there to cause 
problems. I suspect that that is weeks or even months in the future.

Attached is a long message that was sent out to Merit's customers 
this morning talking about our plans.  No need to read it if you 
don't want to.

   -Jeff

--------------------

>Date: Tue, 31 Jul 2001 01:55:24 -0400
>To: michnet-inform
>From: Jeff Ogden <jogden at merit.edu>
>Subject: Merit's Tuesday evening plans related to the Code Red Worm
>
>I am sure that most of us have seen enough announcements about the 
>Code Red Worm by now to last a lifetime, but here is one more.
>
>I want to outline Merit's plans for the possible reemergence of the 
>Code Red Worm starting more or less at midnight UTC/GMT on August 
>1st (that is 8 pm EDT Tuesday evening here in the eastern U.S.). I 
>say more or less because many systems don't have their clocks set 
>exactly right or don't have their timezone set correctly, and so we 
>could see some activity start earlier or later than the expected 
>time by anything from a few minutes to as much as four or five hours.
>
>First let me say that we at Merit don't know and I don't think 
>anyone else really knows what, if anything, is going to happen 
>starting at 8 pm Tuesday evening. There are new variants of the worm 
>and they may behave differently. There are of course several 
>variants of the worm that we've seen already and so we do have some 
>idea of what to expect from them. We hope, but don't really believe, 
>that most vulnerable systems will have been patched over the last 
>week or ten days and that this will minimize the extent of any 
>future problems (see below for information on why this isn't likely 
>to be the case and about problems that may occur even after the 
>patches have been installed on all of your local systems).
>
>At least initially Merit does NOT plan to take any unusual steps to 
>deal with the Code Red Worm on Tuesday evening.  We are going to 
>start out treating this as a host computer problem. Host computer 
>problems are things that the people who are responsible for the 
>individual computers need to deal with. We will have staff watching 
>the network a bit more carefully than usual to spot and track signs 
>of unusual activity or problems. We plan to work directly with some 
>of the MichNet sites that were severely impacted by the Code Red 
>Worm last time, both to help these sites if there are problems and 
>to use the sites as something of an early warning indicator for what 
>we might expect elsewhere. We will be tracking developments 
>elsewhere including mailing lists and Web sites that have 
>information about Code Red developments.
>
>Sites with MichNet attachments can and should report network 
>problems to the Network Operations Center (NOC) by e-mail or by 
>phone. We would like to help where we can. We may be able to provide 
>assistance, but even if we can't help, reports will give us a better 
>view of what is actually happening across MichNet.
>
>If it would be helpful, we can install packet filters similar to the 
>ones we installed the last time around in routers that Merit 
>manages. These filters block packets inbound to port 80 on host 
>computers. This time we'd like to install these filters at the 
>request of individual sites rather than taking this action on our 
>own. If your site would like us to do this, contact the NOC. When 
>you call please have a list of the IP addresses for any host 
>computers that shouldn't be blocked. Of course many sites can and 
>probably should take these steps themselves in the routers or 
>firewalls that they manage.
>
>While we hope this won't be necessary, if we start to see serious 
>widespread problems, we may have to switch as we did last time and 
>treat this as a network rather than as a host computer problem. If 
>need be, we will be able to call in additional staff to work on 
>problems either Tuesday evening or Wednesday morning. If this 
>becomes necessary, we will post announcements to the MichNet-Inform 
>e-mail list and on the telephone recording that the NOC maintains.
>
>Estimates as of last Sunday are that at least 30% and perhaps as 
>high as 80% of the 350,000 plus systems that were infected with the 
>Code Red Worm a little more than a week ago have not yet been 
>patched. No matter which end of the range you believe you still get 
>big numbers. And no one knows how many vulnerable systems are out 
>there that weren't infected the last time around, but which may be 
>infected in the future. Estimates are that this is another large 
>number.
>
>Systems that only access the Internet over a dial-up line may be 
>infected or vulnerable. New systems right out of the box may be 
>vulnerable. Systems that belong to people on vacation or at schools 
>that are out for the summer, may be vulnerable when they are turned 
>back on days, weeks, or months from now. It seems certain that we 
>are all going to be working on the Code Red and related problems for 
>quite some time to come.
>
>See
>
>    http://worm-security-survey.caida.org/
>
>and
>
>    http://www.caida.org/analysis/security/code-red/
>
>for details about the rate that patches are being installed and some 
>very interesting analysis of the spread of the Code Red Worm ten 
>days or so ago.  If you don't have time to read all of this 
>information, at least look at the conclusions 
>(http://www.caida.org/analysis/security/code-red/#conclusions) which 
>are sobering.
>
>Even if your organization manages to patch every single vulnerable 
>system, your site may still see network performance problems due to 
>probes of your systems from infected computers located elsewhere. It 
>was side effects from these probes (ARP floods caused by large 
>numbers of probes to unused IP addresses), rather than the infected 
>systems themselves or the traffic from the probes, that seemed to 
>cause most of the network performance problems that individual sites 
>on MichNet experienced ten or eleven days ago.
>
>There are some things that individual sites can do to protect 
>themselves beyond installing the patches in the vulnerable systems. 
>Pay particular attention to comments about ingress and egress 
>filtering in the section on "Good Practices" in the CERT's 
>announcement (http://www.cert.org/advisories/CA-2001-23.html). Sites 
>with large amounts of unused IP addresses space seem to be more 
>vulnerable than other sites and so using filters in routers or 
>firewalls to block access to ranges of unused IP address may be 
>useful. Individual sites are in a much better position than Merit to 
>install all of these types of filters.
>
>Finally, there is a very real concern that with so much attention 
>focused on the Code Red Worm and installing the patches from 
>Microsoft, that we may be missing other security problems, assuming 
>that problems are due to Code Red when in fact they are not, or not 
>installing other patches and security fixes for other equally 
>important problems in a timely fashion. We all need to keep in mind 
>that the real problem here isn't the Code Red Worm, but inadequately 
>maintained systems. We all need to put procedures in place to ensure 
>that security patches and other fixes are installed in an on-going 
>and timely fashion in the future.
>
>Here is the list of some of the URLs related to the Code Red Worm 
>that people may find useful or interesting:
>
>   http://www.digitalisland.net/codered/ (includes step by step instructions,
>     slides, and audio from a 30 minute lecture on Code Red)
>
>   http://www.cert.org/
>   http://www.cert.org/archive/html/coderedannounce.html
>   http://www.cert.org/advisories/CA-2001-23.html
>   http://www.cert.org/advisories/CA-2001-20.html
>   http://www.cert.org/tech_tips/home_networks.html
>
>   http://www.microsoft.com/technet/security/bulletin/MS01-033.asp
>
>   http://www.cisco.com/warp/public/707/cisco-code-red-worm-pub.shtml
>
>   http://www.caida.org/
>   http://worm-security-survey.caida.org/
>   http://www.caida.org/analysis/security/code-red/
>
>   http://www.securityfocus.com/
>   http://www.securityfocus.com/bugtraq/archive
>   http://www.securityfocus.com/templates/column.html?id=13
>http://www.securityfocus.com/templates/archive.pike?list=1&start=2001 
>-07-15&fromthread=0&threads=0&mid=197828&end=2001-07-21&
>
>   http://www.net-security.org/text/articles/coverage/code-red/ (very
>      comprehensive collection of materials)
>
>   http://www.umich.edu/~virus-busters/bady.html
>
>   http://www.eeye.com/  (the folks that identified the vulnerability
>      originally back in June)
>   http://www.eeye.com/html/Research/Advisories/
>   http://www.eeye.com/html/Research/Tools/codered.html
>
>   http://www.nipc.gov/
>   http://www.nipc.gov/warnings/alerts/2001/01-016.htm
>
>   http://www.symantec.com/
>   http://www.symantec.com/avcenter/venc/data/codered.worm.html
>   http://www.symantec.com/press/2001/n010720a.html
>
>   http://www.nai.com/
>   http://www.mcafeeasap.com/asp_subscribe/trial_cc_wormscan.asp
>
>   http://www.merit.edu/mail.archives/nanog/
>
>Hope this is useful. Sorry there are so many of these messages and 
>some are so long.
>
>   -Jeff Ogden
>    Merit