Category: Security

The Curious Tale of MS03-007

7

This is a story about how I knew within a window of 48 hours when the invasion of Iraq (2003) was going to happen.

It was early March, 2003.  I didn’t know exactly who the guys in suits were, but I knew they weren’t Microsoft.  Only one person I knew wore a suit daily to work at Microsoft, that was Raymond Chen. And he wore a much better class of suit than the guys who suddenly appeared late one evening on floor 6 of Building 40 on the Microsoft campus.

I had joined the Microsoft Security Response Center in November of 2002.  The Slammer attack was my first introduction to *the entire Internet* going offline as a result of a Microsoft security issue.

We were only just recovering from that event.  While all the appropriate and smart people had been mobilized to deal with Slammer, we were not happy with how ad hoc the response was.  So during the month of February and March we developed the Microsoft Internet Security Emergency Response process, MISER. Bill Gates hated the name. It was soon changed to Software Security Incident Response Process, SSIRP.

All I knew was that I had just been given one of the largest offices in the building, where I had installed a bar and held press calls on the security updates for all of MSRC and the ones I had program managed through the Windows team. Back then security updates happened every Wed. morning at 10am Pacific time, instead of every second Tuesday of the month like today. 

As release manager at the time, I would fire up “Yo, Pumpkin Head” on my computer and crank the speakers up as the updates propagated across the cluster of Microsoft.com and Windows Update.  We’d gather in the hallway and chatter as we made sure the updates and security bulletins reached their checkpoints while listening to the music. The entire process took almost exactly long as the song, around four minutes. When that music flooded the hallway, you knew updates were being launched. After that four minutes, I took press calls from CNN, MSNBC, ZDNET, NYT, etc for the rest of the day.

Point being, I was finally settling into the role vs. being in emergency mode for weeks over Slammer.

Then the guys in the suits showed up.

Our process was pretty established.  Microsoft issued security bulletins with updates to fix the problem. We didn’t issue warnings or advisories, we were dead set on issuing the transparent communication of the issue only when there was an update to correct it. At the time we viewed warnings or advisories as the equivalent of leaving a box of guns on the street corner and issuing a notice to citizens that there was a murderer in the area, go get your guns.  As many bad guys would get them, if not more, than attentive good guys. We learned better later, but this was the state in 2003.

I had just settled into the job as I mentioned.  I even had theme music. Then the guys in the suits showed up.

I wasn’t even involved at first.  I walked past our reserved emergency conf. room and in it were George, Ian my boss, Dr. Lipner, and the dudes in suits. I just walked on.  The most prized skill in information security is knowing when you do not want to be burdened with knowing what you do not already know.

It wasn’t until later that Ian showed up in my office to talk about it.

“You know what’s going on?” Ian knew I usually had my ear to the ground.  On this I didn’t.

“Dudes in suits. Usually US government.” I replied.  Ian had served in foreign military, specifically artillery. If it was US gov. in the room I’m sure they were roiling over what they would have to make him sign.

“Yea but do you know what’s going on?” Ian said.

“Nope!” I said.  I’d been knee deep in the regular reported vulnerabilities and MSRC work.

“How much do you know about WebDAV?” he asked.

Turns out I knew a lot.  Back then, WebDAV was a godsend to moving files around over the Internet vs. FTP or trying to use straight up HTTP.  WebDAV essentially treated certain web stores like a mapped network drive.

And in Windows 2000 it had a huge gaping hole.  It was enabled by default.  On all versions.

Ian explained carefully the issue to me, and that the guys in suits, from a section of the US government I’m not going to specify, had discovered it because they were attacked.  And that section of the government had a very important operation about to begin within 14 days.

“How soon do you think we could do a patch?” Ian asked.

I knew the Windows Sustained Engineering team’s schedule and backlog and made a scratch guess.

“No test, smoke test, full test, 14, 21 and 30 days.”  No test meant make the update, someone next to you tests that it fixes it, and you just ship it. Never mind the hundreds of millions of configurations in the world. It was the worst kind of update to ever release.  One we had never done before. 

Smoke test meant some more testing meaning seven days of in house testing.  Full test meant we would release the update to a number of high profile volunteer customers without letting them know specifically what it was for, so that we could understand the full impact.

“No good,” Ian said.  “We need to have it before mid March.”

“Ok, But that’s going to be a realignment of just about everything in the pipe.”

“This issue is worth it.”

That was no easy thing, and Ian knew it.  Before long I found myself in the room with George and Dr. Lipner and Ian and Mike Nash our VP.  Oh and the guys in suits, who I was never introduced to.

Here was the crux of the problem.  All Windows 2000 machines were essentially open to a trivial wormable attack like Slammer through this WebDAV vector.  It had been discovered by a government agency who had been attacked. Suddenly we had to re-evaluate how we communicated about updates.  This was bad enough we would have to consider going with how to block the attack before we actually had an update.  At the time that was anathema to the MSRC.  But this situation caused us to rethink everything.  We drew a line a long time ago before I joined, that no government got preference over users. But this wasn’t about an update per se it was about the existence of the hole. We had to figure out what to do if it became known, not for the agency involved but for everyone.

We handled it like we did any other update.  The reporter in this case we decided didn’t matter.  The severity drove the update, not who reported it.

The Windows team worked night and day to produce a fully tested update within 10 days.

On March 15th I wrote the very first Microsoft “Security Advisory” without a patch which contained information describing the issue and how to manually disable the functionality.  It was never released. We sweated the next two days until Wed, March 17th 2003 and released the update.  The security bulletin for the update contained much of the content I wrote for the advisory.

That particular event ended up forming the nascent idea that we should consider advisories when issues might take time to fix.

As I played the music down the MSRC hallway in building 40 that day, I was approached by a member of the senior staff. (Nope, not saying who)

“You know who got hit right?”

I had a good idea.  But just nodded. “Kinda ironic the patch is 007.”

“Watch the news in the next 48 hours.”

War fever has been gripping the US for the past 2 months, it wasn’t difficult to figure out what was about to happen.

On March 19th, the United States of America invaded Iraq.

Trustworthy Computing Ten Years On.

0

On December 7th, 1995 something extraordinary occurred at Microsoft.  For years our primary focus for software development had centered around narrowly scoped features that centered around the isolated experience of the personal computer.  To the extent connected experiences mattered it was always in the context of corporate networks.  The idea of a personal network in a home connected to a giant world-wide network of computers wasn’t a scenario that factored into our planning.

Until that day.  On that day a memo from Bill Gates to the entire corporation arrived in my inbox.  It laid out in precise terms how we’d come late to the game on the Internet experience and we would now be focusing all of our energy on it.  It was a galvanizing event.  A ship as big as Microsoft turned overnight.

The power of such a memo is easily diluted.  If used too often it loses the effect.  If used for small issues it can lead to too much energy being applied to something.  Bill didn’t send another memo of its like for a while, but when he did it had the exact same effect.

On January 15th 2002 Bill sent a memo to all employees entitled “Trustworthy Computing.”  In it, he articulated the case to pivot all our efforts in creating what was then the .NET platform to lead something he termed “Trustworthiness in Computing.”  Casting computer security against the industry and the world at large (including the terrorists attacks of 2001) Bill laid out key pillars of this effort: Availability, Security, and Privacy.  He tied the impact that a security vulnerability has to trust in Microsoft and our products.  He then made what I believe was the most fundamental change in our development methodology that would achieve the goal of more secure software:

“So now, when we face a choice between adding features and resolving security issues, we need to choose security. Our products should emphasize security right out of the box, and we must constantly refine and improve that security as threats evolve.”

The greatest impact of that memo is that I see younger people today in the industry read that line and shrug and say “well of course.”  But the computer industry was very different at that time.  People rushed to focus on features first, not just at Microsoft but other companies as well.  In general, my experience with software developers across the industry in the late 90’s was that security audits were routinely seen as a “tax” on development, and anyway if someone exploited a bug as an attack then that’s a crime and the law should handle it.

Bill’s memo transformed overnight the mindset of our development to think as much about misuse of features as use of them. That security was a fundamental aspect of software quality.

Today, security is at the forefront of software development.  Computer security is very much a journey, not a destination; much remains to be done.  But I look at the world of software and development today and I see a much different world than in 2002.  It’s fair to say that much of it started with a memo from Bill Gates on January 15th. Great work is still occurring every day, and to celebrate an amazing ten years the Trustworthy Computing team has made a special post, you can read about it here.