"This really could have screwed up our lives, and you know, a whole bunch of little geeks saved us."
- Paul Saffo. Director of the Institute for the Future in Menlo Park, California, in an interview with American RadioWorks

It was a fear fest of epic proportions. Magazine headlines predicted that the end of the world would shortly befall us. They told harrowing tales of feral computer systems going awry the minute the clock struck midnight on January 1st, 2000--planes would fall from the sky, power grids would fail, the global economy would crash, nuclear power plants would go into meltdown mode, lines of communication would be cut, and the contents of bank accounts would vanish.

"I cannot be optimistic...It's clear we can't solve the whole problem, so we have to allow some systems to die so mission-critical systems can work... Pay attention to the things that are vulnerable in your life and make contingency plans.... Don't panic, but don't spend too much time sleeping, either."
- Senator Robert Bennett, then-Chairman of the Senate's Special Committee on the Year 2000 Problem, Y2K Citizens Action Guide

The cause of all this excitement was the purported Y2K bug. In response to dealing with computers that were about as capable as the iPhone, programmers prided themselves on being thrifty with their code to ensure it didn’t stress memory or capacity. One way to do this was to skip putting a “19” in front of the year and use only the last two numerals. And why not? These programs would be long retired by the time 2000 rolled around.

However, not only were many expired programs, operating systems, and applications still chugging away as 2000 crept around the corner, they were also running critical systems like power grids, government agencies and financial systems, hospitals and airports, elevators and public transportation. And so, the race was on to fix the code before the bug bit.

Often the fix was a simple patch, but in some cases, programmers struggled to figure out what little time bombs lurked in ancient and undocumented code. Around the world, attempts to squash the bug are estimated to have cost over $300 billion.

Mac users were mostly immune as the system software was programmed to accept dates as far into the future as 29940. Assuming third-party applications didn’t go nuts, Macs would be fine. Apple happily pointed this out by airing a commercial during the 1999 Superbowl featuring the poster boy for naughty computers, Space Odyssey’s HAL.

 

 

At the time, Apple's "acting chief executive" Steve Jobs had issued a statement saying: "HAL is the perfect spokesperson to address the Y2K issues because he lives in the year 2001 and speaks from experience. Plus, HAL is the foremost expert on things that can go wrong with computers."

As the big day drew closer, gas masks, radiation kits, safes and water purification systems were hawked in magazine ads and late night commercials. People even began to horde survival supplies. An October 15, 1998 story in the New York Times shared the results of a poll stating that “10 percent of the nation's top executives are stockpiling canned goods, buying generators and even purchasing handguns.” Ominous quotes from experts seemed to indicate that life as we knew it might cease to exist for awhile.

"Suddenly, those individuals who have insisted that they will be withdrawing all their cash from the bank before the end of the year do not seem quite so misguided. The prospect of the millennium bug eating your savings may be more than just the nightmare of overactive imaginations. At a meeting in Washington recently, delegates were stunned to hear Henry Kissinger announce that he intended to withdraw all his money from the bank as 2000 nears."
- The Times, London, March 20th, 1999

But 1/1/2000 arrived and nothing dire happened. Perhaps enough code was fixed in time, perhaps the whole thing was insanely overhyped. Probably both. Even those who were on the frontlines of Y2K lunacy disagree about what might have happened. This is how they remember it.



What's All the Fuss About?

I'd like you to do me a favor. Get a steaming mug of coffee, herbal tea, or whatever beverage puts you in that relaxed, contemplative mood. Now close your eyes and drift back in time with me to early Spring of the year 1998. William Jefferson Clinton occupies the Oval Office, Dale Earnhardt still dominates NASCAR, Denver dethroned Green Bay in the Super Bowl, the U.S. unemployment rate is 4.3%, and the federal budget is enjoying a rare $70 billion surplus. Life is, all things considered, good.

Fade in on a second floor conference room of the John Wesley Powell federal building on Sunrise Valley Drive in Reston, Virginia, headquarters for the U.S. Geological Survey. Around the table sit a half dozen (mostly) somber business casually-dressed technical types. The urgent mission that has brought them together is a discussion of the threat posed by and mitigation strategies for the impending Y2K disaster, looming a scant twenty months into a forbidding future.

Speculation has been rampant, even this far from the target date.  Voices of doom permeate the airwaves, print, and cyberspace.  Aircraft will tumble, willy-nilly, from the skies. Trains will crash headlong into one another at high speed. Satellites will cease to communicate. Bank accounts will be drained.  Personal information will be lost forever, or exposed for all to see.  Twenty-four times, at the top of every hour, on average a little over 4% of the world's computers will freeze up or begin to spit out random nonsense as midnight processes along its inexorable westward path.

The damage, the carnage, the impact on humanity will be horrifying. Society's misguided reliance on doped silicon semiconductors will lead to our downfall. All is lost. Cash in your 401(k) now and spend it all on Friends laserdiscs before Y2K drags us clawing and screaming into the slavering jaws of oblivion.

One by one, the grim-faced custodians of the information systems that help guard our nation against a plethora of natural hazards--flood, volcanoes, earthquakes, invasive species, and more--give their candid assessments of the situation. The prognoses are poor. There is no cost-effective way to dodge the Y2K demon. Large-scale models are being created to simulate the event and ameliorate the consequences to whatever degree possible, but no one really knows what will happen on that fateful day. As we go around the table, administrators of the agency's thousands of Windows NT4 and 3.1 machines shake their ashen-countenanced heads at the terrifying uncertainty presented by this technological monster. Finally, it is my turn to report. I’m the Y2K coordinator for the Telecommunications Services Branch of the Office of Program Support at USGS HQ

I am reading ;Login: magazine and don't hear my name called the first time. They try again, with more stridency.

I look up, eyebrows raised questioningly. "Please give the Y2K status for the Telecommunications Services Branch," the facilitator commands.

I clear my throat. "We have two Sun 4500 clusters and about 250 Data General AViiON workstations. All of them are running some flavor of UNIX, whose designers intelligently provided the date function 32 bits to work with while Bill Gates was still mucking about in prep school. At 03:14:07 UTC on Tuesday, 19 January 2038, when this issue has some relevance to TSB, I will be long retired and quite possibly dead. I really don't see any good reason to make a fuss over it at this point. End of report."

And that, dear children, is how I got excused/banned from all future meetings.

Robert G. Ferrell is still an Information Systems Security Professional for the US Government and, ironically, now writes the /dev/random column for ;Login: magazine.


A High Stress Non Event

I was one of the senior network engineers at Long Beach Memorial hospital in Los Angeles. We had five hospitals in our circle of responsibilities. Long Beach which was the home site, Orange Coast, Anaheim, Saddleback and Miller's Children's Hospital. Plus there were many small offices, labs and remote sites that "kinda of sort of" fell under our roof.

There was a very chaotic assortment of very old software and new software with a lot of it falling between the two extremes. One of our oldest applications was the surgery scheduling software which ran on even then "old" Netware 3.1 servers using Btrieve. This was akin to a "made in the garage" application and was not well supported even in the best of times. This was not the best of times.
We had new software down in Radiology where they were using high speed networks to digitize and read x-rays etc remotely. Not to mention all the odd appliances like routers, switches, print servers, neonatal data transponders, wireless, security systems and so on. And I'm not even counting all the COBOL applications that had been custom written by long past employees or consultants, DOS, Visual Basic apps, home brew Access Databases and yes, the rogue server or two. Or three…or four.

In a word, messy.

We spent over a year and a lot of effort trying to identify problem apps and put fixes in place either from the vendors or hacking it ourselves in the case of the unsupported applications. The idea was make this as much of a non-event as we could.

When it came time for the actual roll over, all IT staff were required to be on site, not on call, be ON SITE which really tweaked more than a few spouses. In my department, we had our own itty bitty party and we got to see the fireworks from the Queen Mary from a parking structure.

Right after 9PM we got a call from a sister facility back east saying one of their business scheduling apps died on the stroke of midnight. We used this same application, as did many other facilities, and we got calls as each time zone rolled over. Since it was a business app, I did not have anything to do with it. I was much more worried about my servers and network hardware. Happily midnight came and went and all my stuff was running fine. There were some instances where the date got jacked up but the firmware still ran even though the log files were nuts insisting it was 1961 or other silly things like that.

In the end, Y2K was a high stress non-event for most of the IT staff at all our facilities.

Mike Sweeney is now a Network Security Manager at an undisclosed location.


Beanie Babies And Tulip Wars

(Note: Mac|Life changed two names in the story below to protect the ignorant)

Prior to joining True North at the end of 1998 (now InterPublic Group), which is the largest ad agency in the world, I was the CIO of Fallon McElligott, a much smaller ad agency.

Fallon's CFO, Mr. X, and I began the "Y2K is like the Tulip Craze of the 1600s" war. My argument was that Y2K was just an end of the millennia mania, much like the Beanie Babies at the current time and the Dutch Tulipmania of 1633. All hype and not much reality. I argued that Beanie Babies were as worthless as the Dutch Tulip bulbs and in the same vein, Y2K would come and go with as many or as little problems as every other year. Mr. X, being a member of the bow-tie wearing, afraid of everything crowd, thought that Y2K would cause everything from elevator stoppages to airplanes falling out of the sky. I remember asking if he was related to Chicken Little. I was summarily let go from Fallon, only to "fail up" to being named the CIO of the world's largest ad agency.

There I fell in with a much more enlightened crowd -- at least at the level of Management Executive Committee, or MEC (which was comprised of the CEO, the CEOs of the 14 major holding companies, the CFO, Chief Legal Officer, and myself). When I came on board at True North, the Y2K plan was like any other. We had the requisite letter-to-the vendors campaign (asking what they were doing), review of all software and OS configurations, and review of our Unix mainframe code.

Because we were an ad agency, many of the thousands of computers were Macintoshes. No problems there. Many were PCs, and Microsoft was already addressing any patches there. That left outside vendors, physical plants, and our Unix mainframe. A quick call to the mainframe guys told me the one thing I needed to know -- if there was going to be a glitch, they'd catch it the day after, patch it, and we'd be back billing just fine. Besides, if someone doesn't get a bill for a couple of days, it wouldn't kill us.
An hour or two of research on my part found that:

1.    International monetary float would not be affected -- no international financial downside to any Y2K problems

2.    Nobody would be billing the first few days after the 1st of the year anyway -- no world-wide downside to any Y2K problems

3.    Our ad system (the system that runs ads on more TVs, print, and radio than you'd ever expect worldwide) was always pre-bought anyway, and I was assured by all TV stations that "every ad would run, even if we have to manually push a button" -- no client problems there, even if Y2K did go crazy.

In short, if we could bill, and we could put client product out the door -- who cared what else would happen? Well, that was the logic that the MEC shared with me.

Then I started to meet the IT team. Our CTO wannabe, Mr, Z, was convinced, much like Mr. X back at Fallon, that the sky was going to fall down."Microsoft is going to have more holes than Swiss Cheese" he would run around saying.

Our EDS contractor thought that his letter campaign would help any liabilities (at $150 an hour on his part). Our security guys thought that someone would find some date-based loophole to break into our systems. Our physical security guys wanted signs on all elevators saying "do not enter on 12/31/99 unless you want to be here for awhile." My own IT-CFO (my budget was in the high tens of millions) thought that every billing system we had would fail. I had found Mr. X’s missing flock of Chicken Littles.

To ease everyone's nerves, I promised not to do anything "radical" until we convened my own CIO's council, made up of all of the holding company CIOs and the national CIOs. Over 100 people. We all met in Chicago at the beginning of 1999. Think over 100 Chicken Littles. Squawking in 40 to 50 languages that the sky was going to fall.

I couldn't believe it.

It took me until June to convince my own IT staff to transfer the EDS $150 per hour letter-writer to the legal side of the house so I could fire him.

I had to permanently assign my CTO wannabe to the Unix mainframe programmers (after taking them out for a great Steak and Wine dinner in NYC and promising that I'd make it up to them) so that he could randomly test any code with dates in it. We purposely set up the tests for twice a month because there were really only about a dozen or so places where a date could be entered. We figured that would keep him busy until at least December 15th and they only had to deal with him twice a month.

I even had to acquiesce (because of the rabid IT leadership complainers) to sending out an "international testing task force" to review all of the Y2K binders in each country and at each of the major holding companies. A wry note here -- I noticed in the travel bills that no one went to countries like Canada, Mexico, Bolivia, Congo or Ethiopia for testing. Of course the UK, France, Italy, Australia, Japan, Brazil and China; i.e., the "fun" places were visited.

In short, I slashed the Y2K budget as much as possible. I would have killed it all had I been given the chance.

By August the IT betting pool had me being "canned for short sited Y2K stupidity" at 2-1 odds.
By October they upped the odds to 4-1. I took the bet.

By November my CTO wannabe was writing "an official memoranda for posterity" and sending it to the head of HR and Legal that I should be canned for "Y2K Ostrich-like behavior". And he had additional signatures.

The odds were now 6-1 against my making it to December 1st.

However, our security analyst stopped staring at the sky and he convinced our IT CFO (who liked the "savings compared to our peers"), telephony, and physical plant staff to do the same.

December 1st rolled around and the CEO and I had dinner on that night just for laughs (and to show support that leadership wasn't falling for any of the complaints). December 15th rolled around and the odds makers were now betting "for" me. I hedged my bet just in case.

December 31st I held a party at my house in Chicago for anyone in IT leadership who wanted to attend. My CTO was at our HQ's call center "just in case." At "minutes after midnight" in every country in the world, I would get a phone call saying "all is fine". I told the CEO that by the time it hit Chicago, if I was still sober, I'd call him. About the time the fireworks on Lake Michigan died off, I called the CEO. I was semi-sober. He wasn't even close.

When we got back to work I shipped everyone who had complained about my Y2K attitude a little "Clucky the Chicken" Beanie Baby and a Tulip Bulb (paid for by the big payoff on the bet I wouldn't be canned). Most didn't get the joke.

Every December when I plant my Tulip bulbs here in California, I can't help but laugh.

Dorian J. Cougias is now the founder and Lead Analyst of Network Frontiers and is the co-creator of the Unified Compliance Framework.

NEXT: More Stories from IT Battlegrounds