American Dreaming - 1990's / Part XV

So, I refrained from formulating my email with the “I told you so” message for the time being and began the efforts to try and salvage what we could from the present flooding situation.

In lieu of the prepared disaster recovery program that we had proposed two months earlier, it was time for an impromptu disaster recovery plan.

The first calls were to our telecommunications service providers to see if we could forward our clinical line to some other number (temporarily) and how soon could such a change be put into place. We were told it would likely be at least two business days before that could take place from their perspective. Shortly after that news met my ears, we were told by our telecommunications specialist that we could forward the lines to corporate offices as long as the PBX within our building remained functional. We opted to begin that process and within a half hour we had successfully shifted our clinical line to Virginia and vacated the building of the clinical staff. Side note – unfortunately for the clinical staff they had parked their cars in the basement parking garage and those vehicles were summarily toast.

I was met at the location by two of my network engineers. I dispatched them to start to shut down the networking communications gear within all the floors of the building. My son and I went to the data center and began to shut down the servers. Within an hour of that effort starting we had basically shut down all of the data center, we had physically uninstalled two of the primary servers and transported them by hand down the stair well and into my car.

I then transported the primary file server, workflow server and email server to my home along with the backup tapes from the past week of all relevant functions (including our PBX configuration tapes).

So, in essence, the disaster recovery plan was to move the data center for a fortune five hundred regional office into my kitchen.

The water did stop rising around the time we took these actions and fortunately, the electricity never did get cut off for the building. It took the balance of the weekend for the water to recede to the point where the parking lot was useful (the basement still had a massive amounts of clean-up to be done to make that space useful again).

Somewhere along the line some kind of assessment of the structural integrity of the building was done and it was deemed that the building had not suffered any material damage. Therefore come Monday morning the effort was made to bring the servers, the backups and all the reset of the communications and network gear back on line. We were essentially back up and running by 9:00 on Monday morning.

We had definitely dodged a bullet. It was now time to resurrect my disaster recovery plan and the “I told ya so” emails…