Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Would your company survive a disaster?

By fossilcode in Op-Ed
Thu Mar 01, 2001 at 09:27:44 PM EST
Tags: News (all tags)
News

Yesterday's 6.8 earthquake near Olympia, Washington is now old news. People share their stories, the damage assessments are tallied and people not directly involved turn their idle curiosity to the next big story in the news. After my dishes stopped rattling and I checked to make sure I didn't have a gas leak I began to ponder: was I ready for a real disaster?


It's easy to watch the news and make quick judgements about natural and man-made disasters around the world. It's easy to sit back, especially if you live in a region not prone to the specific sort of disaster you're watching and say "It will never happen to me."

In an age where we depend more and more on computers and networks for commerce, entertainment and vital services, are we really looking at the risks any kind of disaster could have on these important conduits of information and income? If you work with computers, are you secure in the knowledge that your company has taken the steps necessary to ensure its survival if the unexpected happens?

Many industries have relied on computers for decades. For them, business resumption plans are serious affairs. Companies such as Sungard and Comdisco have large operations devoted to providing space, hardware and networks to be used by paying clients should one of them lose their main data center. For firms such as banks, which in the United States are prohibited by law from being closed for more than a few consecutive days, telephone companies and other utilities, the ability to get back on line is an absolute must. Yet even in these industries, with "mature" IS implementations, disaster recovery planning is often relegated to a very secondary status, with test exercises performed in a cursory way to please auditors, rather than with the more important intent of guaranteeing that the company can continue to function.

Disasters come in many forms. Depending on where you live, you might picture hurricanes, tornadoes, floods or earthquakes as the disasters for which you should prepare. These are the more extreme disasters. The disaster that is most likely to strike any business is fire.

With the ever-increasing use of desktop workstations and LANs in today's business environment, the ability of any company to protect its information assets can be severely hobbled.

Think about your own company. Now answer these questions and see if you think you could resume business quickly if you had a disaster.

  1. Is all vital data backed up to a secure storage medium on a regular basis and stored at a location not likely to be affected by a local or regional disaster? Or are key business functions being performed on desktop machines with no regular backup?
  2. Does your business have a remote location where resumption could take place if the building you occupy is unusable?
  3. Assuming you have a remote location, do you have the routine supplies you require to get going: pencils, paper, telephones, etc.?
  4. Can you get key personnel to the remote location in a timely fashion? What if normal transportation arteries are cut off?
  5. Can you contact key personnel in the event of a disaster? Or are the only people in your organization who carry 24-hour pagers the junior dweebs who haven't earned enough seniority to escape their off-hours support responsibilites?
  6. If you got this far without answering "no" does your company practice for disasters? Do you have surprise "mock disasters" where life stops and you activate your remote center?

Disaster planning only seems to become important after a disaster strikes. Planning and testing the plan can help prepare companies for the unfortunate circumstance of dealing with the real thing. It can make the difference between having a business (and the jobs it provides) and losing it. No company, regardless of size, can afford to ignore the possibilities.

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Poll
Is your business prepared for a disaster?
o Yes, most definitely. 3%
o Mostly. 10%
o Not really. 14%
o Hadn't thought about it. 10%
o Every day at my business is a disaster. 29%
o Business? We're in business? 31%

Votes: 57
Results | Other Polls

Related Links
o 6.8 earthquake
o Sungard
o Comdisco
o Also by fossilcode


Display: Sort:
Would your company survive a disaster? | 20 comments (20 topical, editorial, 0 hidden)
Really? (none / 0) (#1)
by spaceghoti on Thu Mar 01, 2001 at 05:16:57 PM EST

24 hours later, and the earthquake is now "old news." Wow, how times have changed.



"Humor. It is a difficult concept. It is not logical." -Saavik, ST: Wrath of Khan

hm... (none / 0) (#5)
by regeya on Thu Mar 01, 2001 at 05:45:45 PM EST

Judging from the comments left on yesterday's eyewitness account, it was old news before it happened.

[ yokelpunk | kuro5hin diary ]
[ Parent ]

Well, no. BUT... (none / 0) (#2)
by fuzzrock on Thu Mar 01, 2001 at 05:29:14 PM EST

I got to answer "no" to all but #5 in your list. On the other hand, I do think that the company I work for would be all right in the event of a major disaster. We'd lose a couple of weeks maybe, but in the long run we'd be fine.

I think this, because while it would be severely damaging to have nobody in the company being productive for a while, the real asset the company has is in the team - not in the machines (which are insured), and only slightly in the codebase. It would be infuriating for a few weeks, and we'd have to play catch-up a lot, but I think many companies would actually be in more or less the same place.

Of course, I work for a software company. One has to assume that a company that was less about mindshare might not do as well.

-fuz

Not quite true (4.25 / 4) (#3)
by trhurler on Thu Mar 01, 2001 at 05:33:15 PM EST

Disaster preparation is like insurance. You consider the likelihood of the disaster, the potential cost, and you compare it to the cost of the insurance. If the insurance costs more than the disaster, then you don't buy it. This is one of the principal caps on the cost of insurance. Competition can be another. However, there is no competition in disaster recovery, so frequently, this is the ONLY cap on the cost of disaster recovery.

As you might imagine, for smaller companies, disaster recovery of any sophistication at all would cost more than their annual revenues, so they don't even attempt it. Some of them have the smarts to keep off site backup tapes and so on, but that's hardly a disaster recovery capability. This is one of the cases where a big company has an edge, but only if they do it right, which almost none of them do anyway.

In any case, it simply is not true that all companies must have a disaster recovery capability; most cannot do it, no matter what, because the cost is prohibitive.

--
'God dammit, your posts make me hard.' --LilDebbie

Would you run you business without insurance? (4.00 / 1) (#4)
by fossilcode on Thu Mar 01, 2001 at 05:41:51 PM EST

I agree that disaster recovery of the type provided by Comdisco would be cost prohibitive for many businesses. My point (and I'm as guilty as anybody) is that companies become complacent. Especially in industries that rely heavily on computers and electronic records to function, failure to have something in place would equate to instant business failure should a disaster strike.

The thing is, you know your market, you have a good product, you know your competition, you have equipment and employees. You probably even have fire insurance and all the stuff the law requires you to have. But could you really resume business if something catastrophic happened? Do you even have off-site paper copies of you important records such as customer lists and the last accounts receivable register? Would your company survive?


--
"...half the world blows and half the world sucks." Uh, which half were you again?
[ Parent ]
When it's infeasible (4.00 / 1) (#10)
by Miniluv on Thu Mar 01, 2001 at 07:32:18 PM EST

I hate to tell you folks this, but not every business can have an "adequate" disaster recovery plan. Certain technologies mitigate against it. Geographic diversity is not always possible, let alone reasonable, to try and attain. It's an admirable goal, but I often wonder how much money has been blown on ridiculous over-protection.

I worked in a building that rode two power grids, and had three separate fiber backbones for Internet connectivity, it also had three fully independent diesel generators with two underground diesel tanks. It was owned and maintained by the computer industry's defining "blue chip" stock, and it had never had a power outage. Not even once. Sure, you insure in the hopes you never need it...but isn't all that just a little excessive?

"Its like someone opened my mouth and stuck a fistful of herbs in it." - Tamio Kageyama, Iron Chef 'Battle Eggplant'
[ Parent ]

Not as expensive as you might think (5.00 / 1) (#13)
by jkeene on Thu Mar 01, 2001 at 09:39:10 PM EST

But make sure you are fairly comparing the costs.

Consider that if a publicly held company fails after experiencing a disaster, the stockholders are going to start some serious lawsuits against the directors. Those directors will be very aware of their exposure once they focus on this fact.

Consider also that you can cover some of the smaller, non-firestorm type disasters for less than you think. I worked at a company that did plan for a whole range of disaster types, and we actually had reason to use them live from time to time.

You might think that on the sixth floor of a building you don't have to worry about flooding. But when on the seventh floor, one of the other tenant's cleaning crews burns some popcorn in the microwave, the sprinkler system goes off. And it does a lot of water damage to the sixth floor, which was just where one of the specialty trading desks resided.

Activate the business continuity plan, which is our own hot-site with fully networked computers connected to the corporate network, but located out in the farmlands where the real estate is cheap, and that trading desk opens only an hour late.

The systems out at that hot-site, and the staff and floor space to support it, was only another 5% of total operating budget.

Granted it's not enough to recover from a chemical spill next to a headquarters building, but had that trading room been down for three days instead of an hour, the financial impact would have been visible in the annual reports for years to come.

[ Parent ]

Yes, but (none / 0) (#17)
by trhurler on Fri Mar 02, 2001 at 10:50:04 AM EST

Most companies are not of the size you're talking. Keep in mind that the average company in the US employs something like 30 or 40 people. There are TONS of these little bitty privately held companies. Sure, the big dogs can have a second building for 5% of annual operating costs, but a typical small company's second largest expense after employees is the building. Frequently, that building is already out in the boonies to take advantage of cheap real estate. You can't just go and get a second one and expect to survive in that circumstance.

--
'God dammit, your posts make me hard.' --LilDebbie

[ Parent ]
Take 5% and see what you get (none / 0) (#20)
by jkeene on Mon Mar 05, 2001 at 10:40:50 PM EST

I'll bet you could take 5% of the budget of one of those small companies and do something quite useful for continuity planning. There's probably some central group of systems that is maybe 5-15% of the hardware, but that somehow affects 80% of the business.

Do some simple continuity work on that, even if it is as simple as having an off-site duplicate of a process control system, or a nightly copy of the day's file server changes.

There's really aren't that many parts of the world that aren't at sometime affected by weird weather. And other tenants in a building are as prone to exhibiting human failings as anybody.

So I'm distinctly in favor of having some modicum of business continuity planning done, at companies of all sizes.

[ Parent ]

Scary statistic (3.50 / 4) (#6)
by pwhysall on Thu Mar 01, 2001 at 06:29:26 PM EST

(I can't remember where this came from...)

Of all the companies that suffer a *total* data loss, 80% go out of business within 6 months, irrespective of size.
--
Peter
K5 Editors
I'm going to wager that the story keeps getting dumped because it is a steaming pile of badly formatted fool-meme.
CheeseBurgerBrown

About that other 20% (none / 0) (#7)
by marlowe on Thu Mar 01, 2001 at 06:44:06 PM EST

How do they manage it?

-- The Americans are the Jews of the 21st century. Only we won't go as quietly to the gas chambers. --
[ Parent ]
Shaky stab in the dark (none / 0) (#15)
by pwhysall on Fri Mar 02, 2001 at 05:23:57 AM EST

I've never been there, so this is conjecture on my part, but I would guess that for a company whose primary capital is in the heads of its people, say, a design company, it would be less hard to recover than, say, a software company.

But I'd bet a pint that the 20% are shadows of their former selves.
--
Peter
K5 Editors
I'm going to wager that the story keeps getting dumped because it is a steaming pile of badly formatted fool-meme.
CheeseBurgerBrown
[ Parent ]

multi-site backup is a must (none / 0) (#8)
by rebelcool on Thu Mar 01, 2001 at 06:45:17 PM EST

Any company (or individual) should always back up their data to multiple places, and i dont mean just tapes. I do alot of code work at home, but what if there is a fire here? Even if I have disk backups (which i do in event of a drive crash) what happens if it all burns up? All that hard work just gone.

Everybody should spend a little money in one of those internet storage places and put their most important files there.

COG. Build your own community. Free, easy, powerful. Demo site

Would your company survive a disaster? (3.00 / 1) (#9)
by SIGFPE on Thu Mar 01, 2001 at 06:47:03 PM EST

I'm sure many people (especially .commers) are too busy asking the simpler question "Would your company survive?"
SIGFPE
Telephone companies (none / 0) (#11)
by Miniluv on Thu Mar 01, 2001 at 07:34:47 PM EST

The nature of the PSTN network mitigates against any substantive redundancy. When a CO goes down, those customers stay down until the CO comes back up, regardless of what any other facility is doing. They were mentioned in the write up, and I'm curious how they actually apply compared to, say, a bank's processing center which could be quickly replicated someplace else and encrypted traffic rerouted to the new location transparently.

"Its like someone opened my mouth and stuck a fistful of herbs in it." - Tamio Kageyama, Iron Chef 'Battle Eggplant'
Factors for banks (4.00 / 1) (#12)
by fossilcode on Thu Mar 01, 2001 at 08:11:19 PM EST

There are other factors in addition to the computers and the network.

  1. Equipment: disaster site needs specialized equipment such as check sorters. Not easy to obtain "on-the-fly."
  2. Network: Even in disaster mode, some business function may not be available because of collateral damage to the telephone network supplying the leased lines banks use for things such as branch terminals, security systems and ATM's.
  3. Volume of data: Banks process enormous amounts of data each business day, much if it in batch. The size of the databases makes restoration of foreign hardware a critical-path item, taking as long as 2 or 3 days. Mission-critical applications are restored first, secondary ones later.
  4. Transportation: Banks rely heavily on paper. Secure courier service to move the documents (checks among other things) from the branches in affected areas might not be available.

There are other factors too, but this gives you an idea.


--
"...half the world blows and half the world sucks." Uh, which half were you again?
[ Parent ]
Further consideration (none / 0) (#14)
by Miniluv on Fri Mar 02, 2001 at 01:45:11 AM EST

Equipment: disaster site needs specialized equipment such as check sorters. Not easy to obtain "on-the-fly."
Actually in a typical setup check sorting is done at the branches on paper and then electronically at the central data center, so this is a minor issue.

Network: Even in disaster mode, some business function may not be available because of collateral damage to the telephone network supplying the leased lines banks use for things such as branch terminals, security systems and ATM's
The nature of the "disaster" affects this greatly. A nuclear attack versus a building fire, for example. The point is very valid though.

Volume of data: Banks process enormous amounts of data each business day, much if it in batch. The size of the databases makes restoration of foreign hardware a critical-path item, taking as long as 2 or 3 days. Mission-critical applications are restored first, secondary ones later.
Processing large amounts of data isn't nearly as hard as it used to be, and in a well thought out disaster recovery plan obtaining similar equipment and having either cold standby, or a timeframe based recovery plan is relatively easy. In a worst case environment it would be possible to, with clever coders, simulate one environment inside of another. Say, simulate the JCL/MVS/CICS environment in Unix, or vice versa to kludge through data processing.

Transportation: Banks rely heavily on paper. Secure courier service to move the documents (checks among other things) from the branches in affected areas might not be available.
This is less true than it used to be, and that gets truer every day. With the way technology is affecting the financial world secure data links become more important than secure couriers with each passing day.

"Its like someone opened my mouth and stuck a fistful of herbs in it." - Tamio Kageyama, Iron Chef 'Battle Eggplant'
[ Parent ]
You made my point about check sorters. (4.00 / 1) (#16)
by fossilcode on Fri Mar 02, 2001 at 08:00:01 AM EST

"Actually in a typical setup check sorting is done at the branches on paper and then electronically at the central data center, so this is a minor issue."

Sorting checks electronically at the central data center requires a reader/sorter (IBM 3890-type machine). If you're forced to a backup site, you'd better have one there, because they're huge and require a fair bit of set-up time.

It's true that the largest banks run multiple capture sites or they couldn't handle the volume or be able to tranport items in time to make posting deadlines. So for banks configured in this way, diverting the items from the lost site might be the plan instead of having a disaster site with the equipment installed.

"...in a well thought out disaster recovery plan obtaining similar equipment and having either cold standby, or a timeframe based recovery plan is relatively easy."

Sometimes. If your company uses specialized or out-of-production equipment, it can be a real problem. There's also the issue of laying down the applications and the data on virgin hardware. Synchronizing the timestamps on everything is more difficult in actual practice than it is in theory. The more independent but inter-related applications you run, the more difficult the task becomes. Most companies cannot afford to dedicate hardware to "hot" recovery, so there is always a delay while you lay down "your" OS and sysgen on the hardware, lay down your data in a disk farm that isn't exactly the same as the one its replacing, and try to ensure that the applications are synchronized to a recovery point. Then assuming of course you accomplished all these steps without failure, you still have to run prior work to catch up to the point of the disaster. All these factors weigh heavily against recovering your system in a timely fashion.


--
"...half the world blows and half the world sucks." Uh, which half were you again?
[ Parent ]
No, I didn't. (none / 0) (#19)
by Miniluv on Fri Mar 02, 2001 at 10:44:53 PM EST

Sorting checks electronically at the central data center requires a reader/sorter (IBM 3890-type machine). If you're forced to a backup site, you'd better have one there, because they're huge and require a fair bit of set-up time.
Reread what I said, because it directly contradicts the point you're trying to make. Banks do not do central check sorting anymore in a large number of cases, instead doing it at each branch office and sending electronic data to the central processing environment for batch processing.

Sometimes. If your company uses specialized or out-of-production equipment, it can be a real problem. There's also the issue of laying down the applications and the data on virgin hardware.
These are situations that ought to be taken into account in your DR plan. These machines should not, even if they're cold spares, be sitting completely untested, but instead ought to be preinstalled and you ought to practice failover procedures to insure they work as expected.

"Its like someone opened my mouth and stuck a fistful of herbs in it." - Tamio Kageyama, Iron Chef 'Battle Eggplant'
[ Parent ]
oh hell no! (none / 0) (#18)
by argent on Fri Mar 02, 2001 at 01:16:54 PM EST

Shortly after I came on board where I work. I was given the task of coming up with a disater recovery plan. I took a survey of what machines we had, what got backed up when, the whole 9 yards. Came up with a plan to do a homegrown SAN network that would provide an off site box with all the necessary back ups. Made a proposal with it. No. Can't do that. Not feasable. (this was about 6 months before SAN became a buzz word, that's what I get for being ahead of the curve.) So I got stuck with having to manually make back up tapes, Which took a full two days out of my week to do, and phyically carry them to a safety deposit box down the street.(about as off site as I could get them to do) I do this weekly for about 5 months. Only to find out that they necessary database that we would need if we ever needed to restore from them, wasn't getting backed up at the same time, and there was no way to get it onto tape at the same time. If we get hit with any type of disaster at this point, first thing I'll do is hit monster.com and update the resume.......... argent
cd /pub more Beer
Would your company survive a disaster? | 20 comments (20 topical, 0 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!