Opinion: When good disks go bad
It's easy to forget that when things do go wrong, they can go very wrong
September 18, 2007 (Computerworld) --
Despite their increasing complexity in terms of both size and functionality storage systems have achieved an impressive level of reliability. This is particularly noteworthy given the fact that they are engineered around electro-mechanical devices (i.e. disks) that are among the components most prone to failure within the datacenter. The safeguards and redundancies designed into modern storage systems routinely handle most device failures as a regular matter of course with little or no impact to the overall operation.
As is often the case with technology, they have reached a point where this reliability is often taken for granted, especially by those who aren't spending their days (and nights) on the storage management front lines. It's easy to forget that when things do go wrong, they can go very wrong. Occasionally, it's helpful to be reminded.
A friend of mine, who is the CIO for a mid-sized organization, recently shared his 72-hour nightmare experience with me. Their storage system, which housed key applications and email inexplicably went down one evening. After contacting vendor support, they learned that the apparent reason for the outage was a known firmware bug that caused the controller to think that there were multiple drive failures.
Now, one might ask why the organization, under a valid support contract, had no prior notification about the firmware update to address such a serious problem. It seems that such a notification should have occurred, but hadn't.
If this had been the only problem, the outage, while serious, could have been resolved in fairly short order. Unfortunately, the problem was exacerbated by a series of tech support mishaps in the firmware update and system recovery process that led to multiple rebuilds and an extended period of time with the system needlessly operating at risk in a severely degraded mode.
This organization fell victim to what can be described as the Achilles' heel of storage infrastructure - the intersection of technology bugs and human error. This is a highly unpredictable type of risk, and unfortunately the opportunities for prevention and avoidance are few. Some things that can be done to reduce the likelihood of such a situation include:
- Verifying that you are receiving notifications of critical patches and updates
- Keeping configuration management information current
- Establishing a process to quickly flag and update at-risk systems
- When dealing with vendor technical support in critical recovery situations, triple-check, escalate, and obtain expert approval.
Even with these steps, however, there is no guarantee of risk elimination, as my friend can now attest.
Jim Damoulakis is chief technology officer of GlassHouse Technologies Inc., a leading provider of independent storage services. He can be reached at jimd@glasshouse.com.
|
|
|
"The often-overlooked hard disk password feature can offer improved security. On some computers, hard disk passwords can be used in..."
Read more...
"One defensive computing topic that doesn't seem to get the attention it deserves is protecting the hard disk with a..."
Read more...
Read more Storage posts  or See all Blogs
|
The downturn has softened the IT talent market but done little to weaken demand for SAP, .Net and other technical skills.
Every computer user hits a speed bump now and then. Here are some speedy, simple solutions to hardware, software, network, Internet and mobile-device crises.
From the iPhone 3G to 'unibody' MacBooks, 2008 was a standout year for Apple.
We've got reviews and videos of the new Ubuntu 8.10, Fedora 10 and openSUSE 11.1.
Get the latest news, reviews and more about Microsoft's newest desktop operating system
Find wage data for 50 IT job titles.
|
 |
| Enabling Data Centers that Are Both Automated and Dynamic  Enabling Data Centers that Are Both Automated and Dynamic
View this webcast now! Go to the webcast |
|
| Virtual Reality  Download this Computerworld briefing, a $49.95 value free, compliments of Riverbed Technologies.
(Source: Computerworld) Is your organization facing the struggles of ineffective capacity utilization, growing data volumes, labor intensive storage management, and a need for better disaster recovery?The data center is real, but storage is turning virtual at many organizations that need to manage these exploding storage needs. Learn how your organization can benefit from storage virtualization in this new Computerworld Report, available free for a limited time, compliments of Riverbed. Download this executive briefing |
|
| Brocade and the File Area Network - A Taneja Group Solution Profile  Get this white paper now!
(Source: Brocade) This Taneja Group report examines how Brocade FAN solutions are creating a stateless end-to-end file and block data infrastructure. Download this white paper |
|
| White Papers  Read up on the latest ideas and technologies from companies that sell hardware, software and services. | | View more whitepapers |
|
|
Subscribe to
Computerworld 
or
Other Storage Stories
Business Continuity Zone
MessageLabs is offering a complimentary 30 day trial of its managed Anti-virus and Anti-spam security solutions. MessageLabs guarantees complete protection against all know and unknown email threats. By providing 24 hour support, your business can increase productivity and decrease risk. Register for a complimentary trial and receive a free datasheet.
Computerworld