When Disaster Strikes
Campuses along America's southern coasts are familiar with the annual battering they get from tropical storms. So they put a lot of effort into ensuring the safety of their main assets--students, faculty, staff, and facilities--and critical data. When the first storm of the 2005 season threatened Florida's coast at the beginning of summer, people like Bernard Chapple were able to see just how well their planning would pay off. Chapple, who is chief information officer and director of Information Technology and Telecommunications at Edward Waters College in Jacksonville (Fla.), has made a career doing disaster recovery planning and management at organizations as diverse as Merrill Lynch and Toyota.
Though all kinds of businesses and organizations will have many emergency needs in common, there are some needs that only come up in higher education. The needs can be institution-specific. For example, at Chapple's college, which is affiliated with the African Methodist Episcopal Church, collecting students in safe places to wait out a storm involves keeping the men and women in separate dorms. In terms of technology needs, however, many schools are pursuing similar strategies of outsourcing or replicating much of their data to protect it in the event something takes down the on-campus servers. Technology leaders are also working closely with their staffs to develop plans, and then run them through worst-case scenarios. They're meeting regularly with representatives from other departments, too, to make sure the entire organization would work together in times of crisis. "The chief information officer, in particular, must believe in and articulate the importance of developing and maintaining such a plan in order to earn staff buy-in," notes Linda Deneen, director of Information Technology Systems and Services at the University of Minnesota, Duluth.
At St. Mary's College in Notre Dame, Ind., Director of Information Technology Keith Fowlkes says a disaster recovery and business continuity plan for information technology and telecommunications--luckily unused, so far--has been in place for about 18 months.
St. Mary's plan for its IT systems "states that securing and isolating storage devices and systems is our top priority," says Fowlkes. A closed network protects and secures data and equipment, and the school has agreements with several external vendors and service providers to help out in time of a disaster, for both telecommunications and hardware repair.
Yet, he warns, "even iron-clad contracts are a variable in a regional or national disaster event." That's why having good internal staff members who are committed to the institution's mission and have the best training possible is so important. Fowlkes also advises having "as many backup components as your budget will allow. This is especially true for small schools that will almost inevitably be lower priorities to vendors who also service large institutions and businesses in their regions."
many backup components
as budgets will allow. This
is especially true for small
schools that will inevitably
be lower priorities to vendors
who also service large
institutions and businesses
in their regions."
St. Mary's College (Ind.)
Institutional data is the top priority for Rosie Quelch, network manager at The Isle of Wight College (UK). "Software applications and server hardware can all be replaced," she says. To provide that backup service, the college relies on storage and backup consulting services by GlassHouse Technologies. The company's team designed a backup and archiving system (using IBM Tivoli storage management, xSeries servers, and Overland storage systems) that reduced backup time for the college from 14 hours to two, as well as shortened the times of other IT management tasks.
Toronto's Humber Institute of Technology and Advanced Learning also considered Tivoli but instead went with storage and backup technology from CommVault Systems. The solution delivered the same benefits Isle of Wight College got--speedier backups of critical data on a system that is easier for the IT staff to manage.
"We are now very confident in the data," says C. M. Shum, manager of Technology Services Support at Humber. His team is using the time freed up by the new system to research other technologies that will further the institution's goal of data protection and integrity.
Chapple seems to take a bit of delight in reeling off the names of the four hurricanes that he had to deal with in rapid succession in his first year at Edward Waters. (For completists, Charley, Frances, Ivan, and Jeanne slammed into Florida in August and September 2004.) Such disasters are not what-ifs; they are practically an annual event. So it makes sense that the school brought in a disaster-preparation expert. Chapple, a member of the faculty of the International Disaster Recovery Association and co-author of Information Security: The Complete Reference Guide (McGraw-Hill Publishing, 2003), joined the school in mid-2004. Immediately, he set about creating plans, holding meetings, and directing his staff to be ready for crises.
This year, he's continuing those efforts. Chapple plans to outsource the school's web systems so, for example, Edward Waters students traveling out of state can access the school's web systems and get information in the event of trouble. He's working on a plan to outsource the institution's critical data storage to an off-campus company, both for access and recovery in the event of natural disasters and to increase security of the data. Colleges are becoming juicy targets of data hackers, Chapple notes.
Tabletop rehearsals with the various business units on campus are another way Edward Waters' administration stays prepared. Chapple once created a fictional hurricane (named after his daughter) and had it moving toward the school on a specified route. " 'It's going to hit Jacksonville in X hours. What are you going to do? Admissions, suppose you have recruiters on the road; what are you going to do?' " he recalls asking his team.
"It's not rocket science," Chapple says, "but you have to have dexterity."
campus help Edward Waters College (Fla.) administration
stay prepared. With, say, a fictional hurricane expected to
hit in X hours, what is each department going to do?
For IT staff at University of Minnesota, Duluth, the top priority during a disaster is to restore help-desk service, "followed closely by restoring our phone and data services," says Deneen.
Enterprise data is already managed off-site, at the Twin Cities campus of the University of Minnesota. "We do have important data [in Duluth], but most of our critical data is stored at the Twin Cities' campus." Duluth's disaster plan was renewed and updated this year. Though it has not yet had to be used in a real emergency situation, administrators have run tests of the system during each of the past two years.
Key to restoring lost data and resuming business operations after a disaster strikes is a safe and sound backup system. Companies such as Blue Hill Data Services, LiveVault, and NSI Software provide backup services in the event that on-site data is lost. Fusepoint Managed Services, Rackspace, and others offer a range of services such as backup, server hosting, and disaster recovery plans.
"We are very pleased with the software we use for backup, redundancy, and recovery," says Fowlkes from St. Mary's. SyncSort software, with redundant backup storage systems placed in different campus locations, and tape backup to an ADIC Scalar system get the job done. "All these storage systems are linked via their own isolated gigabit network. The functionality of the SyncSort software is great and platform independent, which is valuable to us because we support Sun Solaris, Windows, Linux, and Mac OS X server systems." Off-campus storage of media is currently handled by staff members, but Fowlkes says his department is planning to move to a local provider for off-campus storage in the near future.
Off-campus providers are central to many disaster-recovery and business continuity plans. Options range from off-site storage (almost every organization at one time or another used to have a staff member designated to take backup tapes home every day, though now most have moved to automated off-site backups and storage) to rerouting of internet and telecommunications traffic (so schools can keep in touch with all concerned parties, which often includes students on out-of-state or even international research trips).
Experts say the key to getting the right third-party support service is specificity about the needs of the institution and the capabilities of the provider in the event of a disaster. Make sure that any vendor has a package that matches the school's needs. For example, would you just have data replication and backup services with them? If so, would you be able to continue the school's business if your on-site network and servers were actually destroyed and not just taken offline for a couple days?
Once a vendor is selected, be clear about the services you expect, as well as the level of quality control you demand (and do demand it). Also, be a pessimist, experts advise. Ask questions like: What if our backup fails? If the backup service is across town, would we likely be okay if the emergency is a campus-based, man-made one? And then there are natural disasters to consider. If a hurricane or tornado devastates the community, your cross-town backup has just been an expensive lark. They're all tough questions. But asking them now sure beats an "Uh-oh!" exclamation later.
John Burton is the West Coast correspondent for University Business.