San Francisco not an exception in giving network admins too much control

July 23, 2008 (Computerworld) The city of San Francisco's IT department is certainly not the exception when it comes to allowing just one person to have unfettered rights to make password and configuration changes to networks and enterprise systems.

In fact, it's a situation fairly common in many organizations -- especially small to medium-size ones, IT managers and others cautioned in the wake of the recent Terry Childs incident. Childs, an employee working for San Francisco's IT department, used his privileged access to lock everyone out of a crucial network for days.

A network administrator working for San Francisco's IT Department of Telecommunications and Information Services (DTIS), Childs was arrested on July 13 for allegedly tampering with the city's FiberWAN network. He is also alleged to have planted network devices that enabled illegal remote access to the FiberWAN network, which carries almost 60% of the city government's traffic.

He was jailed on $5 million bond after refusing to divulge the passwords he had used to block access to the network. Childs pleaded not guilty to the charges against him at a hearing in San Francisco Superior Court last week and asked for his bail amount to be lowered.

At a bail hearing Wednesday, a San Francisco Superior Court Judge refused to lower the bail, even though Childs in a dramatic move earlier this week disclosed the passwords to Mayor Gavin Newsom in a jailhouse meeting. His next hearing is scheduled for September.

The episode and the city's struggle to regain full access to its locked network have highlighted the dangers involved in handing over too much administrative control of networks and IT systems to a single individual.

But the situation exists more often than imagined, said Matt Kesner, chief technology officer at Fenwick and West, a San Francisco-based law firm. "It's probably more common than we'd like to think. This is the kind of nightmare I could see happening at many organizations," Kesner said, citing his own experience working at other companies.

"There often is a single networking guru who really does have the keys to everything. You have to work very hard to make sure that more people have the keys," and that there's infrastructure and processes in place to enforce that, he said. Often, though, companies simply don't have the resources or the skills needed to really do this.

"Unfortunately, it is not that uncommon to come into a situation where one or two people have created a situation where not only are they the only ones that know what is going on, but they are the only ones that can do anything," said Lou Michael, director of network and infrastructure services in Virginia's Arlington County department of technology services.

The issue isn't just control over passwords, but also over documentation relating to configurations and changes. Often in situations such as this, "requests for access, passwords and documentation are frequently taken as hostile acts by those that have been holding the keys to the kingdom," he added. "In my experience, I have encountered this type of situation on more than one occasion," he said. In one incident, a mainframe systems programmer had to be fired for changing access rights because he disapproved of others' activities on the system, Michael said. In another case, the individual resigned when he "realized that the pressure to follow processes and procedures was not going to go away despite the protesting," Michael said.