A Federal agency spent over a year and nearly $3 million dollars on expensive actions against non-existent, system-wide malware. The drama finally halted when the agency ran out of money after its request for $26 million to continue the battle was denied.
Some clear thinking before any action was taken would have saved millions of dollars and hours of wasted time. The incident addresses only too well the value of clear thinking—before the desire to act overtakes knowing what to do.
The incident began when Homeland Security advised the Department of Commerce (DOC) of a potential malware infection within its systems. The DOC’s IT security identified the potentially infected computers as belonging to the Economic Development Agency (EDA) and notified them of the problem, first, mistakenly implying it was a system problem and then, in an attempt to clarify the initial notification, only reinforcing the miscommunication.
It is here that some clear thinking could have resolved this incident quickly and inexpensively. The DOC audit of the incident (see link below) found that “Malware infections on EDA’s systems were overstated and the disruption of IT operations was unwarranted.” The DOC vaguely communicated the problem to the EDA which immediately took action without clarifying the situation, assuming it was system-wide, shutting down their systems, hiring expensive consultants, ignoring the consultant’s advice and destroying nearly $200,000 worth of hardware before running out to money.
Asking what is going on in a logical way—and specifically, gathering current impact data, would have quickly indicated that just two computers were infected—not the whole system—requiring an entirely different response. Instead an initial vague communication by the DOC followed by a garbled attempt to correct the first communication and the failure of the EDA to clarify the situation before acting, left the EDA so invested in their actions that when consultants finally specified that only two computers were affected, remediation had already spun wildly out of control and no one was willing to pull back.
The ability to rely on a tried and true protocol brings calm to a threatening situation and accelerates movement towards the best, balanced course of action. For example, KT Situation Appraisal focuses troubleshooters on using clear thinking to understand what is going on through a series of questions. Even within a field of unknowns, the process guides users to face their fears and consider: what is the current impact? What evidence do we have? When someone reports a problem, it is wise to move forward with all the available evidence before throwing money at an unspecified problem based on an unknown threat.
As sequestration forces cut backs, the application of some clear thinking could go a long way to helping the federal government control costs and respond effectively to a rapidly changing security landscape.
Full DOC Audit – http://www.oig.doc.gov/OIGPublications/OIG-13-027-A.pdf