By Jay Heiser

Last month's Business Software Alliance report on cyber security (pdf) concluded that cyber terrorism was going to be really serious, so everyone should protect themselves by giving more money to the members of the Business Software Alliance. How did it reach this conclusion? No, not by using professional intelligence experts or foreign affairs specialists, but by asking corporate security officers for their opinions.

OK, so it's hardly the first time that a commercial interest group has conducted such a flawed study. But it is disappointing to see professional academic researchers following the same pattern of asking security experts if they feel under-appreciated, and then claiming that their unanimous affirmative response is categorical proof that security expenditures are too low.

Created at Dartmouth College, the report Law Enforcement Tools and Technologies for Investigating Cyber Attacks (reg req'd) starts with an assumption that is not substantiated within the document: cyber attacks are a significant threat. It implicitly suggests that because the digital forensic tools are so bad, law enforcement will be unable to protect us from these attacks. The explicit conclusion is that there must be a national agenda for the research and creation of law-enforcement specific investigation tools.

Typical questions posed to law enforcement investigators read "In general, I completely satisfied with the tools I have available for..." It's hard to imagine anyone choosing 'strongly agree,' when asked if they are completely satisfied with any software, let alone forensic tools. Questions on the perceived shortcomings in investigation tool features had ' lack of law enforcement-specific features' as one of the possible responses, and it should not be surprising that this was a popular answer.

Any system administrator can sympathize with the difficulties in analysing log files, but it is hard to imagine what features would be useful to law enforcement that haven't already been considered by the dozens of startups that have yet to provide a useful log consolidation and reporting tool for corporate use. All investigations-both physical and cyber-include long and boring manual examination of evidence. We didn't need this report to explain that the analysis of system logs is boring.

It's easy to envision the staff at Dartmouth brainstorming topics for interesting research topics that would help put their new Institute for Security Technology Studies on the map. Did they deliberately design a survey that would inevitably conclude such research topics were vital to national defence? This report, bankrolled by the US Department of Justice, gives that impression. It will now be used as evidence to justify requesting additional public money on security software, an area where 25 years of government sponsorship has resulted in virtually no useful technology.

Like all the other self-serving surveys, much of the substance of this report is reasonable. Forensic experts recognise that better tools would be a big help, but few would claim that the relative immaturity of today's tools is 'one of the critical public security and national security issues of the 21st century'. It was always clear that digital forensic products could withstand improvement, but nowhere does this report ever offer any evidence that the future costs of cybercrime (or as they prefer to refer to it 'cyber attacks') will be unacceptably high without immediately ploughing more public funds into R&D.

Why should we accept the conclusions within studies such as this and the BSA report, when the studies themselves are so contrived? Sponsored by organizations which want to obtain more of our money, and eagerly devoured by reporters who would rather titillate than educate, flawed 'research' doesn't help decision makers better understand what needs to be spent to provide an appropriate level of protection.