Fergie's Tech Blog

The EFF Digs Deep Into The FBI's 'Everything Bucket'

Jon Stokes writes on ARS Technica:

Earlier this week, the EFF published a new report detailing the FBI's Investigative Data Warehouse, which appears to be something like a combination of Google and a university's slightly out-of-date custom card catalog with a front-end written for Windows 2000 that uses cartoon icons that some work-study student made in Microsoft Paint. I guess I'm supposed to fear the IDW as an invasion of privacy, and indeed I do, but given the report's description of it and my experiences with the internal-facing software products of large, sprawling, unaccountable bureaucracies, I mostly just fear for our collective safety.

The idea behind the system, which the FBI has been working on since at least 2002, is that the Bureau can dump all of its information in there so that it can be easily searched and shared. IDW contains more documents than the library of congress—a stew of TIFFs with OCRed text, multiple Oracle databases, news streamed in from the Internet, reports and records in various in-house data formats, watch lists, telephone data, and an alphabet soup of smaller databases and records repositories—all accessible as one sprawling system that processes batch jobs, runs queries, and issues alerts. In short, the IDW is an "everything bucket" for the FBI.

Complicating the picture is the fact that some parts of the system are classified as "secret," while others aren't. I'm sure the entire thing is a joy to use.

posted by Fergie @ 4/30/2009 09:30:00 PM