Speaking of Automated Systems

You may also like...

3 Responses

  1. Samir Chopra says:

    Danielle, believe it or not, one chapter that got left on the cutting floor addressed precisely these issues – and it was a direct spin-off from your paper. I’ll try and write a separate blog post on that later. Thanks for reminding me of this.

  2. Danielle Citron says:

    I so look forward hearing all about it! Much thanks for the terrific book and for letting us discuss it. Danielle

  3. Danielle, the history of development of large software projects is replete with expensive failures. One, the FBI’s Trilogy project, was so troubled that they asked the National Research Council to study the situation. The report is at http://www.nap.edu/catalog.php?record_id=10991; the committee’s conclusion was blunt: the effort was “not currently on a path to success” [emphasis in the original]. Regardless of the issues of legal personhood for automated agents, the problem of managing such complex efforts is immense. One can argue whether the blame lies with the procuring organization or the contractor; often, they share the responsibility.

    To use a very bad analogy, the systems resulting from such flawed efforts are, if legal persons, “handicapped” because of incompetence or negligent choices by its “parents”. Where then, should liability, civil or criminal, lie? Bad outcomes are not inevitable but they’re remarkably common. (Those interested in the subject should follow Peter Neumann’s RISKS Digest; see http://www.csl.sri.com/users/neumann/#3 for a precis, and perhaps skim ftp://ftp.csl.sri.com/pub/users/neumann/illustrative.pdf or read Peter’s 1995 book Computer-Related Risks.)