Friday, August 27, 2010

Court Approved?

I continue to hear this phrase mentioned by fellow forensicators in email lists and at conferences, so I thought I would, once again, help to dispel the myth.

THERE IS NO SUCH THING AS COURT APPROVED TOOLS.

Saying that one tool is court approved and another is not, is like saying you can take crime scene photos with a Nikon, but not a Kodak. It's just silly, and it's a myth perpetuated by those who seek to benefit from the existence of such a rumor.

Now, there ARE tools that have been used in court cases, which may be more familiar to attorneys and/or judges. This does NOT make them court approved, it simply means that they have been used before...nothing more. Pay careful attention to what I am writing here...simply using a tool...any tool...DOES NOT make your findings any more relevant, valid, or indisputable then if you had used any other tool to come to the same conclusions. The data is simply the data.

Your job as a forensic investigator is to produce forensically sound results. This too is a term that is often used incorrectly or as a buzz word. Forensically sound means that if given the same set of data, any other investigator, using any other tool, would come to the same conclusion.

Now really think about what this means. Let's say you have been asked to identify a date range for files in a specific directory. If given the same image, 10 different people, using 10 different tools, should come to the exact same results...EnCase, FTK, TSK, MFL, Perl scripts, Python, whatever...the conclusion should be the same because the means by which you would extract that data is the same.

The implications of a conclusion being forensically sound invalidate the entire premise of something being court approved. How can one tool that comes to the same conclusion as another tool be approved while the other is not? They DO the same THING. While the GUI may change, or the vendor - open source code versus proprietary - Linux versus Windows...it doesn't matter. The data is the data.

6 comments:

  1. Chris,

    Good post...

    "How can one tool that comes to the same conclusion as another tool be approved while the other is not?"

    This is actually a very good question...because in a lot of cases, I do think that analysts look to the tools to make the conclusions for them.

    Tools display data, or provide a layer of abstraction, translating binary stuff into something understandable to a person or an analyst, as decided/designed by the tool designer. Tools do not interpret, nor do they make conclusions. That's the analyst's job.

    ReplyDelete
  2. I have been repeating this mantra lately a lot as the question of best forensic tools seems to keep coming up:
    "
    There are no forensic tools.

    There are tools that forensic practitioners use in the course of gathering evidence and performing analysis.
    "
    The CF books and LE guides/PDFs from 5-10 years ago are so still so relevant because they are methodology not tool based to their approach to investigations because there were limited commercial tools.

    I have get so sad/disappointed/angry when the question is "What should I use?" instead of "How should I approach the situation?" I would like to see the mindset change.

    ReplyDelete
  3. Chris,

    Your post brings up something that I think is overdue in the forensic community. People like Harlan and others have led the charge when it comes to getting people to stop resisting methods like live analysis.

    The next thing I'd really like to see is getting away from the definition of "forensically sound" being that you didn't change any data. I think a definition that revolves around the concept of repeatable results using sound practices makes more sense.

    ReplyDelete
  4. Court approval of tools and methodologies for evidence happens as part of the Daubert process for US Federal courts, and most states have adopted Daubert or Daubert-like guidelines. Those states that haven't adopted some form of Daubert, which are few the last time I checked, use either the previous Frye or Frye-like Federal standard, or completely make up their own evidence standard. By any standard, a tool and methodology must still stand on it's own merits on a case by case basis. Thankfully, proper methodologies for evidence collection, handling and analysis are well documented, and proving a tool's output is reproducible is relatively easy these days thanks to NIST and countless other publications.

    ReplyDelete
  5. This statement simply is not true. You assume that all tools produce the same results and that they are only used to affirm findings. Both assumptions are wrong.

    Tools are also used to disprove findings. For example, if a tool finds matching code in two software programs to determine copyright infringement, it's not particularly relevant how that code was found (though some tools will be more efficient and thus find more copied code). However, in proving that no code was copied, the tools must be the most accurate available and must perform the analysis correctly. In other words, a result showing no copying must not be due to problems in the tool.

    Suppose you are involved in a murder case and you use a DNA matching tool that isn't very accurate (has not been used previously and has not survived a challenge in court). You probably wouldn't want a suspect ruled out based on the DNA tool not being able to correctly find a match even though one existed.

    I have unfortunately been on the opposite side of experts who have used unproven, unaccepted tools to make a case that simply wasn't true.

    ReplyDelete
  6. Bob...I think you missed the point of my post. The issue is not the accuracy of a tool, but a tool selling itself as being "court approved". Which I totally stand by...in the digital forensic world, there is no such thing. I challenge you to produce one if you disagree.

    Now, a tool is nothing more...something that does something. If that something modifies the sample in any way, then it should not be used. If the tool misses a match that is there, then it should not be used. I think again, that is a different issue. That is the analyst knowing the tools in his toolkit...knowing how to use them, and knowing what the results are going to be.

    When presenting in court, I will use several tools...all of which come to the same conclusion to eliminate any appearance of error on my part or the chosen tool's. ALSO, I know WHAT precisely my tools are doing. I don't operate on "auto-pilot" and simply load a sample, click a button, and report on the findings. I think it's easy to consider the former, "forensics" and consider that practitioner an, "expert"...which is nowhere near the truth.

    ReplyDelete