How Do I Get There From Here - Part 2

So, I have had more folks ask me about a career in Incident Response and Computer Forensics lately, so I thought I would expound a bit on my original post, How Do I Get There From Here.

I think it's worth mentioning again, that the thing that will propel you in your career, regardless of what that may be, is sheer desire.  Getting up everyday, and thinking that you are not going to work because you have to, but rather that you get to go work doing something you love, makes a huge difference.  I cannot stress that enough...to be a really good investigator, there is no other way.

Something else that I have recently discovered (thanks to some great Detectives) is a great skill to have is the ability to spot patterns and anomalies.  So much of what we do in solving cases begins with finding something that just doesn't look right.  You don't have to know exactly what it is, but you know something is just off will lead you down the path of taking a deep dive into that, "thing" which will either prove or disprove your hypothesis.  Then, Sniper Forensics baby, you either use that finding to guide your investigation further, or you step back, formulate a new hypotheses, and drive on.  But that initial "hrm...what are you" moment, is something you should experience throughout your investigations.

I spoke about this at a conference once, and I was asked, "How do I learn how to spot anomalies "  Which is a valid questions...to which I answered, "By knowing what "normal" looks like".  You need to put in the chair time.  You need to know what processes should be running, from where, what is common, why - basically what makes a normal system look normal.  I was a sysadmin for many years before I ever moved into security, which helped me tremendously once I moved into the DFIR world.  If you don't have that background, then virtualization is a great thing.  Fire up some VMs of different operating systems and just look at it.  It sounds boring...but you know...wax on wax off...

Spotting patterns is a bit different.  It requires you to be able to look at data elements and find similarities in them that could be anomalous.  The best example I can think of is reviewing web logs for IOCs of SQL Injection or RFI.  If you have ever seen these attacks in logs before, you know what I am referring to.  You can actually see patterns of the attacker walking the database structure.  If he's using an automated tool to do this, you can spot it a mile away - if you scroll through the logs, it looks like a series of shark fins.  The same holds true for RFI attacks...you can spot the pattern of the attacker trying to get the system to upload his file.  This is also the case for several different kids of attacks...they have visible patterns that after you put in some chair time, you can spot.  Again, even if you don't know exactly what you're looking at just that it's unique when compared to it surroundings.

OK Chris...that's all well and good in theory, but that does not help me find a DFIR job.  Do you have any recommendations that will help me actually get in the door?  Great question...and Yes...yes I do.

OK...Bit of history...when I was a sysadmin at American Express in Phoenix, I used to admin both Windows  and *nix servers (Solaris, AIX, and Linux).  It was pretty cool, but kind of boring as it didn't present anything in the way of challenges (at least for me...no offence to Sysadmins...that's my roots!).  So, I started looking into this whole security thing (this was about 2001).  Pentest looked kewl to me.  I knew how to make stuff work...let's see if I can learn how to break into those same systems.  Since I didn't actually have a security job, I couldn't actually DO anything security related at work.  So, I bought a copy of VMware, and started playing with tools.  What was Metasploit and what did it do?  What is ARP spoofing...can I do that at home?  Basic research in my home lab.  So, when I finally found an opening and got an interview, I was able to tell the hiring manager that all I have is what I found in the open source community, and my home lab, but I practice and research at home.  I read books, blogs, and whitepapers trying to get as much knowledge as I could without actually doing the job.  Well, I got the job for that very reason.

All of that to say...do that.  If you want a job in DFIR and you are not currently working in DFIR, then research in your home lab.  Take images of your systems, your ipod, your buddies laptops...whatever and start to play with the tools of the trade.  Learn how to mount images, create timelines, parse data on the command line, learn how to use grep, gawk, and cut, use RegRipper to inspect registry hives...etc.  Knowing which tool to use, when and why is critical!  Remember, I rarely ever use commercial forensics tools.  You can conduct comprehensive investigations without ever spending a dime!

So, if when somebody interviews you, and you tell them...I don't do this for a living but I want to and here is what I am doing to prepare myself for that, that should speak volumes about the type of employee you would be.  I know for me, you would certainly shoot to the top of my list.

I hope that helps clarify things a bit for those of you that are seeking careers in DFIR.  If there is something you would like me to expand on, please let me know!  Or, if there is something I mentioned that you would like me to dig deeper into, please let me know.  I am more than happy to help!  After all, I may be interviewing you someday.  It would be great to hear that you read my blog posts and so you did X.

Best of luck to you!

A Changing of the Guard

True to form, I am sitting the airport in Tulsa headed to Seattle for Blue Hat, and writing another blog post.  A lot of has gone on over the past couple of months, so let me bring you up to speed and make a bit of an "announcement".

First, and most important (as it sets the stage for my "announcement") I was recently promoted from Managing Consultant of the US DFIR team to Director of the DFIR practice at Trustwave.  I had a great boss in Colin Sheppard who was preparing me for the role, and once he left Trustwave, opened the door and made the recommendation for me to take on his role.  I am truly excited about the opportunity and look forward to the new challenges it brings.

So, that brings me to my "announcement"...I will no longer be posting strictly technical content to The Digital Standard.  I have truly enjoyed blogging about my work over the past few years and have even had the pleasure of meeting some of my readers.  However, since my role has changed significantly, and I won't be working many cases going forward, I simply won't have the content to be able to write as many technical posts that I think would actually be worth reading.

Now, part two of the announcement is that I will be changing the focus of The Digital Standard from in the trenches DFIR work to DFIR Leadership and security management.  Some of you may or may not know, but before I ever became technical, I was a business manager.  I actually have my Bachelor's degree in Business Management.  Additionally, I attended the US Army Warrant Officer Academy.  As those of you who have attended military officer producing schools know, there is a heavy focus on situational leadership (AKA being skull drug) - a skill you either develop or you wash out (for the most part).  That all to say, I actually have some experience in leadership, and feel that I have something of value to share.

Like all of my previous posts, I will intentionally be leaving out certain pieces of information.  Not only do I have NDAs to adhere to, but I have to respect the anonymity of the people I am working with and that work with me.  

For those of you that continue to read my sporadic posts, I hope you find value in my writing.  Working in the DFIR community presents some of the most unique leadership challenges anywhere in the professional world.  We joke with our clients, but it's so true, "Your worst day, in my every day".  Living in someone else's nightmare certainly helps to hone your leadership and problem solving skills!

Also, if there is something that you would like me to research and/or write about, please let me know. 

Also, Also - I am hopeful that my new position will permit me the time to do something I have been wanting to do for a couple of years now, write a Sniper Forensics book!  No promises, but it's high on the "To Do" list for 2013.  I have wanted to do it for many years, and due to the amount of case work I have been doing, I simply did not have the time.  Now that I am not actively working cases, I may actually be able to write for a couple of hours every day.  If any of you have any topics you FOR SURE would like to see covered in the book, please let me know.  I am writing it for you all...I have the information in my head already.  It wouldn't do me any good to write on something I think should be in there, if there are other things that you as the read (and purchaser) of the book would prefer to see.

Here is where I normally say, "Happy Hunting"....but considering the circumstances, I will have to work on something else to end my posts with...hrmmmmmmmm

DFIR 2012

The SANS DFIR Summit is this Tuesday.  This is by far, one of my favorite events of the year.  I will be making the US debut of Sniper Foreniscs: 3.0 - Hunt.

Hope to see you there!

Wetware

And here I sit at another airport, Dallas-Ft. Worth this time, writing another blog post.  And yet again, I am reminded by an issue that continues to plague my forensic brethren.  The heavy reliance on tools.

I am a member of several forensic/IR mailing lists, I read the blogosphere, and I try to keep up with many of you on twitter.  In addition, I have a strong relationship and presence with many law enforcement agencies (local, state, federal and foreign) and the officers assigned to perform DF and IR.  I intentionally don't comment very much, mostly because I don't think very many people would like my answers, but I help out when and where I can.

So to get right down to it, I still see a strong reliance on tools to solve cases for you.  I have also seen a number of posts and tweets recently where investigators are trying to make certain tools do certain things they are either not well suited to do, or where a much better solution exists.  To all this, I say, "stop"!

Stop stop stop stop it!  

Relying on tools to solve your case for you in like relying on a pile of wood and a nail gun to build your house.  It doesn't work.  It's never going to work.  The sooner you come to that conclusion, the better off you will be.  Instead of simply ranting about tool reliance, allow me to explain myself.

All of our investigations are made up of data elements.  Some have evidentiary value, while others do not, but it's all there...plain ole data.  Just sitting there waiting to have something done with it.  The question investigators SHOULD BE ASKING FIRST is what question am I trying to answer, not what tool do I need to use!  How in the world could you possibly know what tool to use before you know what you are going to do, and why?  You can't!

Now, I understand that in some cases there are just "goto" tools.  For example, I use fls in each and every case to create a timeline, I use Log2timeline or regtime to add registry hives into my timeline, I use Reg Ripper to parse my registry hives into human readable text, I always dump log files into flat text with DumpEl, and I always use pstools to dump running process information.  So I get that you have to use certain tools by default to get you to a good starting point...I do the same thing.  But that's about where it ends for me.

I don't always pull web history, I don't always scan an image with AV, I don't always extract the $MFT, and I rarely use EnCase.  Why?  Because I don't always have to!  

For example, when I know malware has been deployed on a Point of Sale (POS) system by RDP, why would I need to pull web logs?  Answer...I don't.  We browser history has nothing to do with my case.  BUT, if I see that malware may have been downloaded...let's say by reviewing ntuser.dat hives from admin users, or from evidence I find in my timeline, then OK, I will grab web history to see if I can find an additional data point that would indicate that my malware was downloaded via the web...it makes sense in that case to do so.

I don't always scan an image with AV.  Why would I?  For those of us that pretty much live and breathe malware,  we know that scanning with AV is only going to be marginally useful, if at all.  It's going to point out known samples or common variants, and that's about it.  If the malware is custom, or is a new variant the scans will be of no value.  You are FAR better off identifying the running processes and looking for common IOCs and APIs used by the different types of malware depending on functionality.  BUT, if I am asked to find all occurrences of malware on a specific system, regardless of what it is or what it does, sure...I will scan it...because that's what I was asked to do.

I don't always extract the $MFT...b/c I don't always have to.  Since a timeline is generated from the Standard Information ($SI) attribute anyway, I already have half of the $MFT don't I?  The only time I would extract and parse the $MFT...which Harlan's MFT.pl is awesome for...is when I suspect chronological (aka timestopming) modification has taken place.  HOW do I know that timestomping has taken place if I don't first parse the $MFT and compare the $SI to the File Name ($FN) attribute?  I have seen it before, and I know what the IOCs are.  I know what signs to look for that would lead me to believe that some kind of modification has taken place.  Things like pre-fetch files that are identical to creation times save for the year, the mili-seconds field being set to all zeros, and files located with other files I know to be components of the malware, with different creation times. 

So OK Chris...what's your point here?  To simply berate us for using and relying to tools to give us information?  We NEED that information to solve cases...how the heck else are we suppose to do our jobs?

GREAT point!  So let me answer...USAGE of tools is OK, and like you said, you cannot do your job without them!  Neither can I.  RELIANCE on tools to do the work for you is not OK...and as Cory would say, "it is the suck".

Step back for a moment and breathe in...and breathe out.  Clear your head and just think.  What are you doing?  What question are you trying to answer?  Why?  What information do you need to answer that question?  What does the data tell you?  These questions are the essence of the Sniper Forensics methodology.  I (among others) have been talking about this philosophical shift for four years and yet there is still considerable resistance in the community, which I really don't understand.

The best tool in your toolbox is your brain.  What Harlan has dubbed, "Wetware".  Think through your cases.  Ask a LOT of questions.  Actually take the time to answers your questions.  Let the data guide your theory.  It's really not that complicated when you break it down into smaller, more manageable components.

I will close this post with a short story.  I was recently asked to assist a LE Officer with a case he had been working on for a month.  I started by asking him a lot of questions...what are you trying to do?  What was the crime?  What information are you hoping to identify?  How will that information help your case if we find it?  How will it change your investigation if the data is not there?  What is the timeframe of the incident?  How do you know that was the timeframe?  What supporting evidence do you have that indicates that timeframe is accurate?  After listening carefully to his answers and writing them down in my case notes, I knew exactly what to do. 

I created my investigation plan.  Indicated what I was looking for and why.  I took notes on where I would likely find that data, what it would generally look like, and what I would do if I found it, as well as what I would do if I didn't find it.  In total, about 30 minutes of pre-work...maybe 45 since I was drinking a cup of coffee and typing at the same time.

When I actually put fingers to keyboard I found what the officer was looking for, and helped him solve the case in...wait for it...waaaaaaiiiittt for it....15 minutes.  He had been haphazardly looking for "bad stuff" for a full month...four weeks...30 days.  It took me longer to write my notes and drink my coffee than it did to find the evidence he was looking for.  Why you ask?  Because I took the time to use my Wetware!  I actually THOUGHT about what I was going to do, why, and what I was looking for before I ever put my hands on the keyboard, mounted an image, or touched any piece of data.

OK Chris...whatever...you show off...it's ONE case.  You got lucky.  My cases are complicated...it's simply not that simple for me!

Good point...and maybe you are correct.  BUT, I have been using this methodology for four years, in each and every case.  For us in the SpiderLabs, that equates to just under 1000 cases (yes, we keep track).  So my team, in almost 1000 cases have seen this methodology work each and every time.  Without fail, and without exception.  Small cases with a single piece of evidence (like an SD card) to huge cases with hundreds of systems.  It just plain works.

So, for all you naysayers out there, for the skeptics and the old school "pull-the-pluggers", I say,  "try it".  Try doing it my way.  What do you have to lose?  Certainly not more time!  What do you have to gain?  How about solving your cases in a fraction of the time you currently solve them in?  How about clearing your ever increasing backlog?  Sounds like a pretty safe trade to me.

Happy Hunting.

Sniper Forensics: Reloaded Part 1

Check it on on SpiderLabs Anterior!

http://blog.spiderlabs.com/

SecTor 2012 First Round!

Sniper Forensics: Reloaded has been accepted in the first round of CFP selections at SecTor 2012!

This is one of the best security conferences of the year!  If you have never been, I HIGHLY recommend it.

Repins Forensics

OK...so this my interpretation of, "Bizarro" forensics, which sadly, REALLY still happens.  I was recently reminded of not only this, but of just how big the forensic world is, how many investigators there are, and how far we still have to go.

Sniper Forensics (SF) is the targeted approach to conducting forensic investigations.  It helps the investigator to use logic to guide his/her investigation to find answers, not just gather data.  Now, I am not going to rehash SF, but I just wanted to mention it briefly for the purpose of comparison.

The opposite of SF would be to illogically and haphazardly gather data that may or may not be relevant to the case (who cares if it makes sense, just pull the plug and gather everything).  You would form your theory of what you or your client thinks happened, and force all of your evidence into that theory.  You would ignore any evidence that was contrary to that theory, and think anyone who actually questioned what you did or the way you did it, is just plain wrong.

When asked questions like, "Did you perform Registry Analysis, Memory Analysis, or Pcap Analysis" you say..."Yes...I found nothing of evidentiary value".  Then I may ask, "OK, what kind of analysis did you perform?  What were you looking for?"  You would answer something like, "I did analysis...I didn't find anything".

You would then defend your lack of findings by stating that the evidence was not clear what took place.  While you can never be 100% certain of what happened, based on my analysis and experience, there was no breach that is evident.

This may sound ridiculous, but it is sadly true.  There are still investigators that think like this, and cases that work like this.  Do you know of any?  I would love to hear your stories of Bizarro Foreniscs!  Email them to me and I will create a blog series with the best of the worst.  Should be entertaining!

I am working on my example now.  It's pretty bad...and yes...we found the breach and it was ugly!

Delivering GSR at ACFE Greater Chicago Chapter

I will be delivering the Trustwave Global Security Report at the ACFE in Chicago this week!  Really looking forward to it!  Hope to see you there!

The Core Duo

"The Core Duo"

Once again I am sitting in an airport (LaGuardia this time) writing another blog post.  The good news, is that it looks like my life is taking back on some semblance of normalcy, and I can start writing again.  Since my promotion to Managing Consultant, in conjunction with the release the Trustwave Global Security Report, I have been swamped.  But, that's all I have to say about that.

So I have recently been doing a lot of speaking and teaching, and came to an interesting conclusion about what are the core (an in my opinion, critical) skills of our trade, which I have affectingly dubbed, "The Core Duo".

When I really started to think about it, what we do (Forensics and Incident Response) really boils down to only two things. 

1. Spotting Patterns

2. Spotting Anomalies

Now, I know this sounds really simple...maybe too simple, but let me explain.  First of all, simplicity is something that I think is frequently minimized as being undesirable.  I think there are a lot of folks who think something to the effect of, "If something can be explained in simple, easy to understand terms, it must not be very complex".  I challenge that this is not the case.  I think, that even the most complex situations (which we all know, cyber investigations are among the most technical and convoluted anywhere) is made up of components that can be broken down and simplified.  Being able to do this is a critical element in actually understanding what you are doing and why you are doing it.  That in turn leads to be successful at what you are doing.  Which finally, leads to you solving the case, and potentially, some bad guy going to jail.

Ok...so think about your "typical" case.  You have stuff (technical term)...RAM dumps, volatile data, forensic images, maybe some log files.  You are asked to find something (again, technical term) within that stuff.  What are you asked to find will depend on the case, but the theme is the same...go find something specific.  Now the fun begins as far as I am concerned...how do you find the something within the stuff?

Well, there are these things we commonly refer to as, "Indicators of Compromise" or IOCs.  They are data points that indicate the presence of something within the stuff.  So I was thinking, "What makes one data point an IOC and another data point not an IOC?"  I argue that it's because that something has to fall into one of two categories.  It is either an anomaly, or a pattern.

Let's first explore what I mean by using the term, "Anomaly".  It just means that the something is different than all of the other somethings.  There is something about THIS something that makes it different.  WHAT is different is, why it's anomalous, what does the anomaly mean, etc...that's the easy part.  Binaries can be extracted, log files can be parsed, ripped reg hives can be clearly read and compared with forensic timelines...all the things we normally do in a case.  BUT (and this is a pretty big but) ALL of that...all of the tools that exist, all of the book and blog posts that have been written, all of the conferences we attend, and all of the money we spend on training...they all hinge on this one thing...can we spot the anomaly.  Can we find the one or the two (or whatever) files, amongst the tens or hundreds of thousands that are like the kids on Sesame Street, doing their own thing?  THIS is the first core of our trade.

The second core item of the "Duo" is the ability to spot patterns.  In my opinion, this applies more to incident response than it does to forensics, although by no means exclusively.  Think about a case in which all you have are log files.  This is common in my world in E-Commerce cases involving things like SQL Injection, Remote File Inclusion, and Web Shells.  Some of these cases literally involve millions of lines of log files that at first glance, all look more or less the same.  Anyone who has worked on an E-Comm case knows what I am referring to.  Line after line after line after line of logs...and you are thinking to yourself, "What the heck am I even looking for?" Me?  I am looking to identify the first sign of an anomaly, then for a pattern of those anomalies.

Some of the logs entries will have something different about them...it's WHAT is different, and HOW we spot them that is the real meat of the analysis.  Then, once you spot the anomaly, you would look for a pattern of that anomaly.  Does it occur at regular increments?  Does it originate from the same location?  Does the access show the same thing or things being accessed over and over...can you spot the pattern?

So what does all this mean.  OK Chris, we agree with you, to be good at Forensics and/or IR you have to be good at spotting anomalies and patterns.  So what?  What does that mean to me?  Well, I am glad you asked!  A couple of things.

1. It gives you an area of focus.  Still, one of the most common issues I see while training investigators is analysis paralysis.  Simply freezing  when confronted with so much data...what do I do...where do I go...how do I even begin?  How do I go from gathering data to actually starting to solve cases.  

I think that by knowing, that in every case, regardless of what the case may be, you are either looking for an anomaly or a pattern can help you focus on the task at hand, and get down to making progress (not just throwing tools at data and expecting it to solve the case for you.  FORENSIC HINT:  THAT will NEVER happen .

2. It helps you as you try to tie your data points together.  In an "average" case, we may have data points in our timeline, registry hives, RAM, system event logs, and from binary analysis.  We have a lot of data points that all indicate that a breach took place, data was accessed, malware was installed, basically tell the story of what happened during that breach.  By understanding that what you are looking at are a series of anomalies that form a pattern.  This pattern is the backdrop against which you will formulate the rest of your investigation.

So, in conclusion, I know it may sound simple, but what I have named, "The Core Duo" of forensics/IR is the ability to spot Anomalies or Patterns.  Knowing and understanding this concept can help investigators (both new and seasoned) begin difficult investigations with a better idea of what something they are trying to find within the stuff.

On a side note, since I have begun using this concept, the students that I have taught (and those in the audience at the conferences I have spoken at) have really liked the idea, and have indicated that it really does work.  Once gentleman recently told me that he wrote them down on a sticky note, and attached the note to his monitor.  

It very simply stated.

1. What is the anomaly?

2. Can you spot the pattern?

I think that's a great idea!  While I don't have sticky notes on my monitor, I DO have Case Notes, and in my notes, I now have an entry under my "Misc" tab that indicates the same thing.  I urge you to give it a try and let me know how it goes.

Happy Hunting!

Will You Come to Church With Me This Easter?

So, I'm not sure how many of you know about my personal life...not many I imagine, but I am a Christian. This is my testimony.

I am a sinner, and in need of a Savior. I believe that Jesus Christ is the Son of the living God, who gave up his life on a cross on a hill more than 2000 years ago so save me from my sin. Because of Jesus, I can live a life of victory. Free from my sin, free from the grip of the enemy, and free to be everything God created me to be (like a Lethal Forensicator).

This week at my church, our pastor (Dr. Alex Himaya) challenged us to invite three people to church this Easter Sunday, and to pray for them. Well...I work remote...my kids go to a Christian School...and pretty much all of my friends in Tulsa are Christians. Not wanting to dismiss his challenge with a simple, "Sorry...nobody here for me to invite", I thought of my sphere of influence. I wanted to change my mindset to who CAN I invite, not why I can't invite anyone.

So, I was thinking...my sphere is the forensic/IR world. I have contact with hundreds of fellow Forensicators via this blog, Twitter, and FaceBook. So, I am going to reach out to all of you, around the world, and ask you...

"Will you come to Church with me this Easter Sunday?"

If you would like to attend, you can listen LIVE online! Just click on the link!

It is my prayer that wherever you are in your life, you realize that God loves you because of who you are...not because of what you have done, or what you could ever do. You cannot earn God's love...it's a gift. Given freely. You just have to accept it.

I hope you take this opportunity and join us this Easter Sunday at The Church at Battle Creek, in Broken Arrow, OK via the web.

Happy Hunting...er...Hoppy Easter! (haha...get it...)

New York Metro Infragard

I will be presenting the Trustwave Global Security Report at the New York Metro Infragard Summit on March 14, 2012. The event is open to the public and free of charge.

Hope to see you there!

Sniper Foreniscs v3: Hunt - Picked up at SANS DFIR Summit!

Sweet! SANS DFIR 2012 summit picked up Sniper Forensics v3: Hunt!

St. Baldrick's Support

The son of fellow SpiderLabs team member, forensicator, blogger, and long time friend Grayson Lenik is raising money to help fight childhood cancer.

Grayson's son Max raises money through donations at...

http://www.stbaldricks.org/participants/mypage/503596/2012

...then shaves his head at an event sponsored by the Montana City Fire Department.

Max's goal this year is $1,000. I challenge the forensic community to jump in, support the son of one of their own (who has joined the fight against childhood cancer) and help Max crush his goal like so much crappy malware.

Please consider throwing some chips Max's way! He is a great kid and this is a great cause.

2012 Global Security Report Released

Trustwave has released the 2012 Global Security Report. I wrote the section on data breaches and the Breach Triad. It's a great report all the way around. The team at the SpiderLabs did an outstanding job again this year, and this report is a wonderful summation of a year of hard work.

I highly encourage you to download a free copy from the Trustwave website and set aside some time to read it. It will give IT professionals of all levels a solid picture of the current state of cyber security, methods and trends. I can only imagine how use that will be for those of you trying to budget for security projects in 2012.

Again, kudos to the whole team at Trustwave that put this report together! In my opinion, it's the best one yet!