Bullshit Hunting: Digital Forensics Edition
Bullshit Hunting: Digital Forensics Edition
A guest post from a seasoned forensicator.
This is a guest post from Shafik Punja, a Canadian digital forensics expert. We are certain you’ll enjoy. - Bullshit Hunting Crew
Several weeks ago, one of my brothers from another mother, Justin Seitz, reached out to me about a criminal case he’d been reviewing. Justin had questions, (and, it turned out, rightly so), about the digital forensics process/analysis/investigation performed by a law enforcement agency, somewhere in California.
Justin sent me just under five hundred pages of trial transcripts, mostly centering on data extracted by law enforcement from a mobile device, and I dutifully reviewed.
Reviewing materials like this was something I’d done countless times over my professional career as an officer with a municipal police agency in Canada, sixteen of which were on that agency’s digital forensics team. 1
Here, my scope and objective were narrow: to review and advise if the digital forensic processes described at trial adhered to, or even vaguely resembled, best practices.2
But what are digital forensics best practices?
“Physical evidence cannot be wrong, it cannot perjure itself, it cannot be wholly absent. Only human failure to find it, study and understand it, can diminish its value.3” Digital evidence—despite how easily it can be copied, transferred and changed—is physical evidence, and must be carefully collected, preserved, studied, and understood, to be of use in an investigation or in a courtroom.
Accordingly, several organizations, agencies, and regulatory bodies have developed best practices and standards for digital forensics, including the National Institute of Standards and Technology (NIST), Scientific Working Group on Digital Evidence (SWGDE), SANS, and international bodies such as NATO, UN, Interpol. A list of resources has been provided at the end of this post.
There are key tenets that should drive the practical application of digital forensic techniques, as explained in Harlan Carvey’s June 24, 2023 DFIR Core Principles, blog article summarized below:
Everything is the result of some action.
When two systems interact, changes are made to both (Locard’s Exchange Principle).
Direct actions on a system result in direct artifacts as well as collateral or indirect artifacts.
Direct and indirect artifacts is not new, it was identified, by Harlan Carvey from August 10, 2010 blog article Artifacts: Direct, indirect where he states:
“Direct - Artifacts created by the compromise/infection/installation process, such as files/Registry keys being created, deleted, or modified.”
“Indirect - Ancillary (secondary, tertiary) artifacts created by the operating system as a result of the installation/infection process running in the environment.”
Building on the concepts of artifacts, ‘Artifact Constellations (or Clusters)’ were described in Harlan Carvey’s blog post (July 07, 2020) On Artifact Constellation and Toolmarks. This refers to the events that occur within close temporal proximity. Note, the plural of events and not the singular reference to a single event. The constellation of events occurs “as a result of some action, taken either by the user or threat actor. When someone interacts with the operating system and applications, there are direct artifacts as a result of that interaction. There are also very often indirect artifacts, created as a result of the events occurring within the "eco-system"; that is, events generated by the operating system but are not a result of direct interaction by the user.”
When I examine the concepts articulated by Harlan Carvey, above, and relate it to threat hunting, I repeatedly come back to this statement from ‘The Elastic Guide to Threat Hunting Murphy, B., French, David, 2020, Page 14’: “Two or more forms of evidence are needed to prove a hypothesis with a high degree of confidence. A single source of evidence or visibility can turn out (usually at the most inconvenient time possible) to be misleading or insufficient.”
The above statement also subscribes to the view of multiple forms of evidence to provide a sufficient and credible understanding of the data. And it ties into the following comment from a Bullshit Hunting blog post by Some Lawyer: “A single record appearing to match exactly the data sought isn’t a solution.” This is even more the case where it appears—as it does here—that the testifying investigator cannot describe where he or she obtained that record, what process created it, or why the particular tool or process was employed.
Another key concept in digital forensics is the ‘Order of Volatility’ described in RFC3277 (https://www.ietf.org/rfc/rfc3227.txt). This is the order in which data is collected from electronic devices, starting with most volatile data first, if that data still exists at time of collection. The order of collection in a typical system consists, starting first with the most ephemeral data and working toward the stable forms, prescribed as follows: Memory, System Information, Network Data, Processes & Drivers. An evolution to RFC3277, applied to mobile devices, is the information presentation provided by Mattia Epifani ‘Order of Volatility in Modern Smartphone Forensics’.
But more than the above, what is crucial is the investigative mindset, described by Brett Shavers. DFIR practice is about the investigative mindset, and not necessarily a skillset with tools. It is the investigative mindset that speaks to the practical ability to solve technological investigation challenges using the tools at hand, within a best practices approach framework.
What were my concerns?
By the end of my review, I had concerns about the digital forensics process carried out by the law enforcement investigator. The digital evidence presented centered around a mobile device and the data extracted from it by use of a tool known as Cellebrite.
The detective who testified hadn’t performed the extraction, but had attended an “eight-hour law enforcement class which dealt with Facebook and cell phones. Stuff like that,” but admitted they’d “never done it personally in a murder investigation.”
The detective’s relative unfamiliarity with the subject matter was made plain, in testimony describing what Cellebrite does, explaining that “there's a device that is a -- Cell Bright [sic] – that we can plug into the phone, and it downloads any memory or media on that phone,” and represented that the process had been used, in this instance to generate a 574-page report, including photographs and text messages.
Describing what Cellebrite is or does (or any digital forensic tools for that matter), should ideally be explained by a competent, trained person in digital forensics, not by a person who observed it being used. At the very least a basic understanding of what mobile forensic tools are doing, depending upon the type of extraction being undertaken, is important.
The phrase “any memory” was a red flag. What could this detective mean by “any memory?” Did they mean “all” memory4? What could the detective possibly mean by that representation? Is the detective aware of the many different types of extraction that can be performed on a device and the difference between them?
The use of “memory” is a common term of reference by lay persons to describe their understanding of the data stored on a phone. A non-professional will refer to this data as “memory” when in fact what they mean is the data storage. A digital forensic practitioner will provide a more precise definition of what is being discussed; storage or RAM. It is essentially a difference in terminology resulting from an imperfect understanding of the technology.
Extractions (of the “memory”) can range from a simple logical extraction (which only includes the files that the operating system is willing to give) through to a Full File System (FFS) extraction (which includes all/almost all live files in the file system) to a Physical Extraction (which obtains all bytes of the addressable memory space) but even then, there would potentially be even more data accessible with a low level extraction of the memory chip and controller. These differences can play a vital role in what data you are able to see.
The examples cited below demonstrate the various types of data extraction scenarios. They are intended to provide a glimpse into the diversity of technological challenges that are encountered in digital forensics.
This is easily observed in hard drives (electromagnetic, SSD) where the actual size is larger than what the operating system or user can access, versus what is reported by the vendor. Access to the memory space in flash memory chips is governed by the controller chip or system on chip. But if the flash memory chip (once removed from its parent device) is read directly through a chip reader, there is potential to access the entire memory storage space. The same is true of hard drives that are read using specialized data recovery tools like the PC-3000. There is a variant of this product that also reads flash memory.
The process of imaging or data extraction from a device (computer, mobile device, tablet, laptop, hard drive, memory card/stick, USB stick/drive) can be performed by digital forensic free open source software (FOSS) and specialized commercial (hardware and/or software) tools. Depending upon the type of device, either hardware or software write blocking tools will be used to prevent the data from being changed. This is referred to as read only. However, mobile devices cannot be truly write-blocked like hard drives. Acquiring RAM (volatile memory) from computers and laptops is performed when the device is on using specialized software tools. Thus, this too, is not write-blocked, but rather completed when the computing device is running in a live state.
What about computers like the current generation of Apple laptops and computers, Microsoft Surface tablets, Google Chromebooks etc., where the flash memory storage is soldered onto the motherboard, and cannot be removed like a traditional drive? Let’s add some complexity and indicate the aforementioned type of devices have their own vendor provided encryption enabled. The password/passcode/passphrase or recovery key is required, to access user data in an unencrypted format and obtain a logical acquisition of the user volume/partition. This is not the entire memory and in fact does not account for the volatile RAM.
The detective testified, further:
“Once the computer downloads it, it puts it into a file. I have no idea what it pulls off first. I get a complete file once it's downloaded, and then I can access, the file, and I can access photos. You can access emails, address book. Anything that's on that phone. Even some deleted messages that are still in the memory of the phone will be pulled out by that program.”
What type of extraction was performed on the device in question? This is not clarified in the testimony I reviewed. Is it a logical extraction or Full File System (FFS) extraction? Was there a physical SIM in the device at time of extraction or was it removed, either when it was seized or prior to device data extraction? If applicable to the device, was there a memory card present? If so, was it removed and processed in a write protected manner. What type of file was created by Cellebrite, that the testifying Detective used to generate the 574-page report?
After the “download” is completed, the detective testified that data was downloaded from a computer that Cellebrite was connected to. Prior to extraction, was the device radio isolated from any and all wireless (Bluetooth, Wi-Fi, Cellular, NFC) connections? How was the extracted data analyzed and reviewed?
Was the review/analysis of the extracted data completed using Cellebrite Physical Analyzer or Cellebrite Reader? Or another tool entirely? Why does this matter? Both Cellebrite Physical Analyzer and Cellebrite Reader are visually similar tools designed for the analysis of data extracted from a cell phone. However, only Physical Analyzer will open the data extraction from the device. Cellebrite Reader is a tool designed to show the data previously decoded and therefore the amount of data within the Reader report is contingent upon what the original examiner chooses to include. Parsed records and files that were omitted from the Reader report will be unknown to the Reader user.
Manual verification is used to compare the data displayed by the device against the data that is extracted and parsed by the mobile device analysis tools, used by the digital forensic analyst. This is a reasonable and necessary procedure to ensure accuracy in the computational interpretation of the data. To illustrate, some examples include, but are not limited to, the accuracy of date and times, number of records (non-deleted), and correct identification of the state of the records (incoming, outgoing, draft, missed, etc.).
What does the trial transcript reveal about the manual verification?
First, the cited paragraph below in the testimony of the Detective, with a clear Boolean response of ‘Yes’, that he/she compared certain text messages and certain images.
“And prior to your testimony today, did you have an opportunity to review the cell phone and compare it with certain text messages and certain images to make sure that everything was the same as what was downloaded on that cell bright report?
Yes.”
The next two cited paragraphs, from other areas in the trial transcript, are stated by the court, affirming that the police officers went through the cell phone line by line to make sure what was downloaded (extracted) from the cell phone matches the content on the cell phone.
“[…] that downloaded everything from the cell phone, and then they independently looked at the download then you went through the cell phone line by line to make sure that what was in the cell phone was an actual derivative from that cell phone based on what was printed out […]
[…] they independently went back and looked at what came from cell bright [sic], looked at the phone, compared the two. They're the same. That satisfies me in terms of any foundational gaps or problems with that.”
In my professional experience, comparing the data from a cell phone extraction line by line, is unrealistic and not practically feasible. First, you cannot obtain everything from the cell phone, even with a full file system extraction, which only obtains the allocated space. Second, you cannot compare any recovered deleted data against the device; this requires manual analysis to ensure accuracy of parsing. Third, certain types of data such as call history records are only viewable to a specific point in time on the device. In other words, in the extracted data, there can exist more call history records, than what is displayed by the device. Ultimately, forensic tools are there to make access, analysis and reporting of common data sources faster and easier. But they must be operated by a knowledgeable user to ensure the data is being interpreted correctly and this cannot be fully accomplished by comparing a few records.
A lack of understanding the capabilities of mobile extraction tool(s) (in this case Cellebrite), that is used, is not ideal, when trying to explain to the court what a specific tool or technique does, when you did not perform these actions.
For those intrepid readers that want further insight into the differences between the types of extraction I refer you to this article from Cellebrite, on ‘DFIR Extractions Explained’: https://cellebrite.com/en/episode-23-i-beg-to-dfir-data-extractions-explained-ffs-afu-bfu-advanced-logical-digital-forensics-webinar/
And another article on the crucial role of full file systems by Heather Mahalik (now Heather Mahalik Barnhart) and Paul Lorrentz: https://www.forensicmag.com/3425-Featured-Article-List/609993-Closing-Cases-The-Crucial-Role-of-Full-File-Systems-in-Law-Enforcement-Investigations/
Digital forensics best practices provide a framework that digital forensics practitioners follow in their tradecraft. However, a lack of understanding about this framework, tools and processes is demonstrated by the testimony provided. What has been shown from the narrative described in the above review provides the perception that the court does not clearly understand that the “memory downloaded” is reliant upon the type of extraction performed on the cell phone device. The resulting manual verification process to ensure accuracy, is confined by practical limitations, as explained in the previous paragraphs.
I posit the court’s (and counsel’s) understanding of mobile device extraction and manual verification processes, has been impelled by the Detective's testimony. In other words, the trier of fact may not be fully appreciative of the digital evidence that is being tendered to the court, if a witness has limited knowledge and training in digital forensic practices and tools.
Therefore, it is critical and incumbent upon law enforcement agencies to ensure their members that are involved in any manner of digital forensics, be trained to an appropriate standard of practical knowledge, so they can act as proficient witnesses when required to testify. Equally, prosecutors, too, must be aware of the level of knowledge, skill and limitations of the police witnesses they are calling to testify about digital evidence.
Acknowledgement:
Thank you to Justin Seitz and the Bullshit Hunting team for their review and editing of the original document.
A sincere, deeply appreciative thank you to my fellow former teammates and dear friends, for their honest, technical, peer review: retired Constable (now current active duty Civilian Member) Doug Kraan (Calgary Police Service Cyber Forensics Unit, Digital Forensics Team), Inspector Jeremy Wittman (former member of Calgary Police Service Cyber Forensics Unit, Digital Forensics Team), and Ian Whiffin (former member of Calgary Police Service Cyber Forensics Unit, Digital Forensics Team, now Product Manager Decoding, Cellebrite).
Shafik Punja, February 6, 2024, Canada
Resources
NIST (National Institute of Standards and Technology)
Guide to Integrating Forensic Techniques into Incident Response
https://nvlpubs.nist.gov/nistpubs/legacy/sp/nistspecialpublication800-86.pdf
Guidelines on Mobile Device Forensics - National Institute of Standards and Technology (NIST): http://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-101r1.pdf
SWGDE (Scientific Working Group on Digital Evidence)
https://www.swgde.org/documents/published-by-committee/forensics
ACPO (Association of Chief Police Officers)
ACPO Good Practice Guide for Computer-Based Electronic Evidence, (pages 45-51):
https://athenaforensics.co.uk/acpo-guidelines-for-computer-forensics/
ACPO Principle 1: That no action take is taken that should change data held on a digital device including a computer or mobile phone that may subsequently be relied upon as evidence in court.
ACPO Principle 2: Where a person finds it necessary to access original data held on a digital device that the person must be competent to do so and able to explain their actions and the implications of those actions on the digital evidence to a Court.
ACPO Principle 3: That a trail or record of all actions taken that have been applied to the digital evidence should be created and preserved. An independent third-party forensic expert should be able to examine those processes and reach the same conclusion.
ACPO Principle 4: That the individual in charge of the investigation has overall responsibility to ensure that these principles are followed.
Interpol/NATO/UN
Interpol Guidelines_to_Digital_Forensics_First_Responders
https://www.interpol.int/content/download/16243/file/Guidelines_to_Digital_Forensics_First_Responders_V7.pdf
INTERPOL Global guidelines for digital forensics laboratories
https://www.interpol.int/content/download/13501/file/INTERPOL_DFL_GlobalGuidelinesDigitalForensics
United Nations - First Responder Guide on Digital Devices
https://www.un.org/counterterrorism/sites/www.un.org.counterterrorism/files/guide-first_responders-digital_devices_in_battlefield.pdf
Council of Europe (COE) - Digital forensics lab processes and procedures
https://rm.coe.int/glacy-dfl-guide-version-aug-2017-v8/168
DFIR Mobile Devices
https://cellebrite.com/en/episode-23-i-beg-to-dfir-data-extractions-explained-ffs-afu-bfu-advanced-logical-digital-forensics-webinar/
https://cellebrite.com/en/new-sans-institute-report-codifies-best-practice-in-mobile-forensics/ (6 Steps to successful mobile validation)
https://cellebrite.com/en/part-1-fundamentals-matter-how-to-maximize-your-digital-collection-efforts-with-cellebrite-ufed/
https://cellebrite.com/en/part-2-fundamentals-matter-understanding-the-available-data-collection-methods-offerings/
https://blog.elcomsoft.com/2022/11/approaching-ios-extractions-choosing-the-right-acquisition-method/
https://blog.digital-forensics.it/2023/09/ios-15-image-forensics-analysis-and.html > iOS 15 Image Forensics Analysis and Tools Comparison - Processing details and general device information
https://blog.digital-forensics.it/2023/11/ios-15-image-forensics-analysis-and.html > iOS 15 Image Forensics Analysis and Tools Comparison - Communication and Social Networking Apps
https://blog.digital-forensics.it/2023/11/ios-15-image-forensics-analysis-and_23.html > iOS 15 Image Forensics Analysis and Tools Comparison - Browsers, Mail Clients, and Productivity apps
https://www.doubleblak.com/blog.php
https://github.com/RealityNet/Android-Forensics-References
https://github.com/RealityNet/iOS-Forensics-References
Caveat: My frame of reference is the Canadian legal system, specifically Criminal Law. I am not a lawyer, nor a legal expert. The knowledge I have acquired about the practical implementation/enforcement of criminal law arises from my 25-year career as a police officer. None of the preceding or foregoing is legal advice and should not be construed as such, and further do not represent strategies, views, or opinions of any employers: past, present or future.
This essay is intended as a sequiturial inspection of reviewed testimony from the perspective of an experienced digital forensic practitioner. It is not intended as a criticism of the Detective’s actions or that of the law enforcement agency he/she represents.
Kirk, Paul L. Crime investigation: physical evidence and the police laboratory. New York: Interscience Publishers, Inc., 1953.
Note from Somebody Else’s Lawyer: As a very non-technical person, perhaps landing somewhere between the average lawyer (cannot use the printer on my own) and the average juror (god only knows), I certainly understood the detective’s representation to mean that the device sucked all the information out of a phone, like some kind of vacuum. I now understand that was silly.