RED Camera Video Format Extraction Utility

PROJECT INFORMATION

Project Title: RED Camera Video Format Extraction Utility

Project Start Date: 2016/08

Project Completion Date or Expected Completion Date: 2016/10

Field of Science or Technology: Software Engineering and Technology

Purpose of work

To achieve technological advancement for the purpose of creating new or improving existing materials, devices, products or processes.

Experimental Development:

Overcoming Uncertainties, Work Performed, Achievements, and Outcome

Background:

The RED video format is a proprietary RAW format that is used in high quality, professional video production. A client submitted a RAID array that hosted the storage for their video production files. The files had apparently disappeared and at some point, the file system was reformatted and partially overwritten. Metadata and other structures were overwritten therefore the only solution was a raw file recovery.

The format is undocumented therefore we would have to examine and reverse engineer the RED format by utilizing video files provided by the client, using the same equipment. Due to the proprietary nature and hence lack of technical support, we would have to break down the video header and other structures without any assistance or technical documentation.

Goal:

We had the client supply us with a number of RED video files that were produced using the same camera as that of the files to be recovered.

After deciphering the higher-level file structures, we wrote an initial scanning program used to log the location of all RED video file headers on the volume.

The first task was to extract all fragments utilizing the file start locations written to the scan logs. With the assistance of the client, we viewed the extracted fragments to determine which were complete and those that were truncated (due to interleaving of other file data within the file system).

With feedback from the client, we were informed that many fragments had flicker when played using their video editor and many were also much shorter in time in comparison the actual physical size of the video file.

After examining these files further using an openly available RED video viewer, we were able to determine the exact frames that were causing the flicker. In conjunction with the raw video data, we used this information in order to find the approximate location of the corruption within the file. This information then allowed us to accurately locate stream and frame headers, and then further break these structures down. One of the most important data values we were able to extract from the structures was the video frame timestamp and frame size. Using the timestamp and size, we could locate and identify following frames and determine if they were interrupted by other file data or overwrite.

We broke down the frame metadata further and found that there was unique data for each video file that was useful in segregating the hundreds of individual video files created during production. Our scan utility was enhanced to scan the entire volume containing the deleted RED file fragments and find the locations of continuous video frames utilizing the new information we discovered in the frame metadata.

After gathering the locations of all video fragments, we were able to piece together the majority of the lost files. When verifying the recovered files in the RED viewer, we found that some video files would not load and would hang the viewer during the frame indexing operation. After closer examination, we found that the problematic files had missing frames, likely due to overwrite of some frames. We experimented with good files by adding invalid data to the files as well as removing frames. We found that the viewer could tolerate extra invalid data and was able to resync the video, however a missing frame would cause severe issues with playback.

Since we needed to provide working files to our client, we had to devise a method to patch files that contained missing frames. We experimented by adding frames with a recreated frame header and then padded the data portion with zeroes. The solution now allowed the video to be played, so we went further by copying the preceding frame in to the missing frame (with a corrected frame header) and found the video played flawlessly without flicker.

We are successfully able to recover RED video format files which have been deleted or lost from a working file system. We have obtained enough knowledge regarding the internal RED format structures in order to create other enhanced or custom recovery tools depending on a client’s recovery situation.

PBX Database Conversion Program ALCATEL DEX 600

PROJECT INFORMATION

Project Title: PBX Database Conversion Program ALCATEL DEX 600 (DSC Communications Corp)

Project Start Date: 2016/01

Project Completion Date: 2016/02

Field of Science or Technology: Software Engineering and Technology

Purpose of work

To achieve technological advancement for the purpose of creating new or improving existing materials, devices, products or processes.

Experimental Development:

Overcoming Uncertainties, Work Performed, Achievements, and Outcome

We were required to convert a proprietary database format used by a DEX 600e Private Branch Exchange (PBX) telephone switch, in order to allow legacy PBX switch data to be migrated to newer switches. We had previously developed a utility to convert a DEX PBX database in 2015, but the database format that we received in this case appeared to be a different version.

Similar to our previous research, the biggest challenge involved reverse engineering of the various related data tables using specifications for the original switch design. In order to be successful, we required specific data samples output from the current system, and we were unsure whether we could obtain enough pertinent sample data to properly reverse engineer the format. As mentioned in the initial claim, any available documentation available was for the earliest (initial) releases and little information applied to the more recent databases.

We reverse engineered this different database format through analysis of the raw data alongside table outputs provided by the client. We had the assistance of an administrator familiar with the PBX system who generated some sample tables based on specifications that we requested, in order to reverse engineer the unknown data that differed from the other versions. After a number of iterations of table dumps, we were able to eventually gather enough information to determine the relationships between the various tables.

We were successfully able to reverse engineer the data table structures for this database version. We enhanced the code for our current conversion utility to include this new structure and are now able to support a wider range of phone system PBX telephone switches.

GoPro Deleted Video Recovery Utility

PROJECT INFORMATION

Project Title: GoPro Deleted Video Recovery Utility

Project Start Date: 2016/02

Project Completion Date: 2016/06

Field of Science or Technology: Software Engineering and Technology

Purpose of work

To achieve technological advancement for the purpose of creating new or improving existing materials, devices, products or processes.

Experimental Development:

Overcoming Uncertainties, Work Performed, Achievements, and Outcome

We were required to recover deleted files from a flash storage device used in a GoPro video camera. Similar to most file systems, the metadata and pointers are lost after deletion and due to large sizes of video the files are usually fragmented. Due to the proprietary nature of the video format, we had to reverse engineer the video format and then devise a method of accurately extracting the deleted video file. It was unknown whether it was possible to decipher the video stream headers and content since manufacturers were developing more sophisticated ways of encoding video for enhanced speed, features and compression. We have also found that some video formats that lack obvious stream headers (other than the initial header) – these are likely simpler video formats without the ability to resync if compression fails due to data errors.

We analysed a number of good GoPro files in order to determine whether there were obvious video stream headers or any other overhead that wrapped the various chunks of video data. After analysing the binary data of the files, we were able to identify repetitive headers within the file that had similar structures. We then attempted to break down the structure in to the various components describing the following video stream. We were eventually able to separate one of the most important values in the structures for recovery. Using these values, we were able to deduce fragment sequences by first scanning the file system for any fragments containing a stream header. We then logged this fragment’s location, the position of the first header relative to the beginning of the fragment and also the last header relative to the end. In some cases, it was difficult to determine the actual start and/or end of the fragment so at times we forced to manually determine the starting boundary. After all valid video fragments on the volume were logged, we then attempted to pair up fragments with their following fragments. We did this by first taking a fragment containing the principle header signifying the first fragment of the video file. Then by using the distance between the last header to the end of the fragment and knowing the stream size, we could calculate the size of the remaining part of the stream. We then searched the scan log for a fragment that contained an entry equal to this value. If this value did not occur more than once in the log, it was very likely that it was a match. We iterated through each following fragment using the same procedure until the end of file was reached. This was continued with all other file fragments containing principle headers until all possible files were reconstructed.

We successfully recovered the deleted files and were able to add this format to our list of recoverable deleted video files.

exFAT File Undeletion Utility

PROJECT INFORMATION

Project Title: exFAT File Undeletion Utility

Project Start Date: 2016/11

Project Completion Date: 2016/11

Field of Science or Technology: Software Engineering and Technology

Purpose of work

To achieve technological advancement for the purpose of creating new or improving existing materials, devices, products or processes.

Experimental Development:

Overcoming Uncertainties, Work Performed, Achievements, and Outcome

Part of our data recovery process involves imaging a storage device if it is unstable or contains potential data errors. exFAT is a file system format that is increasingly popular with digital cameras and video devices. The block size (and hence the cluster size) is usually larger than a standard 512 byte per sector seen with standard Windows storage device format in order to support large file system sizes. When the device is imaged to another storage medium (hard disk or image file), the non-standard block size is not recognized by Windows and hence the recovery utilities that we reviewed on the market were not successful in extracting large deleted files from this type of file system. We were required to develop a utility that would recognize the different block sizes and extract deleted video files from the client’s device image using the different parameters of exFAT.

In attempting to develop this utility, we were unsure with the information we could find at the time, whether it was possible to fully recover a deleted file from a file system that was in use for a period of time. We found through information that some pointers and metadata were retained after deletion but it was uncertain how much overwrite that the file system would tolerate before utilizing the deleted space.

We studied the manner in which the file system modified the FAT and metadata when a file was deleted. We found after some study that file metadata (file name, dates, file size, etc) is still retained for a period of time and is only marked as available in the bitmap, but will eventually get overwritten if the directory space is needed. The FAT chain is also retained for a time after deletion but also eventually gets overwritten when the disk storage space is needed. From our observations, the FAT chain was retained longer than file metadata.

Using file system data from a client recovery that contained deleted files, we found that the metadata for requested files for recovery was missing. We then requested video file sample from the video camera that produced the files and performed a scan. We created a utility that scanned for the video headers and recorded the locations. After matching the video headers with valid files that weren’t deleted, we deduced that the remaining file headers that were part of video files that were previously deleted. We matched these file start locations to the related location in the FAT, and if that location showed that it was the beginning of a chain, it was extracted assuming that the FAT was valid. The files were then tested since there was no way to guarantee the file integrity. The end result was a successful tool that is being used in the daily operation.

Why us?

Case No: 433144 |

“Previously sent job to DriveSavers and the recovered DB was unusable, but then I sent the drives to WeRecoverData.com the recovery was successful. We got our data back and it saved us two years of rebuilding information.

Thanks WeRecoverData.com!!”

Geoff Windsor, Geoff,

Tulsa, Oklahoma

Drives:

Model: WD5003ABYX-18WERA0 S.N.: WMAYP0F3CDA9
Model: ST3500320NS S.N.: 9QMBTY24
Model: ST3500320NS S.N.: 9QMBTRXP

Case No: 431209 |

“We had an IBM RAID 6 array that may have been expanding from four drives to five. Something went wrong and there was at least one incorrect rebuild done after that. Kroll Ontrack performed a recovery attempt without success. We didn’t give up and we found WeRecoverData which claimed they recovered successfully many of these RAIDs. We were very hopeful and they finally did it. They recovered it all successfully! They were able to recover all of our files, excellent job in recovering our data… Very fast service, reasonably priced and excellent customer service… very responsive… Would definitely recommend…”

Nerissa Beekharry,

Controller, Hi-Tech Metals, Maspeth, NY

Drives:

Model: MAT3147NC S.N.: AAF9P5401JDN
Model: MAW3147NC S.N.: DAF9P63010M6
Model: ST3146807LC S.N.: 3HY88LPS
Model: ST3146807LC S.N.: 3HY88KW0
Model: ST3146807LC S.N.: 3HY873DY
Model: ST3146807LC S.N.: 3HY85YLE

Case No: 352225 |

“I really appreciate the folks at WeRecoverData.com. After my valuable business data and family photos from the last 5 years being stuck in black hole known as a crashed hard drive, WeRecoverData.com came to the rescue I was recommended to another major recovery service (Drive Savers Data Recovery) by several computer consultants when my drive first failed. After three – four weeks of working with my drive, the competition returned my drive to me unrecovered. I thought my data was gone for good. When my wife suggested we try another company, we went werecoverdata.com. Within 3 – 4 days of shipping my data out, I got the e-mail that that had recovered 100% of my data. Thank you so much to Pedro and the team at WeRecoverData.com You guys are a life saver.”

Reed L. “Skip” Pond, Chattanooga, TN

(Drive Model #: 9NU2A3-500 | Serial #: 2HAP1Z1C)

Case No: 354553 |

“A trusty WD hard drive, which was the only place we stored our comprehensive digital photos, some key pdfs, and thousands of mp3s was accidentally dropped from a couch. When we tried to turn it on, it started making a clicking noise – we knew we were in trouble. Upon consulting my techie friends, one of whom tried to run a data recovery software on the drive, with no success – I was told to seek help from professionals. I tried a local computer repair shop, and they said problems were significant enough that I needed to work with a firm specializing in data recovery. I started shopping around, and sent the drive to werecoverdata.com, based on a friend`s recommendation. They came back with a quote of $ 1800 which was more than I was prepared to pay. I tried another option – a low cost company in Canada – called Accurate Data Recovery. They claimed that their fee will not exceed a thousand dollars, and they had every capability of large companies like Ontrack and DiskDoctors.com. They spent about a week and a half trying to recover the data, only to inform me that the physical damage was too great and they could not do anything for me.

The big boy Ontrack was next – believe it or not, they spent a month, trying various techniques that they referred to as “proprietary”, “state of the art” and even “innovative and experimental”. No success there too. I finally came back to Werecoverdata.com, still hoping against hope. To my great delight, after about 3 weeks they were able to get data of the drive, recovering 95% of the files!!! Throughout the process they were diligent and very professional with updates, and transparent around their process. I am not sure what magic they used, but I am extremely grateful to their skillful people, dedication to results and the customer friendly way they communicated with me. I would always recommend them to friends and colleagues, and will use them if, God forbid, I ever need to again, without any hesitation.
Thanks, Werecoverdata.com!”

Dimitry Gitarts, Philadelphia, PA

(Drive Model #: WD2500JB-00GVC0 | Serial #: WCAL73936006)

Case No: 425209 |

Kevin O’Leary (from Shark Tank) Testimonial.

(Seagate 1TEAPF-500 Drive, Serial #: NA8JEXQ7)

Case No: 347245 |

“When my house burned down, I lost practically everything I owned. I remember after the fire I was digging through the rubble when I came across my external hard-drive. It was filled with rust-colored water and the plastic was baked onto the electronics. I tossed it back into the ashes of the house thinking it was ruined and that all my data was lost. It wasn’t until a friend reminded me that it was free to have a hardware recovery evaluation done that I figured that I might as well try When WeRecoverData.com told me later that week that all the files could be recovered successfully, I was in disbelief. Not only was the recovery successful, but they laid out every file exactly as they were before the fire. After this experience, I can confidently recommend for everyone to bring their damaged hardware to WeRecoverData.com no matter how melted, soaked or cracked it may be. They truly are experts at what they do.”

Brian Webber, Chicago, IL