Pulling Data from DJI .dat log files to Matlab...

After a friend of mine that I've been working for experienced a software failure in his DJI branded octocopter (which is worth more than my car btw), DJI customer support had been giving him the grief by refusing to analyze his log data because the fault of the crash was likely their own. Since I've always had good experience exporting my APM log files for analysis in MatLab, I offered to take a look at his log data to pinpoint the cause of the crash. When I received the log file, I was stunned to find that DJI chooses to use .dat files to store their log data. Dat files are notoriously difficult to read without having knowledge of how the data in the file is arranged or the process in which the data was stored. Since DJI is not an open source platform, it was going to be nearly impossible to decipher the contents of the .dat file. After doing some research, it ends up DJI doesn't want its customers analyzing the log files beyond the use of the "DJI Data Viewer" that can be downloaded online. The "DJI Data Viewer" is rather worthless on its own, because it only allows you to view graphs of the log data, and not raw data itself. So frustrated and annoyed, I came up with a way to pull the data from the graphs themselves: 


Below is a screenshot of the DJI Data Viewer. After choosing the string that needed to be pulled, I changed the line viewing option to "dots" because makes it easier on the program to differentiate between multiple strings with minimal discontinuities (I only coded for 3 strings at a time).
3691147147?profile=original

Then, I took a screenshot of the graph at the highest resolution possible to maximize the spacing between the pixels. I applied a RGB filter in Photoshop to remove the grid-lines....
3691147476?profile=original
Lastly, I saved the completed graph as a bitmap (.bpm) to accurately preserve each individual dot in the picture (the picture below has much lower resolution than the original that was used in the script. I shrunk it for viewing convenience. If you want to try running the script, use the full resolution bitmap that I attached).
3691147505?profile=original
I then wrote a Matlab script that would take this final bitmap, pick out 3 different colored strings individually and assign coordinates to their respective pixels. The strings are then exported to .xls (Excel) files for later analysis! Here is a graph of the data obtained through the Matlab script and compared to the original graph in the DJI data viewer...
3691147446?profile=original

Looking closer at a single string can show just how accurate the process can be...
3691147493?profile=original
I've attached the .m file that contains the script. Its rather simple but whatever works, works. If anyone has come up with a better way to pull data from DJI logs, please let me know! This process is rather labor intensive because it requires a bit of photo manipulation, but from all the searching I've done, it seems like it hasn't been done before. 

DataPullFromDJIGraphBMP.m

dotmap_456.png

You need to be a member of diydrones to add comments!

Join diydrones

Email me when people reply –

Replies

  • Hi Striker

    Excellent!!

    One question....were you able to find out why the quad crashed?

    Regardfs

  • Great piece of work!  I have been trying to decode .DAT files for a month now and am very frustrated with DJI and their data viewer software...  This is a nice find!

    I was wondering if you have a way to automatically apply proper scaling to the data?  The y-values appear to be the pixel values.  I would like to import several channels from a lot of flights and don't want to have to do the scaling for each graph by hand...

    Decoding the .DAT files would be the ultimate way to do it, but the format is not obvious and the folks who have been able to crack these files in the past haven't yet been able to come up with a way to crack the A2 / iOSD Mk II combination.  Do you know of anyone who has been able to do this?

    Thanks,

    --Ed

    • Thanks! You are correct, the x and y values are just pixel coordinates. For the nature of the analysis I was doing, this was good enough. The values were arbitrary and I only need to do it for a single flight (The flight where it crashed)....

      To scale the plots numerically all you would need to do is know the corresponding values of 2 of the y values that were pulled form the graph. The goal is to find a ratio that can be multiplied with the y values to properly scale the data. The compass for example, we know that the the yaw readings range from pi to -pi (our 2 known values), so as long as the multirotor made at least 1 full rotation we could find the max and min values of the yaw channel (the yaw y values being yaw(:,2)) and relate them to the known values of pi and -pi. The difference between max(yaw(:,2)) and min(yaw(:,2)) can then be used to find a constant (ratio) to scale the pulled values by. Mathematically, that constant C would be: C= (pi-(-pi))/(max(yaw(:,2))-min(yaw(:,2))). Lastly, redefine the yaw y values using the constant: yaw(:,2)= yaw(:,2).*C
      This obviously wont work for every scenario, because 2 corresponding values aren't always known, so the 2 values will need to be user inputs. For this reason, and the fact that taking screen shots and manipulating photos is labor intensive, this method might not be the best for decoding tons of data strings.
      Right now, this method is more of a work-around.

      And yes, decoding the .dat file would be your best option if it were easy to do.

      The only way I can think to make this method feasible for use with many flight logs would be to automate everything...
      It would be possible to use JAVA to interact with the DJI data viewer directly and automatically select data strings and save screenshots of them. Then Matlab can take over, and import all the screenshots and do the photo manipulation automatically (Matlab is actually very competent when it comes to basic photo manipulation). That way, the whole process is automated and all the screenshots can be taken at known scale. If the scale that the screenshots were taken is known for every string, it would then be easy to automatically scale the data in Matlab. But unfortunately that's a lot of coding...  
      I wish I had more time to help because it sounds like a fun challenge. 

This reply was deleted.

Activity

DIY Drones via Twitter
Review of the new Nivida Jetson NX for robocars https://t.co/bWjd3sH2o4
11 hours ago
DIY Robocars via Twitter
RT @chr1sa: My review of the new @NVIDIAEmbedded Jetson Xavier NX in robotics and @DIYRobocars applications https://t.co/O7TfJZhiF9 https:/…
yesterday
DIY Drones via Twitter
RT @chr1sa: Somehow I missed this, but last week @Nvidia released a new version of its Isaac robot development framework. Lots of new Lidar…
yesterday
DIY Robocars via Twitter
RT @Smartphone89: DonkeyCar Racing League Korea #donkeycar @diyrobocars https://t.co/mDlHDR1d98
Friday
Jørn Ramnæs liked VIDEO LINK's profile
Friday
Mark Harrison left a comment on Moderators
"Wow, Ning2 looks really great on mobile. I had sort of drifted away as my browse-time went more on my phone, so this is great."
Friday
Hank Deucker left a comment on PIXHAWK
"I am sure most will say not again, Pixhawk telemetry. I do need help and all can be reassured that I have spent many hours scouring the net for useful information. I am not new at quadcopter flying, I started in 2012 and have progressed through many…"
Thursday
DIY Robocars via Twitter
https://t.co/rLmFr8M2UK
Thursday
Hank Deucker left a comment on PIXHAWK
"Tomorrow, I am thinking about the wording at the moment."
Thursday
DIY Drones via Twitter
Welcome to the new DIY Drones design!. You may have noticed that DIY Drones looks a little different today. That's… https://t.co/cg2EmeDGMQ
May 27
DIY Drones via Twitter
New! Online Training on Drones for Disaster Response https://t.co/brqRuujjY2
May 26
DIY Robocars via Twitter
https://t.co/8EMCvYW6oz
May 25
DIY Robocars via Twitter
RT @OttawaAVGroup: We are going to try something new this week. We will be having a drop-in style meeting Wednesday @ 7-9pm EST. Pop in and…
May 25
DIY Robocars via Twitter
RT @tawnkramer: Thanks Everyone for another fun virtual race! Amazing Ai racers competing head-to-head, Virtual Makers Fair event. Catch us…
May 23
DIY Robocars via Twitter
RT @make: Find out the latest from @diyrobocars at 10am PT from @chr1sa of @3DRobotics on Virtually Make Faire! #makerfaire #maketogether #…
May 23
DIY Robocars via Twitter
RT @Heavy02011: Join us Saturday online: Virtual Race League: @DIYRobocars Race #2 - Parking Lot Nerds https://t.co/aJUu4sdutx #Meetup via…
May 22
More…