After a friend of mine that I've been working for experienced a software failure in his DJI branded octocopter (which is worth more than my car btw), DJI customer support had been giving him the grief by refusing to analyze his log data because the fault of the crash was likely their own. Since I've always had good experience exporting my APM log files for analysis in MatLab, I offered to take a look at his log data to pinpoint the cause of the crash. When I received the log file, I was stunned to find that DJI chooses to use .dat files to store their log data. Dat files are notoriously difficult to read without having knowledge of how the data in the file is arranged or the process in which the data was stored. Since DJI is not an open source platform, it was going to be nearly impossible to decipher the contents of the .dat file. After doing some research, it ends up DJI doesn't want its customers analyzing the log files beyond the use of the "DJI Data Viewer" that can be downloaded online. The "DJI Data Viewer" is rather worthless on its own, because it only allows you to view graphs of the log data, and not raw data itself. So frustrated and annoyed, I came up with a way to pull the data from the graphs themselves:
Below is a screenshot of the DJI Data Viewer. After choosing the string that needed to be pulled, I changed the line viewing option to "dots" because makes it easier on the program to differentiate between multiple strings with minimal discontinuities (I only coded for 3 strings at a time).
Then, I took a screenshot of the graph at the highest resolution possible to maximize the spacing between the pixels. I applied a RGB filter in Photoshop to remove the grid-lines....
Lastly, I saved the completed graph as a bitmap (.bpm) to accurately preserve each individual dot in the picture (the picture below has much lower resolution than the original that was used in the script. I shrunk it for viewing convenience. If you want to try running the script, use the full resolution bitmap that I attached).
I then wrote a Matlab script that would take this final bitmap, pick out 3 different colored strings individually and assign coordinates to their respective pixels. The strings are then exported to .xls (Excel) files for later analysis! Here is a graph of the data obtained through the Matlab script and compared to the original graph in the DJI data viewer...
Looking closer at a single string can show just how accurate the process can be...
I've attached the .m file that contains the script. Its rather simple but whatever works, works. If anyone has come up with a better way to pull data from DJI logs, please let me know! This process is rather labor intensive because it requires a bit of photo manipulation, but from all the searching I've done, it seems like it hasn't been done before.
You need to be a member of diydrones to add comments!
Replies
Hi Striker
Excellent!!
One question....were you able to find out why the quad crashed?
Regardfs
Great piece of work! I have been trying to decode .DAT files for a month now and am very frustrated with DJI and their data viewer software... This is a nice find!
I was wondering if you have a way to automatically apply proper scaling to the data? The y-values appear to be the pixel values. I would like to import several channels from a lot of flights and don't want to have to do the scaling for each graph by hand...
Decoding the .DAT files would be the ultimate way to do it, but the format is not obvious and the folks who have been able to crack these files in the past haven't yet been able to come up with a way to crack the A2 / iOSD Mk II combination. Do you know of anyone who has been able to do this?
Thanks,
--Ed
Thanks! You are correct, the x and y values are just pixel coordinates. For the nature of the analysis I was doing, this was good enough. The values were arbitrary and I only need to do it for a single flight (The flight where it crashed)....
To scale the plots numerically all you would need to do is know the corresponding values of 2 of the y values that were pulled form the graph. The goal is to find a ratio that can be multiplied with the y values to properly scale the data. The compass for example, we know that the the yaw readings range from pi to -pi (our 2 known values), so as long as the multirotor made at least 1 full rotation we could find the max and min values of the yaw channel (the yaw y values being yaw(:,2)) and relate them to the known values of pi and -pi. The difference between max(yaw(:,2)) and min(yaw(:,2)) can then be used to find a constant (ratio) to scale the pulled values by. Mathematically, that constant C would be: C= (pi-(-pi))/(max(yaw(:,2))-min(yaw(:,2))). Lastly, redefine the yaw y values using the constant: yaw(:,2)= yaw(:,2).*C
This obviously wont work for every scenario, because 2 corresponding values aren't always known, so the 2 values will need to be user inputs. For this reason, and the fact that taking screen shots and manipulating photos is labor intensive, this method might not be the best for decoding tons of data strings.
Right now, this method is more of a work-around.
And yes, decoding the .dat file would be your best option if it were easy to do.
The only way I can think to make this method feasible for use with many flight logs would be to automate everything...
It would be possible to use JAVA to interact with the DJI data viewer directly and automatically select data strings and save screenshots of them. Then Matlab can take over, and import all the screenshots and do the photo manipulation automatically (Matlab is actually very competent when it comes to basic photo manipulation). That way, the whole process is automated and all the screenshots can be taken at known scale. If the scale that the screenshots were taken is known for every string, it would then be easy to automatically scale the data in Matlab. But unfortunately that's a lot of coding...
I wish I had more time to help because it sounds like a fun challenge.