Hi,
since I use Copter 3.3 Firmware i have a strange behavior that i like to share here. Maybe the one or the other has the same effect or knows how to fix it.
Description: If I try to arm my Pixhawk I get different prearming check failures (EKF..., GPS Horizon...., Gyros not calibrated etc.). Some of them are not documented by now.
After some reboots the arming works and the copter flies without any visible issues! Also the logs are fine.
First i thought that this is due to the hardware or may the cold temperatures outside in Jan/Feb. But meanwhile i see this behavior also on a second Pixhawk from a different vendor and (even in Germany) the temperature improves.
Is there someone who saw similar effects or may has a solution for this? I would not like to disable the checks because they were helpful in some cases.
Thanks Thomas
Replies
I've nailed the problem down to these issues:
Fault of APM:
* ArduCopter does not honour the manual serial port settings. It is somehow hard-coded to always use 38K baud even in manual mode/auto config disabled. When configured as a NEMA compass (NEMA output only) you can trick it and it will not try to change the baud. That must be a bug and should be possible to locate in the source/fix. For now, do not try to program the GPS with anything else then.
* ArduCopter resets the power management mode to enabled/Conservative. So it will never be possible to work with SBAS as the receiver is only switched on briefly just long enough to get a basic GPS packet. SBAS takes up to 2 minutes to get the first lock and then requires regular updates. Power management must (allowed to be) disabled ("Full Power" mode) when SBAS is desired. Again something is hard-coded there, even if you set the APM parameter to enable SBAS. A logic fault must be in the software here then.
* ArduCopter wants no less than 5Hz update frequency, which would never be possible with an M8N and SBAS. I can't understand why, that should be our choice. Trying to set different update speeds causes error messages and then I suppose the EKF disregards the GPS altogether? Must be checked/fixed in the source, it just doesn't make sense and blocks a very useful hardware feature.
Fault of user:
* We should really RTFM: https://www.u-blox.com/sites/default/files/products/documents/u-blo... but it would be nice if the ArduCopter automatic settings could be updated in integrate the new features/limitations.
* u-Blox documents several limitations based on chosen satellite signals and whether SBAS will be used or not. If you try to configure any of those invalid combinations, then that will cause a long delay, such as:
* Although SBAS is supported in combination with GPA, Galileo or Glonass, it will only be allowed to work with GPS (I guess it's some legal requirement). Exactly that is written in their manual!
* SBAS and/or Galileo requires NEMA version 4.1 (default is 2.x). It simply is not part of the older version 2.x standard according to the documentation.
* Some of the SBAS signals (known as PRN codes) are already discontinued since the u-Blox firmware 3.01 update. Checking Wikipedia tells you the codes for the valid satellites. Considering we are only allowed to use SBAS with USA/GPS signals, the WAAS source must be selected or manual PRN codes 133, 135 and 138 (see http://gpsinformation.net/exe/waas.html).
* Frequency cannot be so high when processing SBAS. For example M8N performs 5-10Hz with basic GPS, but with SBAS some sources say it must run at less than 4Hz else the processor is not quick enough to process the messages. Also various people suggest a high serial baud rate than 38k is really needed.
* Not all signals can be received in parallel. So if you use u-Centre make sure you enable a valid combination, e.g. GPS+Glonass, GPS+SBAS.
* u-Blox defaults to accept signals from satellites as low as 5 degrees. Setting 10 or 15 degrees will filter out confusion / potentially shorten pre-arm delays.
With this configuration I get an instant GPS lock on successive arms. First arm of the day up to 2-3 minutes but that is acceptable. After cold start/not using for weeks, more up to 15 minutes but I think that is acceptable.
The most annoying part is not being able to use SBAS (lower update rate and full power) or super fast updates (high update rate and high serial speed). But so long as there is no pre-arm delay, I'm currently happy with ArduCopter dev 3.4-rc5.
1) When talking about GPS settings especially uBlox rate more often means the MCU speed/refresh rate. When SBAS is enabled the max is 4Hz, without it's 5-10Hz depending on the exact model. This is documented in the configuration whitepaper from uBlox
2) NAV is never pedestrian, it's Airborne4G and that is also the ArduCotper default! Makes sense because pedestrians don't jump up in the air 100s or meters. :-)
3) Protocol out must be UBLOX + NEMA to get satellite information for diagnostics, and probably for DGPS but somebody who programs the GPS would have to confirm that. If you only allow UBLOX out you'll get empty boxes when configuring GPS pass-though and trying to look at what's wrong, also when connecting directly with uCenter.
4) I'd like to know why ArduCopter needs updates 5 times a second and won't be happy with 4Hz or less, then we could get DGPS working (SBAS)! I'd prefer a highly accurate 3 or 4Hz update compared to an inaccurate 5Hz.
5) I think ArduCopter is doing some nasty serial port logic, because it only works if you allow it to auto configure the GPS. and then it wants to set the baud rate down to 38400 all the time which is too slow for SBAS or >5Hz updates. For example, you can have perfect settings connected with a PC directly to the GPS, then connect ArduCopter and it's like half the data doesn't come through even when the serial speed is set correctly. Such behaviour suggests the serial speeds don't match, and error messages like "connected at 115200baud" then "connected at 38400 baud" then again "115200 baud" confirm that. Maybe it's messing around with rate/auto-bauding when it should not be. I disabled auto configure and set the baud, which should mean "just connect at the speed I set".
The current 3.3.4-rc5 beta is better but still has a massively annoying 10-15 minute first start delay, even with a cache from the previous day. Following arms are only delayed 30 seconds to one minute, so it is getting better but I still see warning signs here that the GPS startup-sequence and auto-configure needs work.
Derek said:
Rate must match what the pixhawk settings are, I've found 38400 and 57600 to work. . NAV you want pedestrian, not airborn. Airborn is for fast moving airplanes with consistent headings, copters behave more like pedestrians than airplanes. PRT(ports) should be 0+1 UBX+nmea for protcol in, and 0 - ubx for protocol out. Rate you want to set to 200ms or 5hz, because this is the update rate arducopter expects. Also your sv minimum elevation should be 10 or 15, at 5 you are seeing satellites too low on the horizon and multipath errors will skew your position.
The issues I'm having seem strangely similar. I created a new entry on the ardupilot forums: http://discuss.ardupilot.org/t/ekf-checks-not-allowing-arming/11125
@Tony Wall: any updates?
I think I fixed the problem as follows:
1) Increased the INS_GYRO_FILTER and INS_ACCEL_FILTER parameters to 48.
2) My U-Blox M8N (mini PixFalcon version) already had the latest firmware 3.01 with Galileo enabled, but not all the other satellites and options (Bedou thingy and some of the other checkboxes). All options were enabled and auto assignment of channels.
3) Serial port speeds were reduced back down to defaults 56k for telemetry and 38k for GPS. Previously they were increased to maximum 115K. I don't know if that burdens the PixFalcon processor, as there is no hardware flow control, but anyway with such problems occurring I prefer defaults and do not notice much difference in GPS and GCS performance. I guess 38K is enough for a GPS and it didn't make sense to put the telemetry at 115K anyway because the 433MHz radio is not capable of any reasonable distance at that rate.
4) Re-calibrated compass and accelerometers with the "auto-learn offsets" option enabled in Mission Planner.
5) Compass-motor calibration performed (maybe I forgot this last time) which confirms minimal interference.
Two drones were changed to this setting and tested at about the same time of day at the same location. Both drones found enough satellites for 3D fix from cold start within a few minutes the first time and almost instantaneously the following times. There was only one GPS speed error on the second drone, but that went away as soon as the 3D fix occurred and did not happen again in 2 hours of flying.
I won't change any settings, and see what happens the next time we fly, perhaps today. If it works then I'll try decreasing the filter to default (20) again to isolate whether the filter or GPS settings were the cause.
I have tracked this problem down to the M8N GPS (and probably other new more accurate GPS units) settings both in ArduCopter and the U-Center/GPS itself.
First the HDOP values seem to be mostly caused/affected by changing the various EKF filter and delay settings. Next you need to know that the M8N GPS has limits on what it can receive/process/send based on your enabled signals (NEMA configuration) and rates. Third there appears to be some inconsistency in the NAV mode which will switch back from the ArduCopter/drone default of 8 "Airborne <4G" to 0 "Portable". So if you updated your M8N firmware to the new 3.01 you will have to go back and change some of those settings.
My current/most successful testing parameters are:
GPS_AUTO_SWITCH 1
GPS_GNSS_MODE 0
GPS_HDOP_GOOD 250
GPS_INJECT_TO 127
GPS_MIN_DGPS 0
GPS_MIN_ELEV -100
GPS_NAVFILTER 8
GPS_RAW_DATA 0
GPS_SBAS_MODE 1
GPS_SBP_LOGMASK -256
GPS_TYPE 1
GPS_TYPE2 0
EKF_ACC_PNOISE 0.25
EKF_FALLBACK 0
EKF_GLITCH_ACCEL 250
EKF_GLITCH_RAD 25
EKF_GPS_TYPE 0
EKF_POS_DELAY 100
EKF_POS_GATE 75
EKF_POSNE_NOISE 0.25
EKF_VEL_DELAY 100
EKF_VEL_GATE 75
EKF_VELD_NOISE 0.25
EKF_VELNE_NOISE 0.25
Note: The GPS_HDOP_GOOD could be left at the original 140 once you see that the GPS is much better with these settings. I left it at 250 to allow me to arm and fly in areas with bad GPS reception or test in my back garden (near many buildings and trees). Even when the HDOP is higher your drone would still RTL somewhere near you so I guess that is the most flexible choice for most people who just want to fly quickly/racing drones with small batteries (not to be wasted on long GPS initialization time).
And for GPS my current experiments suggest that SBAS is the best protocol to enable and ensure maximum coverage because it not only applies integrity information on the dozens of potential (good and bad) satellites you will receive with GPS, GLONASS and the new Galileo enabled, but should enable DGPS for increased accuracy (although I only saw this once so far working with ArduCopter). My goal is to get a reliable SBAS/DGPS working with ArduCopter every time. But I need more testing for that.
So my current GPS test settings are:
1) Update the U-BLOX firmware to 3.01. To make this easier change the RATE to 9600 baud first, else it may fail to enable the bootloader or appear to fail afterwards (although you really just need to switch the baud rate in the U-Center GUI). The current Mission Planner "MAV serial pass" does not appear to pass-through all data and anyway is risky adding more software in the loop during a delicate update procedure. With a standard CP2102 adapter and self-made adapter cable (cut one spare cable and wire-up TX RX GND and 5V accordingly). I found you have to use a direct serial connection and select the "alternative USB method".
2) Reset to defaults after the firmware update on the CFG "Revert to default configuration" page.
3) Change the power mode to "Full Power" else it's in balanced potentially power saving mode (not what we want on a drone)! Go to the PMS page and select "Full Power" then "Save configuration".
4) Free-up GNSS channels for better purposes. The GNSS page is strange because you can't change more settings a the same time, some kind of channel limit will cancel-out other settings. So first disable and lower channels and "send", before enabling and adding other channels. To start with we don't want QZSS, IMES or BeiDou outside of China and Japan, so disable those and "send". Next besides the standard GPS and GLONASS we want the new "Galileo" in the EU, and globally we want SBAS but with 3 channels minimum (instead of 1, taking the standpoint that SBAS and more quality signals potentially with DGPS is better than 2 more channels of potentially bad satellites). Also on my personal preference in the EU I increased the Galileo minimum to 8 and maximum to 10 (most allowed).
5) Set SBAS configuration to match and enable integrity. Subsystem = Enabled, enable (checkbox) all three services (ranging, correction and integrity). Set PRN codes to "Auto-Scan". Number of channels (match GNSS config) = 3.
6) Navigation settings (NAV). Dynamic = 8 "Airborne < 4G", fix mode = 3 "Auto 2D/3D", minimum SV elevation = 5, DGPS timeout = 120.
7) With all this new information the default 9600 baud serial port speed is too slow. On the port settings (PRT) for Target 1 "UART1" set baud rate to 57600. Note that protocol in should be "0+1+2" (UBX+NEMA+RTCM) and protocol out 0+1 (UBX+NEMA). Note after sending this value the communication will hang, so manually go to the port drop-down and switch to 57600 to continue (if you have the packet console open you will see the messages stop then continue as you do this).
8) RATE is really important because the processor inside the GPS is limited and when you enable SBAS and more services the load is increased so maximum refresh rate decreased. It should be at 1000ms by default. According to the documentation it should work up to 3Hz (which you can't actually set via the GUI with the "measurement period", just 2.xx). I am currently testing 500ms period which is 2Hz but if I don't get DGPS will switch down to the default 1000ms 1Hz and test again. According to the specification my mini M8N which I got with the PixFalcon can sample up to 10Hz but others report glitches as soon as you go over 2Hz, hence my 500ms starting point.
9) Optional (unconfirmed): I noticed the NEMA version (on the NEMA page) was 4.0 default and supports 4.1. I set this to the latest version for my testing. Maybe that is a mistake (need to ask the APM team what they really support regarding NEMA but documentation is sparse, probably have to look at the code). Anyone know anything about that?
The most annoying thing during the testing is that the NEMA information and full UBLOX data are not passed-through mission planner. If that was done I could do realtime diagnostics on the drone running ArduCopter. So I must connect my GPS to a PC to configure and get good SBAS/DGPS, then re-connect to the drone and hope ArduCopter "magically" gets the same value. I suppose I need to learn the RAW GPS logging or something, but didn't want to waste that much time on this issue :-/
So hope that helps and I will share my full configuration files once I get something reliable (a few days/weeks of flying at different times of day/weather).
Tested in two different locations around Germany 100s of Km apart, both with flat surroundings and on the side of a wide open valley and at different times of the day (noon and evening). The same behaviour exists regardless of location, delays up to 10 minutes arming the first time, and anywhere between 3 minutes down to best luck of 30 seconds following times.
I think my new settings did help and are along the right lines, but there must be a bug in the EKF or GPS interface software because the directly attached u-Blox u-Center software gets a fast lock and also DGPS really quickly. I don't think this is a hardware fault or anything to do with serial speed (tried high, low and middle speeds now).
Too much flight time is lost with an active FPV transmitter and LED bar waiting for APM to decide it's safe to start. I'll test a workaround by disabling the GPS pre-arm check, but never take off without a 3D fix of course, as I really want the RTL and GeoFence functions as a minimum. If that's not possible I'll go back to the previous version as I'm spending more time fiddling with parameters then enjoying my FPV race quad :-/
Hi all,
unfortunately the topic was not picked up until now in the support forum. Hopefully i will find some time on the weekend to dig into this a little bit more. First step is that I will try to load some of the older firmwares back to one of the Pixhawk FCs and see what the effect is.
I will post my results here as soon as there is something to tell.
Happy flying
Thomas
I have some more information on this. I managed to isolate the problem and now the error makes sense. The GPS is actually fine. But what is not fine is the accelerometers are waving up and down and also the compass, seen when you click the EKF text in Mission Planner to bring-up the EKF error monitors.
Maybe this makes sense... GPS is relatively solid/stationary, but waving up and down on the AHRS is causing the drone (which assumes it's stationary) to think the GPS is moving (a.k.a. GPS horizontal or vertical error depending on the direction of the wave). But really the drone is logically moving because of the accelerometer and compass "movement".
I switched my INS_GYRO_FILTER and INS_ACCEL_FILTER up to 48Hz and it appeared to help a bit, but I did not have time to test it yet. In the past I had to use even 98Hz filter just to get mini-quads to hover without the infamous "pixhawk wobble". But I found that the new auto-tune does not even work when the filter is set high so preferred recently to lower it back down to the default 20hz and do a good PID auto-tune using the instructions recently posted by Dennis Baldwin on his YouTube channel.
Maybe this is all random. I will attempt to gather logs from my next attempt and also try older firmware...
Thank Tony,
Maybe you'd help me understand why i couldn't do autotune with AC3.3. I think you get us on the right way...
When i downgrade to AC3.1.5 i can arm easier and do autotune.
-
1
-
2
of 2 Next