Using ArduCopter 3.2 Missions for Aerial Panoramas

The latest version of ArduCopter (3.2 -rc11 and above) has several new and improved mission commands and features that make the automated capture of spherical panoramas fairly straightforward. These include a “Command-Yaw” relative command, the Do-Jump command, and an increase in the maximum number of Mission commands (waypoints) for Pixhawk users to 718. It is now possible to create a mission to capture a 10 column by 5 row mosaic panorama with as little as 30 commands using the “Do-Jump” command. If you want to take advantage of the of the increasing overlap in the images as the camera approaches the nadir (fewer images per row), you won’t be able to use the “Do-Jump” command and will have to manually code for each row of photos – but now you won’t bump up against the previous limit for the number of commands in a Mission for any reasonably sized panorama.

The video shows the testing of the Command-Yaw command along with a demo of using it for a panorama. I’ve attached a copy of the Mission file I use for a 5 row, 10 column panorama. The basic flow for the Mission script is to position the frame in a specific yaw direction (e.g., North or East), rotate the gimbal up to row 1, then take a photo. The script then rotates the gimbal to row 2 and takes another picture. After it completes row 5 it yaws CW by 36 degrees and takes another row of pictures. It continues this until all 10 columns are captured. The servo values for the gimbal angles and the amount of yaw for each column are dependent upon the field of view of your camera. 5 rows by 10 columns works well for a Canon SD900. 4 rows by 8 columns works well for a NEX 5N with the 16mm pancake lens and gives about the same resolution panorama. A completed panorama can be seen at the link below.

The view from above Paradise, California (http://www.360cities.net/image/the-view-from-above-paradise-california)

I’m interested in hearing how others are using their copters to do aerial panoramas.

Original Mission

Panorama-Mission-10-Col-5-Row-Start-East-Relative.txt

I’ve been doing some further experimenting after being disappointed in not being able to capture a full size equirectangular jpg panorama (30,000 x 15,000 pixels) with my previous SD900 camera settings and original mission plan. I’ve found that a zoom setting of 2 gives a FOV of each image of about 40 degrees horizontal and 30 degrees vertical would give enough resolution to produce a 30,000 x 15,000 pixels panorama using a 6 row by 14 columns. Unfortunately, a rectangular matrix this large to capture the panorama would require too many pictures (84) and require too much flight time. The alternative to a matrix panorama is one where each row has a different number of images based on its pitch angle. As the pitch angle nears the zenith or nadir, fewer columns are needed in each row. After some tests with my tripod mounted panorama head I came up with the following mission parameters for the panorama:

Row # Pitch Angle # Of Columns Angle Between Images

1 +20 13 28

2 0 14 26

3 -21 13 28

4 -43 11 33

5 -62 8 45

6 -85 2 180

Using this mission the number of photos is reduced to 61 and the time to completion is well within my maximum mission time of 13 minutes (it takes 9 minutes for the photos plus the time to fly to the correct position and then land safely afterwards).

If you are using PTGui or Autopano to build the panorama, the initial layout of the photos is easier if you use a PapyWizard XML file with the camera angles. I’ve attached both the mission and PapyWizard xml files to this post for anyone who wants to try it. There are a few things you need to know about the mission file if you decide you want to use it:

First, you will need to compute the elevation servo settings for your specific gimbal. I taped a printout of a setting circle on the gimbal and put a pointer on the camera then tried various pitch values via a short mission file (on the bench with the props removed) until I found what was needed.

Second, the mission file contains a hybrid of Do-Jump commands for the first three rows and multiple repetitions of similar code for rows 4 through 6. This is because Arducopter 3.2 only allows a maximum of three Do-Jump commands in a mission. That is due to change in 3.3 for PixHawk users but we aren’t there yet.

Third, if a Do-Jump command is preceded by a Condition-Yaw command, there must be a waypoint in between them that provides enough of a delay that the Command-Yaw has time to complete before the flight controller hits the Do-Jump command. If it is not, the Do-Jump command gets skipped. For my frame this is about 3-4 seconds for short rotations, longer for large rotations, yours may be different.

Lastly, you will need to provide enough time for the camera to take an image before moving to the next image. In my case, I need 4 seconds for the images to be captured. I use a small CHDK script (Zoom2.bas) to set the zoom and take two photos (in case one is blurry). It takes about two seconds from the time the shutter command is sent for the script to take the first photo, then almost another two seconds for the second photo to be taken. So there’s a Waypoint command after each Do-Digicam command with a 4 second wait and zero Lat, Lon, and Alt.

You’ll notice that the area between the zenith and about 35 degrees elevation is not covered in the mission. You’ll need to make a separate sky panorama using a pano head to capture the top portion and then blend it with the bottom portion. But, if you’ve gotten this far you already know how to do that.

A test of the mission and methods used above can be found at: http://www.360cities.net/image/over-bille-park-in-paradise-california

Mission with Zoom set to 2 and 61 images

SD900%20_Zoom_2_Pano_61_Images_99_Commands.txt

Papywizard XML file

papywizard_SD900_Zoom2_6Row.xml

CHDK File

Zoom2.bas

Over the last six months I’ve been struggling with understanding how mission commands are processed by the flight controller while I’m trying to capture a panorama. After a great deal of trial and error and an excellent post from Soren Kuula I think I finally have a fair understanding of how complex missions are handled in Arducopter so I thought I’d share it here.

Flying Complex Missions with “Condition” and “Do” Commands

When creating a complex mission with “Condition” and “Do” commands it’s helpful to have a good understanding of how mission commands are processed by Arducopter. You can think of the command processor as having two threads that run simultaneously. One thread processes “nav” commands and the other processes “non-nav” commands. However, it’s not a simple as just processing each command in order.

Think of a mission as being made up of a series of blocks of mission commands. Each block consists of an initial “nav” command (waypoint, loiter, etc.) and the following “non-nav” commands (“do” and “condition” commands). So in Figure 1 think of commands 1 through 4 as Block 1, commands 5 through 7 as Block 2, and command 8 as starting the next block. A block can have many “non-nav” commands or it can be just a “nav” command with no “non-nav” commands.

Figure 1

The command processor processes each block in the following way:

First execute the “nav” command.
When the vehicle is within WP_RADIUS centimeters (default = 200) of the desired waypoint, start the “nav” command delay timer, if any.
When the “nav” command delay timer reaches zero or if there is no delay
1. a) Stop processing “non-nav” commands from the previous block, even if they are not finished
2. b) Start processing the “non-nav” commands in this block in sequence AND, AT THE SAME TIME start processing the “nav” command in the next block.

“Non-nav” commands that are “Do” commands process in a few milliseconds but commands that are “Condition” commands are processed until the conditions are met. In Figure 1 above, command 3, the CONDITION_YAW command, will be processed until the vehicle rotates to within two degrees of the desired yaw angle, at which time it will complete and Command 4, the DO_DIGICAM command, will be processed.

Now the processing gets interesting, notice that Command 5, a WAYPOINT command, has a 7 second delay. Remember in 3b above that the processing of the “non-nav” commands and the “nav” command in the next block starts at the same time. In order to allow enough time for the “non-nav’ commands of the first block to complete, you need to delay the completion of the “nav” command in the second block. Add up the time it takes for each “non-nav” command, add a small margin, and add it as a delay on the following “nav” command. Here are the delays I calculated for each “non-nav” command in the first block:

2 - 0 seconds

3 - 3 seconds to rotate from some arbitrary heading to 90 degrees due east

4 - 3 seconds for the CHDK script to initialize and take two images

Plus 1 second for margin

So command number 5, WAYPOINT, has a 7 second delay. You will need to do this math for every block in the mission that has “non-nav” commands. Even if every “non-nav” command is a DO command you will still need to add at least a one second delay when you use a “nav” command that completes right away such as a WAYPOINT with zero Lat, Lon, and Alt.

Using DO_JUMP

The DO_JUMP command is helpful when you want to repeat a sequence of commands multiple times. However, there is one characteristic of the command that you need to take into account, you cannot jump to another command in the same block! All jumps must be to a different block. There is an additional restriction that you cannot have more than three DO_JUMP commands in a mission in Arducopter Version 3.2.x. This is on the list to be increased in Version 3.3.

Consider this example:

Figure 2

Commands 1 – 10 are the first part of a mission to capture a 360 degree panorama.

WAYPOINT Command 1 starts Block 1. Command 2 sets the pitch angle of the camera gimbal and Command 3 points the entire copter due East. Command 4 takes the first photo.

WAYPOINT Command 5 starts Block 2. Command 6 rotates the copter by 28 degrees relative clockwise and Command 7 takes another photo. Command 8 starts the third block and allows enough time for the yaw to complete and the CHDK script to run after the DO_DIGICAM_CONTROL command.

WAYPOINT Command 8 starts Block 3. Command 9, the DO_JUMP command, sends processing back to Command 6 in Block 2. In addition to moving the “non-nav” processing pointer to Command 6 it moves the “nav” processing pointer back to Command 8 and restarts the 5 second timer. It repeats this 11 times to get the remaining photos for the first row of the panorama.

WAYPOINT Command 10 starts Block 4. This block will change the pitch angle of the camera gimbal and subsequent blocks capture the remaining photos.

Here’s what the command processing looks like in the log, notice the interleaving of the “nav” and “non-nav” commands:

Figure 3

Timing of the CONDITION_YAW and the DO_DIGICAM_CONTROL commands

With limited air time (about 8-9 minutes) and a desire to do a 61 photo panorama, the amount of time taken for each photo is critical. Even one extra second per photo means another minute in the air so I did a lot of testing to minimize the amount of time taken in the main loop between commands 6 and 9 (which is repeated multiple times in the remainder of the mission, not shown here). I’ve shown in Figure 4 below how I arrived at my timings so that others, with different sized frames, can develop their own timings. The time required to run the mission is about 5 minutes and 45 seconds. Combined with a takeoff to mission start of 45 seconds and a mission end to land of 1 minute and 55 seconds I have just enough time to get back on the ground before the battery voltage failsafe kicks in!

Figure 4

Figure 4 is a graph of the desired yaw angle and the actual yaw angle. The CONDITION_YAW relative command is initiated at time T0 which adds 28 degrees to the current Desired Yaw angle. At time T1 the frame has rotated to within 2 degrees of the desired yaw angle and processing of the CONDITION_YAW command is completed. The DO_DIGICAM_CONTROL is then executed. The CAM_DURATION time for the DO_DIGICAM_CONTROL is .3 seconds. On my Canon camera, CHDK does not start until the shutter voltage is “released” so time T3 is when CHDK actually starts. Through separate testing I determined that it takes CHDK about 1 second to take the first photo and another second to take the second photo from that position (the second photo is insurance, I need the second one to substitute for the primary photo for one or two of the images in each mission due to blurring). So the total time required for the photos is almost 2.5 seconds.

There are two big variables, the time for the frame to rotate to within 2 degrees of the desired Yaw Angle and the amount of time for the frame to settle back down after any overshoot. PID tuning and wind play a big factor in both. I’m using the default PIDS, the Autotune PIDs were two jerky for my purposes. As a result of all the testing I ended up using 3 seconds delay after the DO_DIGICAM_CONTROL command by either building it into the WAYPOINT delay as in this example or adding in a separate CONDITION_DELAY after each photo.

Yaw Stability

Wind is a problem for me because the gimbal has to be mounted forward of the centerline to balance the batteries. This creates an asymmetrical cross section that acts as a rudder in moderate (10 MPH) winds and tries to rotate the frame off the desired yaw setting. This can invoke a “feature” of Arducopter that changes the desired yaw setting if it can’t be attained within a few seconds.

Figure 5

In Figure 5 it shows up in two places as an uncommanded change to desired yaw that is exactly 10 degrees off from the actual yaw. It then tries to stay within 10 degrees of the new desired yaw, and if not will continue to reset the desired yaw until it can. If you use a log series of CONDITION_YAW relative commands or you spend more than 10-20 seconds in one yaw position this can cause subsequent gaps in your panorama when you thought you had taken overlapping photos. I’ve minimized this in my mission plan by doing the panorama row by row instead of column by column, not staying on one yaw heading more than three seconds, and resetting the yaw of the copter back to an absolute yaw position at the beginning of every row of photos.

You need to be a member of diydrones to add comments!

Join diydrones

Comments

Phill Simpson April 7, 2015 at 3:17pm
Over the last six months I’ve been struggling with understanding how mission commands are processed by the flight controller while I’m trying to capture a panorama. After a great deal of trial and error and an excellent post from Soren Kuula I think I finally have a fair understanding of how complex missions are handled in Arducopter so I thought I’d share it here.

Flying Complex Missions with “Condition” and “Do” Commands

When creating a complex mission with “Condition” and “Do” commands it’s helpful to have a good understanding of how mission commands are processed by Arducopter. You can think of the command processor as having two threads that run simultaneously. One thread processes “nav” commands and the other processes “non-nav” commands. However, it’s not a simple as just processing each command in order.

Think of a mission as being made up of a series of blocks of mission commands. Each block consists of an initial “nav” command (waypoint, loiter, etc.) and the following “non-nav” commands (“do” and “condition” commands). So in Figure 1 think of commands 1 through 4 as Block 1, commands 5 through 7 as Block 2, and command 8 as starting the next block. A block can have many “non-nav” commands or it can be just a “nav” command with no “non-nav” commands.

Figure 1

The command processor processes each block in the following way:
1. First execute the “nav” command.
2. When the vehicle is within WP_RADIUS centimeters (default = 200) of the desired waypoint, start the “nav” command delay timer, if any.
3. When the “nav” command delay timer reaches zero or if there is no delay
  
  a) Stop processing “non-nav” commands from the previous block, even if they are not finished
  
  b) Start processing the “non-nav” commands in this block in sequence AND, AT THE SAME TIME start processing the “nav” command in the next block.
“Non-nav” commands that are “Do” commands process in a few milliseconds but commands that are “Condition” commands are processed until the conditions are met. In Figure 1 above, command 3, the CONDITION_YAW command, will be processed until the vehicle rotates to within two degrees of the desired yaw angle, at which time it will complete and Command 4, the DO_DIGICAM command, will be processed.

Now the processing gets interesting, notice that Command 5, a WAYPOINT command, has a 7 second delay. Remember in 3b above that the processing of the “non-nav” commands and the “nav” command in the next block starts at the same time. In order to allow enough time for the “non-nav’ commands of the first block to complete, you need to delay the completion of the “nav” command in the second block. Add up the time it takes for each “non-nav” command, add a small margin, and add it as a delay on the following “nav” command. Here are the delays I calculated for each “non-nav” command in the first block:

2 - 0 seconds

3 - 3 seconds to rotate from some arbitrary heading to 90 degrees due east

4 - 3 seconds for the CHDK script to initialize and take two images

Plus 1 second for margin

So command number 5, WAYPOINT, has a 7 second delay. You will need to do this math for every block in the mission that has “non-nav” commands. Even if every “non-nav” command is a DO command you will still need to add at least a one second delay when you use a “nav” command that completes right away such as a WAYPOINT with zero Lat, Lon, and Alt.

Using DO_JUMP

The DO_JUMP command is helpful when you want to repeat a sequence of commands multiple times. However, there is one characteristic of the command that you need to take into account, you cannot jump to another command in the same block! All jumps must be to a different block. There is an additional restriction that you cannot have more than three DO_JUMP commands in a mission in Arducopter Version 3.2.x. This is on the list to be increased in Version 3.3.

Consider this example:

Figure 2

Commands 1 – 10 are the first part of a mission to capture a 360 degree panorama.

WAYPOINT Command 1 starts Block 1. Command 2 sets the pitch angle of the camera gimbal and Command 3 points the entire copter due East. Command 4 takes the first photo.

WAYPOINT Command 5 starts Block 2. Command 6 rotates the copter by 28 degrees relative clockwise and Command 7 takes another photo. Command 8 starts the third block and allows enough time for the yaw to complete and the CHDK script to run after the DO_DIGICAM_CONTROL command.

WAYPOINT Command 8 starts Block 3. Command 9, the DO_JUMP command, sends processing back to Command 6 in Block 2. In addition to moving the “non-nav” processing pointer to Command 6 it moves the “nav” processing pointer back to Command 8 and restarts the 5 second timer. It repeats this 11 times to get the remaining photos for the first row of the panorama.

WAYPOINT Command 10 starts Block 4. This block will change the pitch angle of the camera gimbal and subsequent blocks capture the remaining photos.

Here’s what the command processing looks like in the log, notice the interleaving of the “nav” and “non-nav” commands:

Figure 3

Timing of the CONDITION_YAW and the DO_DIGICAM_CONTROL commands

With limited air time (about 8-9 minutes) and a desire to do a 61 photo panorama, the amount of time taken for each photo is critical. Even one extra second per photo means another minute in the air so I did a lot of testing to minimize the amount of time taken in the main loop between commands 6 and 9 (which is repeated multiple times in the remainder of the mission, not shown here). I’ve shown in Figure 4 below how I arrived at my timings so that others, with different sized frames, can develop their own timings. The time required to run the mission is about 5 minutes and 45 seconds. Combined with a takeoff to mission start of 45 seconds and a mission end to land of 1 minute and 55 seconds I have just enough time to get back on the ground before the battery voltage failsafe kicks in!

Figure 4

Figure 4 is a graph of the desired yaw angle and the actual yaw angle. The CONDITION_YAW relative command is initiated at time T0 which adds 28 degrees to the current Desired Yaw angle. At time T1 the frame has rotated to within 2 degrees of the desired yaw angle and processing of the CONDITION_YAW command is completed. The DO_DIGICAM_CONTROL is then executed. The CAM_DURATION time for the DO_DIGICAM_CONTROL is .3 seconds. On my Canon camera, CHDK does not start until the shutter voltage is “released” so time T3 is when CHDK actually starts. Through separate testing I determined that it takes CHDK about 1 second to take the first photo and another second to take the second photo from that position (the second photo is insurance, I need the second one to substitute for the primary photo for one or two of the images in each mission due to blurring). So the total time required for the photos is almost 2.5 seconds.

There are two big variables, the time for the frame to rotate to within 2 degrees of the desired Yaw Angle and the amount of time for the frame to settle back down after any overshoot. PID tuning and wind play a big factor in both. I’m using the default PIDS, the Autotune PIDs were two jerky for my purposes. As a result of all the testing I ended up using 3 seconds delay after the DO_DIGICAM_CONTROL command by either building it into the WAYPOINT delay as in this example or adding in a separate CONDITION_DELAY after each photo.

Yaw Stability

Wind is a problem for me because the gimbal has to be mounted forward of the centerline to balance the batteries. This creates an asymmetrical cross section that acts as a rudder in moderate (10 MPH) winds and tries to rotate the frame off the desired yaw setting. This can invoke a “feature” of Arducopter that changes the desired yaw setting if it can’t be attained within a few seconds.

Figure 5

In Figure 5 it shows up in two places as an uncommanded change to desired yaw that is exactly 10 degrees off from the actual yaw. It then tries to stay within 10 degrees of the new desired yaw, and if not will continue to reset the desired yaw until it can. If you use a log series of CONDITION_YAW relative commands or you spend more than 10-20 seconds in one yaw position this can cause subsequent gaps in your panorama when you thought you had taken overlapping photos. I’ve minimized this in my mission plan by doing the panorama row by row instead of column by column, not staying on one yaw heading more than three seconds, and resetting the yaw of the copter back to an absolute yaw position at the beginning of every row of photos.
Phill Simpson November 2, 2014 at 6:51am

Are you using 3.2-rc11 or later? Sounds like you might be using 3.1.x. That version had a problem with sending craft to Africa if you used a waypoint with 0 lat and 0 Lon. Setting home to Seoul should be enough.
Kyomo Jung November 2, 2014 at 5:40am

Hi Phill again~

I wrote your mission plan to my Hexa copter and she had my home town - Seoul Korea - as Home. But whenever I started the mission, it flied to direction of west. I checked the plan but I couldn't find any lan, long data from your plan. what should I check? thanks in advance ^^
Phill Simpson November 1, 2014 at 9:59am

@criro1999, nice panos.
Kyomo Jung October 31, 2014 at 5:00pm

@Phill, thanks for your nice answer ^^
criro1999 October 31, 2014 at 4:40pm

Nice article.

I am doing also 360 panorama pictures, see below those.

http://aerialtasks.com/aerialtasks/Photos.aspx?AlbumID=1
Phill Simpson October 31, 2014 at 3:03pm

I used PTGui Pro. When I was looking for stitching software I tried Autopano Giga as well as PTGui Pro. They both do stitching well and have similar capabilities and pricing. I settled on PTGui Pro because the user interface was more intuitive to me and I didn't have to always have the manual in front of me for directions. There are free trials for both, try them out and see which one you like.
MR60

Hugues October 31, 2014 at 2:55pm

Hello, what software did you use to stitch the pictures into a360 deg pano?
Phill Simpson October 31, 2014 at 7:39am

There is a small note on the Waypoints wiki (at: http://copter.ardupilot.com/wiki/common-planning-a-mission-with-way...) that says:

(Note: DO_xxx commands currently need a dummy waypoint placed after the command
eg:

WAYPOINT_1
DO_SET_HOME
WAYPOINT_2

I've found that I also needed it with a CONDITION_ command, a waypoint with 0 Lat, 0 Lon and 0 Alt says "stay in the current position". Also, there needs to be at least a 1 second delay on the waypoint to give the DO_ or CONDITION_ command time to execute. In my case I use more than 1 second to give the copter time to complete the yaw, the gimbal time to complete the pitch up or down, and the camera time to take a couple of photos before moving to the next position.
Kyomo Jung October 31, 2014 at 4:32am

Hi Phill,

It's very helpful for me and some questions.

I don't know what the WPs which don't have no lat, log information mean in this mission.

Could you explain about them?

of 3

This reply was deleted.