top of page

Parallel Processing UAV Imagery

Here at the Clemson Center for Geospatial Technologies we use a wide array of UAV's (Unmanned Aerial Vehicles) to create high-resolution geospatial products, such as true-color orthomosaics and 3D point clouds from sets of photos. We apply Structure from Motion (SfM) processing methods to the raw data which can be computationally intensive, especially for large datasets, so we set up our software on Clemson's HPC Palmetto Cluster to increase the efficiency of our processing, and you can use this page to learn how to use these scripts yourself.

​

By Justin Dowd (CCGT Intern) and Blake Lytle (UAV and GIS Specialist)

Dealing with Large Sets of Aerial Imagery

Dealing with Large sets of Aerial Imagery

Processing large sets of aerial images through PhotoScan on a single device can be time consuming, even if you're using a high-performance computer such as a VR Gaming Desktop. However, using Clemson's Palmetto Cluster, along with our processing scripts you can drastically reduce the amount of time it takes to process data through PhotoScan. All of this is done by typing in one command on Palmetto Cluster after you set up our scripts!

Palmetto Cluster

The Clemson Cyberinfrastructure Technology Integration (CITI) group has a large research cyberinfrastructure and one piece of it is Palmetto Cluster. Palmetto is a high-performance computing resource that is located locally on Clemson's campus. It is put to use by students, faculty, and researchers from all areas of study. It currently contains 2,021 compute nodes, making up 23,072 CPU cores. Palmetto also contains the following:

​

  • ​​NVIDIA Tesla, K20, and K40 GPUs

  • 100GB of personal storage

  • 10 Gbps Ethernet

​

The most common use for Palmetto Cluster is to solve complex computational problems and in our case that is conducting photogrammetric processes, which means we are essentially using Palmetto Cluster to make measurements using images which creates 3D spatial data and with that data we create Dense Clouds, Digital Elevation Models(DEMs), and Orthomosaics of various landscapes, such as a portion of Clemson University, which will be used in our example.

​

A Palmetto account to access these resources is FREE to all Clemson students and faculty and more information about what Palmetto Cluster has to offer and how to get an account can be found here.

Photoscan Software

Agisoft Photoscan is software that conducts photogrammetric processing on images of landscapes or objects and from that processing it will then create a variety of 2D and 3D spatial data, such as orthomosaics, 3D point clouds, digital elevation models (DEM's), and textured meshes. These data are ready to be brought into GIS for mapping and analysis purposes. The software can be run on a single computer or in parallel on a cluster to speed up processing. On a cluster, the software will have a Server Node, which divides processing tasks and distributes them amongst a pool of Processing Nodes, which perform the actual computation. Setting up the software on a cluster can be automated using scripts written in Python.

Our scripts create a PhotoScan Server Node and the parallel processing array and from there the script breaks down each processing step into individual jobs and submits them one at a time to the server node to be processed. From there the server node distributes the task computations amongst the parallel processing array.  First the GIS user will connect to Palmetto, then after connecting the user is going to run our processing script which then creates a PhotoScan Server Node and a Parallel Processing Array. Afterwards Palmetto will then start submitting jobs to the array that do the following PhotoScan processes:

  • Add Photos to the Project

  • Match Photos

  • Align Cameras

  • Build Depth Maps

  • Build Dense Cloud

​

​

Using our Scripts on Palmetto

Using our Scripts on Palmetto

If you wish to process data using our scripts, along with Palmetto, it can easily be done from your personal Windows machine. However, prior to using our scripts we highly recommend attending the following CITI Group workshops to make your Palmetto experience easier:

​

  • Intro to Linux

  • Intro to Research Computing

​

More information about those workshops can be found here.

Getting the Required Documents

Step 1: Obtaining CCGT Scripts and Downloading an SSH Client 

To download our scripts one must first navigate to our GitHub page, from there select "PhotoScan_Scripts", then "Clone or Download" and "Download ZIP". After you have extracted the scripts from the downloaded ZIP file, you must now download an SSH(Secure SHell) client, which is an application that allows us to remotely access a computer, in our case we are accessing Palmetto(for first time users we recommend downloading this SSH client since it is referenced in this example as well as used in our detailed instructions). After the SSH client has been installed you can connect to Palmetto using your Palmetto Account by connecting to the following hostname, "login.palmetto.clemson.edu". After connecting you can move onto the next step.

​

​

Step 2: Downloading and Installing PhotoScan on Palmetto

The latest PhotoScan version to download can be found here, be sure to download the Pro version and its Linux distribution. This should give you a file with a ".tar.gz" extension which you are then going to transfer to your Palmetto storage using the file transfer client that comes with our recommended SSH client. After successfully transferring the install file you will then move to the command prompt of the SSH client and navigate to the location of the install file. After reaching that location you will issue the command "tar xzf photoscan-pro_X_X_X_amd64.tar.gz" (you would replace the 'Xs' with numbers corresponding to the version of PhotoScan you are installing, for this tutorial it would be "_1_4_1" since version 1.4.1 is the version supported by our scripts. That command will extract all the required PhotoScan files and store them in a newly created directory called "photoscan-pro". Be sure to make note of the direct path leading to this location.

Transferring Scripts to Palmetto and Preparing for Processing

Step 3: Transferring CCGT Scripts and Photos to Palmetto

PhotoScan has now been installed on our Palmetto Account and we can now transfer the CCGT Scripts from our local machine to Palmetto using the same file transfer client we used previously in Step 2. We are going to move all of the files in the ZIP file we downloaded from GitHub, except for the file named "arguments.txt" which will be transferred in its own step. Here is a list of those files:

​

  • initNetProcessing.sh                           

  • initProcNode.sh

  • initServerNode.sh

  • procNodeArray.sub

  • interact.sub

​

​

​

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Be sure to make note of the direct path to location where your scripts are located on Palmetto because it will be needed later. We can now also transfer the photos we wish to process using the transfer client as well. Also be sure to make note of the direct path to the location where you have uploaded your photos to.

​

Step 4: Transferring "arguments.txt"

Now we have to upload "arguments.txt" to our home directory on Palmetto which should have the direct path that appears like this: "/home/username". The reason it must be stored in this specific location is so it can be seen globally by the submission scripts.

​

​

​

​

​

​

​

​

​

​

​

​

Step 5: Editing "arguments.txt"

We have successfully uploaded "arguments.txt" and we must now move back to the Palmetto command line in order to edit it based on your specific needs. To do this be sure you are in your home directory, if your are not simply type the command "cd ~" and that will move you into your home directory. From there type the command "nano arguments.txt" to open the file for editing.

​

Note: For beginner users we recommend using "nano" for editing, if your are an experienced Linux user, feel free to use whatever text editor you like. 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

​

 

 

 

 

 

 

 

Following the instructions listed in "arguments.txt" go ahead and edit the file and when finished save it by using the keyboard command "ctrl + o" and hit enter and then use the keyboard command "ctrl + x" to exit the document.

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

  • AlignCameras.py

  • MatchPhotos.py

  • BuildDepthMaps.py

  • BuildDenseCloud.py

  • submitAlignCameras.sh

  • submitMatchPhotos.sh

  • submitBuildDepthMaps.sh

  • submitBuildDenseCloud.sh

Starting an Interactive Session and Opening the Network Monitor

Step 6: Starting an Interactive Session with Palmetto

We are now ready to begin processing of our data, but first we must setup the Agisoft Network Monitor to allow us to monitor the progress of our scripts. To do this we will first have to start an interactive session with Palmetto. An interactive session allows a user to interact with the node you connect to via the keyboard as well as use what is called X11 tunneling software to produce graphical user interface (GUI) windows on your local machine using the graphics hardware found on the node. Luckily we have provided a script that will request an interactive with Palmetto for you. To run that script we must first open a second SSH window and after connecting to Palmetto, navigate to the location where you stored the CCGT scripts. To do this use the "cd" command followed by the direct path leading to that location and hit enter. The complete command should look somewhat like this: "cd /home/username/script/file/location", once there we must now give executable permissions to all of the scripts. To do this type the command "chmod -R 777" and hit enter. Now all the files will be executable and you now type "qsub interact.sub" and hit enter and it will begin an interactive session with Palmetto. The output to your screen should look something like this:

​

​

​

​

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Once the interactive session is running we can then move on to Step 7.

 

Step 7: Opening the PhotoScan Network Monitor

Our interactive session is now running and we now must change to the location where PhotoScan was installed on our Palmetto account. Once again use the "cd" command to change to that location and once there we are going to start the network monitor. To do this type the command "./monitor.sh" and hit enter. The network monitor should now appear via the X11 tunneling software and look like this:

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Submit Processing Jobs to Palmetto and Viewing the Output

Step 8: Submitting Processing Jobs

With the monitor now up and running we can now go back to the first SSH window we opened and navigate to the location of the CCGT scripts using the "cd" command once again. Once there we will then start the processing sequence by typing "./initNetProcessing.sh" and hitting enter. Once you hit enter Palmetto job ID's will now start outputting to the screen, the first two job ID's correspond to the jobs that will first create a PhotoScan network server and the second job is what creates a PhotoScan parallel processing array. After the second Job ID is outputted, an IP address will outputted. Copy that IP address into the network monitor and select connect and all of the parallel processing array will appear. After that the first processing job ID will appear and shortly after that the job should appear in the monitor. Once the first processing step has completed, the next step will be submitted to the server node, and that will continue until all processing steps have been completed.

​

Step 9: Viewing the Output

Once processing has completely finished we are now ready to view our finished product. To do so we first need to close the Network Monitor window using the "X" in the upper right-hand corner. Now go to your SSH window where we have an interactive session running, this should be the same window we launched the Network Monitor from. In this window we are going to type the command "./photoscan.sh" and this will launch the PhotoScan Graphical User Interface(GUI). From here we would go to "File" and "Open" and navigate to the location on your Palmetto Account where you had PhotoScan save your project. Once you reach that location select the file that has your project name with a ".psx" extension, double-click that name and your project will now open and you are now free to view the output.

​

​

​

If you wish to try this yourself using your own data, we have linked our detailed instructions for you to download and follow on your own.

 

 

 

 

 

 

 

 

 

 

Network Processing Instructions

 

 

If you have any questions about our scripts, connecting to Palmetto, PhotoScan, or anything GIS related, please feel free to contact one of the staff members below:​​

​

​​

​

​

bottom of page