Beyond Visual Line-of-Sight: development and challenges of cellular-enabled Unmanned Autonomous Aerial Vehicles (UAAVs)

21 min readMar 7, 2019

by Mauricio Andrada and Derek Ohlarik

Introduction

For the casual observer fitting UAAVs with cellular capability, allowing them to fly using the mobile network to send data to and receive commands from a remote operator seems to be a simple task; some might think it’s as simple as strapping a smartphone to the UAAV or plug a modem to the flight controller.

From a technological perspective this is not far from the truth; but as we all know, the devil is in the details.

In this article we’ll talk about some of the technical challenges associated with flying UAAVs BVLOS as well as some technological solutions for these problems[1].

Then we’ll describe our own experiments using the 4G/LTE network for remote command and control of an UAAV while in visual line-of-sight.

Currently, in the US, one must secure a waiver for flying UAAVs BVLOS[2]. Our experiments were all executed under FAA rules Part 107[7] so although they apply to BVLOS flights they were all executed in visual line-of-sight conditions.

We hope you enjoy this article and get ready for this next technological leap.

Why do we need BVLOS UAAVs?

The main driving force behind development of BVLOS UAAVs is, of course, cost reduction. Human-controlled UAAVs are already bringing significant cost savings in areas like construction, surveying, delivery, news reporting and security. Having the ability to remotely fly UAAVs will introduce even greater cost savings to these and many other industries. Here are some examples:

Surveying

Nowadays UAAVs are routinely used for surveying large areas like farms and industrial sites. The process today requires the surveyor to be physically on-site to manage operation of the UAAV; also the extension of the area that can be surveyed this way is limited by the range of the radio controller as well as the distance the surveyor can maintain visual contact with the UAAV. Data collected by the UAAV is stored locally and retrieved by the surveyor when the flight is completed.

This is already cheaper then having a surveyor to drive/walk around the property and manually collecting measurements but is still not ideal; for example if there’s an issue with measurements collection during flight the operator will only find out after the flight is finished, maybe requiring a new flight starting from scratch.

With a cellular-enabled UAAV allowed to fly BVLOS a surveying company must still physically bring the UAAVs to the location being surveyed but that can be done by a much lower cost: the UAAV can be delivered by mail or carrier service and, depending on the distance, even fly directly from centralized drone hubs.

Once on site, the UAAV can be powered on by someone in the premises and a remote operator can program the mission to be executed and collect data real-time.

Under these conditions the area that can be surveyed is limited only by the flight autonomy of the UAAV; the data surveyed can be transmitted real-time to the surveying company; any issues with the equipment can be detected immediately; one surveyor can manage multiple properties simultaneously saving the company — and the customer — both time and money.

Insurance industry

Inspection of damaged areas for insurance purposes is a core activity in the insurance industry; inspecting hazardous areas in person is both costly and dangerous.

Similarly to surveying companies, insurance companies use on-site UAAVs to inspect damage to properties in a safe and cost effective way, in particular in areas that would be too dangerous to inspect in person (e.g. areas affected by natural disasters).

This activity however suffers the same limitations associated with surveying and can garner the same benefits if moving to BVLOS UAAVs.

Inspection

Another potential use case for BVLOS UAAVs is inspection using a UAAV equipped with cameras and backed by a cloud-based Artificial Intelligence service to detect and recognize defects. Power transmission lines, construction sites, remote cellular sites, etc. can take great advantage of a BVLOS UAAV.

Delivery

Delivery is a clear use case for BVLOS UAAVs. Several large retail companies are currently experimenting with the technology in hopes of optimizing delivery time while greatly reducing costs.

Ok, I’m convinced. Let’s do it then.

Well, as we said before, the devil is in the details. Here is a small sample of the many questions that need answers before we can deploy a reliable and robust infrastructure for BVLOS UAAVs.

How do we remotely tell the UAAVs where to go?
What happens if the UAAV loses connectivity with the network?
What happens if the software has a bug on-flight?
How do you prevent malicious operators to use the network?
How do you track UAAVs?
Are there any impacts of a UAAV in the network (radio interference, etc.)?
How to you manage air space shared with other aircrafts?
How do you make sure UAAVs are only accessible by their operators? In other words, how do you make them secure against hackers?
What fail-safes must be in place (e.g. collision avoidance, loss of GPS, etc.)?

As it can be seen there are a lot of things that must be agreed upon by UAAV manufacturers, operators and regulatory agencies (FAA, FCC) before one can go around flying UAAVs BVLOS safely.

In the next sections we’ll describe some potential answers for these questions.

The Radio problem

In order for a UAAV to be able to fly BVLOS it must be connected to some wireless network. It must be possible for an operator to know where the UAAV is at all times and it’s current state (speed, battery, position, altitude, etc.). The operator must also be able to send both missions and ad hoc commands to the UAAV from a remote station.

There are a few options for connecting a device to the management back-end:

Satellite[3][4][5]

Satellite communication has been used with UAVs for several years now. There are many studies from both ICAO and NASA on the specifications for frequencies, power, limitations, etc. of using satellite communications for UAVs; there are companies (e.g. Inmarsat) that provide commercial systems for both control and data services using IP-based protocols.

However, satellite communications have a number of limitations:

Receiver/transmitter weight: the lightest receiver/transmitter is around 3 lbs so not suitable for smaller UAAVs
Data link speed: speed limited to 200/400 kbps per channel so although suitable for command and control it’s not good enough for applications that need high-speed data links (e.g. high-definition live video); those require allocation of multiple channels
Pricing: satellite data link services are still relatively expensive; in particular for applications that need multiple data channels pricing can be prohibitive.
Satellite infrastructure: satellite infra-structure is complex, requiring satellite monitoring, earth stations, etc.

Cellular

Research has been on going regarding using the cellular network for BVLOS UAAVs for both command/control and data streaming, in particular using the 4G/LTE network and more recently 5G. There are many compelling reasons for using cellular:

It’s ubiquitous
It works very well outdoors
It’s cheap; even the more expensive high-speed unlimited data plans only cost a little over 100 USD per month

There are of course some important issues to be addressed:

Radio propagation patterns

If a cell phone is within the main propagation lobe of the antenna then less power is required for communication
Having lobes pointing down — as opposed to pointing horizontally — means less interference between base stations

Therefore it’s quite obvious that drones using LTE may not take full advantage of the radio signal since they’ll most likely not be flying within the main propagation lobes (LTE antennas typically are built below 200 ft above ground while UAAVs can fly up to 400 ft above ground); instead they are forced to use the secondary lobes that propagate backwards and up from the antenna, which certainly impacts coverage, power need for communication and maximum data rates that can transmitted/received.

Line-of-sight interference

In regular cellular communication antennas are positioned in such a way that a mobile device will always be connected to one base station through that antenna; if the mobile device moves and finds other antennas a handover procedure may take place to disconnect the mobile device from the current base station and seamlessly reconnect it to the next. This handover has a price: messages must be exchanged between different components in the network to guarantee a smooth transition without breaks in the communication; therefore, antennas are positioned in such a way that it reduces the chances a mobile device will see multiple lobes with similar power levels, which would make the selection of the correct base station difficult and may force the device to “hop” around, switching from one antenna to another.

Antenna positioning is also important for reducing interference: if a mobile device is connected to the network using antenna A, we certainly don’t want a lot of radio power from that device to reach antenna B since this signal will just look like noise to antenna B and will affect the performance of other mobile devices connected to antenna B.

Now all this work is done in order to optimize radio coverage for devices that are — usually — on or near the ground.

Now UAAVs may fly as high as 400 ft. above ground; at this altitude the LTE radio on the UAAV may be in line of sight of tens of antennas; even worse, depending on how far it is from them, they may all have very similar signal levels from the UAAV perspective (remember, the UAAV is using the secondary radio lobes to communicate with the network). This has some important consequences:

The LTE radio may keep hopping from one antenna to another, incapable of locking to one antenna for longer periods of time; as we’ve seen this causes a lot of message exchanges between elements in the network and therefore affects the overall network capacity
Changing from one base station to another requires signaling between the UAAV and the network: the network must know which base station is being used by the UAAV so data can be properly routed. A lot of hopping means a lot of signaling; a lot of time spent doing signaling means less time for data transmission and therefore impacts on the overall data rate of the network
The radio signal from the UAAV reaches all antennas in line-of-sight so even if the UAAV is connected to antenna A, all the other antennas see that signal as noise; as we’ve seen before, this impacts the performance of other devices connected to these antennas.

More recently work has been conducted using 5G antenna technology; it has the potential to resolve a lot of these problems. By supporting beam forming techniques the 5G antennas allow directing a main RF lobe to the UAAV and keep it separated from other antennas; the relative short range of 5G milliliter wave used in 5G helps preventing interference with multiple antennas.

This work is still in the early stages so there’s not much data yet to reach a conclusion.

Alternative RF options

In order to circumvent some of these issues there is some interesting on-going research in the subject of using the mobile network for BVLOS flight:

Use a different frequency band
If UAAVs use a different radio frequency band than the one used for regular mobile communication the interference with other smartphones can be greatly reduced
Use directional antennas in the UAAV
If the UAAV uses directional antennas then it can isolate the signal from one or two different base-stations, greatly reducing both interference and hopping. Examples of directional antennas are parabolic antennas and phased arrays (signal for each antenna in the phased array is electronically delayed so the phase difference between overlapping signals emulates the behavior of directional antennas). 5G, with its beam forming technology, is a strong candidate for a solution
Reserve channels within the regular cellular band used for communication just for UAAVs
Install a dedicated set of antennas specifically for UAAVs (e.g. Gogo Inc.)
These antennas would be designed and placed to minimize inter-cell interference while providing enough power to prevent UAAVs to try to attach to the regular mobile network.
This is one of the ways airplanes can provide on-board Wi-Fi services over 3G
Use ACARS (Aircraft Communications Addressing and Reporting System)
Uses VHF (30 to 300 MHz) or satellite
Has relatively low data rates so it can’t be used for high-speed data, only command and control

For our purposes we wanted to validate the feasibility of using the mobile network for command and control of a UAAV so we used a 4G/LTE USB modem; it’s easy to use, lightweight and cheap.

The Hardware Design

So what are the requirements and constraints guiding our design choices?

For controlling the UAAV itself a flight controller is a must. We looked at a number of hardware/software options and found that the Pixhawk[8] flight controller using PX4[9] software worked well for our experiments. PX4 is open source so we could potentially customize it of needed; Pixhawk is a very popular flight controller used in the market by both hobbyists and companies.

Another advantage of the Pixhawk/PX4 system is its support to MAVLINK, which make it possible to use a separate on-board computer to communicate with the flight controller and receive telemetry from it.

For a feasible commercial product to be used in a UAAV operated beyond line of sight we established the following constraints on the network:

High-speed data
Remote access beyond line of sight
Reliable coverage
Low cost

Using the mobile network, specifically 4G LTE on band 13 or band 4, seemed to meet these requirements despite the known issues described above.

In order to be able to access the LTE network the UAAV needs an LTE modem.

We had a few options:

Connect the LTE modem directly to the Pixhawk
Connect a WiFi modem to the Pixhawk and then connect it to a mobile hotspot that would convert WiFi to LTE
Connect the LTE modem in an on-board computer communicating with the flight controllers via UART and a standard protocol (e.g. MAVLINK)

Option 1 is feasible but would require a relatively large amount of work:

Add support to LTE modems to the flight controller software and OS
Resolve potential conflicts between the LTE and the WiFi interface, already supported by some flight controllers

Option 2 is doable but stacking all the required hardware in the UAAV, along with the wiring, makes it difficult to assemble and operate.

On the other hand Option 3 had the following advantages:

Security: bugs with processing the LTE signal that could cause the flight controller to fail would deem the UAAV impossible to fly and very likely would cause a crash; having this part running in a separate on-board computer mitigates the problem
Software: no need to customize the flight controller software to support the LTE modem; customers can add the on-board computer with LTE modem to their flight controllers as is.
Support to sensors: Pixhawk does support a number of sensors but it has limitations; for example if we wanted to plug in a 4K camera to send live stream video from the UAAV the Pixhawk would not be able to support that.

We tried a number of different computer boards until we settled on the ROCK64; it has 3 USB ports, 2 UARTs and support to I2C, along with a powerful enough CPU/GPU chipset in a compact board about the size of a credit card[6].

The Operating System

Now we arrived to the point where we had to make a choice on the OS running on the on-board computer. In that case our constraints were:

Must be an open source OS
Helps keeping costs down
Allows us to make customizations to it
Can’t have a strong left-hand licensing, like GPL
Some customization may be proprietary and we’d not want to publish them
Must be an OS that supports carrier’s existing applications for network monitoring and diagnostics
This is a device that will be using the cellular network; therefore we must be able to track and monitor how these devices affect the network
Must be an OS with a proven record when used with mobile devices connected to the cellular network

Under these constraints our OS of choice was AndroidTM[10], specifically version 7:

Existing applications built for Android smartphones for network monitoring and diagnostics can run in the drone with few or no changes
It supports firmware updates over the air (FOTA) and remote management of applications (e.g. via Firebase services)
Android has been running on billions of connected devices all around the world
It has all the framework for supporting LTE already in place
The “sandbox model” guarantees that if one application fails and crashes it’ll not bring down the whole system and the other applications will continue to run.

The Software Design

Now that we have our hardware and OS defined, we need to define how are we going to control the UAAV over LTE.

Our hardware design implies one software architecture choice:

Commands and missions must be sent from some Ground Control application to the on-board computer via LTE which then sends them to the flight controller via UART via commands using the protocol supported by the flight controller — in our case MAVLINK
Here we have a few options:
* Software in the on-board computer is just a proxy between the Ground Control and the flight controller, blindly passing the commands to the controller and responses back to Ground Control
* Software in the on-board computer has some intelligence, allowing for a more granular mission execution and control; this was our choice and we’ll explain the reasons in detail later.
Status and telemetry sent by the flight controller via UART must be captured by the on-board computer software and sent to the Ground Control application

As it can be seen in the picture we selected a modular approach with an exported API that abstracts the flight controller details and a plug-in approach that maps the APIs to the actual commands supported by the flight controller.

As mentioned before, we chose an approach where the full mission is sent to the Mission Executor module which, in turn, sends each mission item individually to the flight controller using the APIs as opposed to a more traditional approach where the full mission would be sent to the flight controller.

Flight Control Service works as the proxy between applications and the flight controller. It abstracts the communication protocol so the applications don’t have to be aware of which flight controller is in use and verifies if an application requesting control over the UAAV is authorized to do so.

We made this design decision for a number of reasons:

Mission Executor can inspect the integrity of the full mission before sending it to the flight controller. For example, it can check the mission does not try to disarm the controller while in flight or try to take-off before arming.
With this approach a mission can contain certain steps — called here Mission Items — that are not flight related. For example, a Mission Item may launch an application to control a camera attached to the on-board computer or a Collision Avoidance application running in the on-board computer may take over control over the drone, allowing the Mission Executor to resume control once the collision is averted. These tasks would be relatively difficult to manage and control if the mission was fully pushed to the flight controller.
In general individual commands sent to the flight controller can be overridden if necessary without having to cancel the whole mission; if we were to send the whole mission to the flight controller the whole mission would have to be canceled and re-programmed from where it stopped, adding unnecessary complexity to the logic.
Activities that require the flight controller to be shut down and restarted — for example, automatic battery swap — would wipe the mission from the flight controller memory and require the remainder of the mission to be re-programmed; more than that, the mission itself would have to be modified to add steps like arming and taking off before being executed; the on-board computer would have to know if a Mission Item was interrupted mid-execution, etc. Having the mission managed by the Mission Executor itself greatly simplifies these tasks.
Fail-safe procedures for critical situations — loss of LTE, loss of GPS, etc. — are much easier to implement and activate.

The Ground Control Application

The Ground Control web-based application

We have our on-board software running, connected and communicating with the flight controller and ready to receive and execute missions. Now it’s time for a tool to send missions to the UAAV and receive telemetry from it. This tool is the Ground Control Application (GCA).

In our implementation we decided to use a web-based application instead of a standalone PC application; the main reasons for using a web-based approach were:

When the UAAV communicates with a web-based application it can request server certificates and authenticate the application; this is harder to do with a standalone applications since every installation of the application would have to also securely store the private key — used when the certificate was signed — in the local PC, which is a hard problem to resolve.
A web-based application can keep a centralized list of login/password credentials so pilots can only access the UAAVs they are authorized to fly; the same type of cloud-based authentication would be required for the standalone application
Using a web-based application with a publicly available URL and static IP address allows for additional checks from the UAAV regarding source and destination of the data; doing so with a standalone PC mostly likely connected to a network behind one or more NAT routers is much more difficult
With a web-based application whenever a bug is found it’s fixed in one place and immediately available to all users; the same is not possible for a standalone application. A serious security flaw could take days or weeks to be pushed to all affected installations.

We recognize UAAV operators use standalone GCAs like QGroundControl and MissionPlanner; we did some experiments by adding a module to QGroundControl so it could take advantage of our web-based implementation but didn’t have a chance to put it to extensive test.

The Communication Protocol

Arguably one can think of UAAVs connected to the mobile network as a class of Internet-of-Things (IoT) device:

The UAAV provides data about its state (altitude, heading, speed, GPS coordinates, etc.) and its environment (RF quality, neighboring cells, data throughput, etc.) to an authorized and authenticated destination
The UAAV receives and executes commands (arm, take-off, go to way point, etc.) from an authorized and authenticated source

It’s a little more sophisticated than controlling your thermostat remotely but the basic principle is the same.

Therefore, when trying to figure out how to communicate with the UAAV we decided to use a commercial MQTT broker — de facto standard protocol for IoT devices — for the following reasons:

MQTT brokers provide TLS mutual authentication between the broker and the IoT device
Mutual authentication is important to prevent hackers from impersonating UAAVs
MQTT brokers also provided mutual authentication for external servers consuming the data provided by the IoT device and sending data to it
Again, important to prevent attackers from impersonating an application server
Using an existing commercial MQTT broker would save us the effort of building our own server, including all the security aspects of it
MQTT brokers allow secure and encrypted bi-directional communication between the web-based application and the UAAV

Finally, we must decide how the data — telemetry and commands/missions — are going to be encoded. We selected JSON as the encoding format because it’s human readable — useful when you are debugging issues — and it’s fully supported by the Android framework and easy to encode/decode.

Support to external apps

Due to the decoupled nature of the software architecture it is quite obvious that the set of APIs defined for the Mission Executor can be also used by multiple apps simultaneously (e.g. Collision Avoidance).

We defined a relatively small set of abstract methods which map to the core flight functions an app would expect to execute, as well as callbacks — in the form of Listeners — for the application to receive telemetry information from the flight controller.

On top of that we also defined a set of methods that allow an external application to take over control of the drone and start using the APIs to fly it without conflicting with other apps trying to do the same.

To avoid rogue applications taking over the UAAV the methods use a system of permissions and priorities to control access to the API: if the app is not authorized it can’t take control of the UAAV; if an app with a higher priority requests control it’ll be granted until the app releases it; all other apps will be notified they lost control of the UAAV and any attempt to control it will be ignored.

So in very high level:

Mission Executor is running a mission and reaches a way point
Next Mission Item is to launch a Camera app that communicates with a fixed camera in the UAAV
The Camera app then takes over control — if authorized and with a higher permission than Mission Executor — and navigates the UAAV in a some flight pattern to take pictures or film
During the flight, controlled by the Camera app, the Collision Avoidance app detects an imminent collision; it than takes over control — if authorized and with higher priority than the Camera app — and averts the collision
Once the collision is averted the Collision Avoidance app releases control, which is then granted back to the Camera app
Once the Camera app is done it releases control which is then granted back to the Mission Executor which goes to the next Mission Item

Applications of course must be implemented with these characteristics in mind and must be able to handle control interruption accordingly.

Additional software

Besides the software needed for actually flying missions with the UAAV other auxiliary software can be installed to support operation, specially in BVLOS conditions. Such applications are:

Device management tools: suite of applications that allow updating the software in the device over-the-air, wherever the UAAV is located
Network monitoring tools: suite of applications that monitor the network quality and can provide valuable data for carriers to optimize the network for UAAVs

Fail-safe apps: suite of apps that take over control of the UAAV in case of failure. In our experiments we had 4 apps:
* GPS loss: take over control of the UAAV and land it; notify the last known position before loss
* LTE loss: monitors LTE coverage and GPS position. In case of LTE loss waits for 10 seconds, if UAAV does not go back into coverage within that time takes over control and returns to the last known position with coverage
* Collision avoidance: relies on sonars and lidars connected to the on-board computer and take over control in case it detects an imminent collision.
* Low battery monitor: uses incoming MAVLINK messages to keep track of battery levels reported by the flight controller. If battery goes below a certain threshold takes over control and land the UAAV.

In our experiments the low battery threshold was fixed and hard-coded but we think it could be made dynamic based on the battery depletion rate, UAAV altitude and UAAV landing speed.

Conclusion

After almost 2 years of development and testing the UAAV, under visual line-of-sight conditions, in several locations and under different weather scenarios — but always following the FAA safety rules for UAAV flight — we were able to reach a few important conclusions:

In the current conditions the LTE mobile network seems to be suitable for command and control of UAAVs; we did not see any adverse effect due to RF interference for the UAAV operation at the data rates under 2Mbps used for command and control (up-link data rates higher than 2 Mbps are known to interfere with ground users)
Redundancy is very important, both for the on-board computer and flight controller. We lost control over the UAAV multiple times due to application crashes, GPS loss and even failures in the flight controller (although infrequent).
GPS loss is catastrophic for the UAAV; once it start relying on accelerometers and gyros for orientation and stabilization after a loss of GPS it can’t keep stable flight autonomously — cannot even hover in place properly — and requires human intervention. Options here are:
* Improve the stabilization logic in the flight controller
* Use GPS signal from multiple GNS systems to avoid GPS loss
* Connect the GPS to the on-board computer and let it provide the position information to the flight controller; this way it can “fake” the GPS signal and prevent the flight controller to switch to a different mode of operation relying only on IMU, giving some time for the on-board computer to safely land the UAAV
* Use a downward facing flow camera and satellite maps downloaded over LTE to visually orient the UAAV and safely land. The proper maps to be downloaded can be found by using coarse positioning from the mobile network
High-speed/low latency communication is also important, specially for tracking the UAAV. Using a MQTT broker introduced delays that could be problematic in real-life applications
Be aware of the weather. Flying our UAAV in windy conditions was difficult, the flight controller had problems hovering in position. Rain, even light drizzle, affected the electronics and made autonomous flight almost impossible.