Swarms for Entertainment

When Intel put together its first drone light show, the Drone 100, a few years back, the team had to manually create the animations with each drone in each timeframe, as well as run through a few iterations of the simulator to prepare to fly the choreography in time with a live orchestra, said Natalie Cheung, general manager of Intel Drone Light Shows. During that first show near Hamburg, Germany, 100 drones painted 3-D shapes and messages in the sky, setting a Guinness World Record for most UAS airborne at the same time.

Since then, these small, lightweight drones (at about 280 grams they weigh less than a volleyball) have completed Drone 100 shows all over the world and have entertained live audiences at Walt Disney World as well as the Super Bowl, and Coachella. Now the onsite team, which is made up of four or five people instead of the 16 to 20 required in the beginning, can complete the animation and setup for the actual performance much faster. Shows that once took a week and a half to prepare for now only take about five days.

The Intel team learned a lot from that first Drone 100, leading the company to create a more dynamic, automatic system designed to reduce the number of potential failures, Cheung said. The software Intel has developed for the Shooting Star drones helps them understand how to better fly their UAS; it’s no longer based on a person controlling the drones, but on the software and the algorithms behind it.

While the shows are now much easier to execute, they still come with their own set of challenges. Each site is different, Cheung said, so the team must conduct a site survey before they fly. Intel drone fleets have entertained audiences in several countries—and each country has its own regulations regarding UAS flight. Understanding and working within the regulations is key. It’s also important for the team to understand where the audience is and how close they’ll be to the fleet to ensure the drones are deployed at a safe distance.

Before every show, which now can feature up to 300 drones, the Intel team spends time testing the drones to make sure they’re ready for the live performance, Cheung said. These rehearsals usually take place during the early morning hours, around 2 a.m. or so, to help ensure potential audience members don’t accidently get a sneak peek of the show. During the rehearsals, the team checks for GPS quality and communication back to the system controlling the dancing drones. They typically fly 10 to 15 UAS in a simple animation to make sure what they see on the simulator is what audiences will see in real life.

The drones are pre-programmed with the animation before the flight, Cheung said, and only need to communicate back to the control center. The UAS understand what their flight path is and are configured to keep a distance of 1.5 meters at all times to avoid colliding.

“We want to make sure our animation looks good for the audience but we also want to make sure the drones never go beyond our two layer geofence,” Cheung said. “If a drone hits one geofence, it’s redirected back to the original pathway. If a drone ever hits the second geofence, its motors are shut down so it can land safely.”

The UAS are made out of foam and the propellers are encased in a prop cage—all to ensure safety, Cheung said. An LED light is their only payload, but that’s all these entertainment drones need. Cameras and other sensors would only decrease their flight time, which is about 20 minutes. Each has its own personality that comes through during the performances, which is what makes developing them so much fun for the team at Intel.

Raffaello D’Andrea, professor of control systems at the Swiss Federal Institute of Technology and founder of Verity Studios, also uses drone swarms to entertain large audiences. He’s conducted research in his lab, the Flying Machine Arena, for more than 10 years and is always looking for ways to improve the technology that guides these vehicles.

A few years ago, D’Andrea created SPARKED, a short film that explored what drones could do in a theatrical setting. ETH Zurich, Verity Studios and Cirque du Soleil collaborated on the film, with Cirque du Soleil deciding to transfer the concept over to Paramour, a major Broadway production.

The Paramour drones have been performing eight shows a week for about a year—alongside and with the same regularity as human actors, D’Andrea said. The drones, outfitted as colorful lampshades, come to life on the stage and are guided by Verity’s proprietary localization system, which replaces cameras. The redundant UAS rely on fail-safe algorithms for each live performance. They listen in on the positioning system to determine where they are, so there’s no need for them to communicate with each other as they light up the Broadway stage.

As with most technology in this industry, D’Andrea said the performing drone fleets will only continue to become more robust and more mainstream.

“This is still very much in its infancy,” he said. “Performing drones inhabit a theatrical space like nothing we’ve seen before, triggering irrepressible emotions as they invade the spectators’ personal space and perception. I fully expect that small, programmable, flying lights will become a staple at all major live events in the near future.”

Other Commercial Applications

One of the reasons Intel started creating drone fleets for entertainment was to show the possibilities they represent for other applications, such as SAR and inspections, Cheung said. There are seemingly endless opportunities, making this an area many scientists are interested in exploring.

Vito Trianni is among them. He’s a researcher at the Institute of Cognitive Sciences and Technologies (ISTC-CNR) in Italy and is the project lead for the Swarm Robotics for Agricultural Applications experiment, or SAGA. The group plans to demonstrate how swarm robotics principles apply to agriculture, targeting decentralized monitoring/mapping scenarios, and implementing a use case for detecting and mapping weeds in a field via swarms of small UAS.

So far, all the research they’ve done has been through computer simulations in the lab, but they plan to test a small group of three or four drones this summer, and then grow from there. Eventually, Trianni would like to deploy 100 micro drones at a time. The drones will fly using algorithms for collective behaviors, and will exploit a common control link to issue radio signals to other members of the swarm.

“We are implementing protocols for broadcasting messages, so even if a communication radius is limited, information sent by one drone can be sent to the rest of the group so they’re all aligned and have the same knowledge,” he said. “Drones can change the strategy and follow traces left by other drones, simulating behaviors of colonies like bees or ants.”

The idea is to keep the drones small, Trianni said, and to outfit them with inexpensive cameras. This decreases the size of the hardware and the cost of the drone. The challenge is demonstrating a swarm of smaller drones can cooperate and achieve the same precision as one expensive high-end drone.

His drones weigh a little less than 5 kilograms, Trianni said. He’d like that to come down to closer to 2 kilograms.

“If we have machines that are not individually powerful and not individually expensive, we can deploy more of them, and if we lose some it’s not a big deal from an economical point of view,” he said. “The systems will work with more or less the same performance as a more expensive machine.”

Ahmed Hussein, an assistant lecturer and PhD Student at the Universidad Carlos III de Madrid (UC3M) in Spain, has advised startup Drone Hopper on the UAS system they’re developing for farming applications and firefighting. Eventually, they’d like to deploy the drones as collaborative systems. Like the group in Italy, they’ve only completed simulations so far and know it might be awhile before swarms can be deployed in the field, partly because of the regulations that keep operators from flying more than one UAS.

The Drone Hopper UAS also will be modeled after bio-inspired behaviors, Hussein said, whether there’s a queen bee the others follow or they act more like ants working together as soldiers. The systems will be simpler than the high-end options you see on the market today, but because each drone can carry a different sensor and share data over a wireless network, they’ll still have the ability to gather the same information.

“Our strategy is to start including some kind of coordination intelligence in the drones, always subordinated to the human pilot, as a starting point to gain confidence and progress on a future swarm-controlled fleet operation on agriculture and firefighting missions,” Drone Hopper CEO Pablo Flores Peña said. “I believe we will not need a very complex logic; of course you can make it as sophisticated as you can think, but for our operation I see no major issue in terms of feasibility. However, the hard part is how to ensure and demonstrate safety.”

Collaboration among multiple drones also can improve mapping capabilities, Smith said. Today, single UAS typically fly a lawn mower pattern to efficiently cover the entire area they’re mapping. While this is a great time saver over traditional, non-UAS methods, collaborative mapping missions could save companies even more time and money.

“Instead of starting at one corner then doing the lawn mower pattern, what if the drone could start in the center of the area or another corner, and instead of flying one drone to cover the entire area you send two, or three or four. They could go to all four corners and work together to fly inward,” podcast host Smith said. “That would cut down on the time it takes to cover that area by 75%, and would also cut down on the amount of battery changes needed during the mission. It would greatly increase the efficiencies of the entire operation.”

Communication Challenges

Regulations limiting the ability of one pilot to deploy multiple drones and to fly beyond visual line of sight (BVLOS) without a waiver are among the challenges researchers and manufacturers face. Another is finding a way to optimally manage the swarming behaviors, said Souma Chowdhury, an assistant professor in the Department of Mechanical and Aerospace Engineering at the University at Buffalo. That comes down to understanding the principles of swarming behaviors in nature, which again means looking at insect colonies and birds and then modeling that behavior in a mathematical way.

Swarms won’t be controlled by centralized agents, Chowdhury said. Instead, the simple drones will collectively exhibit high levels of intelligence that go well beyond what a single computer can do. But because they’re simple, they’ll have limited computing capabilities as individuals. There will be no algorithms to enable them to make split second decisions when they need to avoid an obstacle or have other problems they need to solve. That’s why communication becomes important when performing inspection missions or other activities that might involve obstacles or changes to the flight plan.

When these drones are flying 50 miles off shore by themselves—to complete an oil rig inspection, for example—they won’t be on a high bandwidth connection or have the ability to access a Wi-Fi network, Chowdhury said. They’ll only send a few kilobytes of data to each other at a time—so that data must be compact and relevant.

“They need to be able to share small bits of optimized information to communicate. They can’t send images or video,” he said. “They’re on an ad-hoc network built amongst themselves and they’re flying several hundred meters away from each other. We have to make sure they’re communicating small bits of data, but very important data. Some of the fundamental questions we’re trying to answer are how to make communication compact, how to model the behaviors of swarms in nature and how to make the computer lightweight enough to accomplish the mission in the shortest amount of time.”

Rather than sending all the information every drone collects as raw data, the important data is compressed and time stamped before it’s passed on, Chowdhury said. That makes the data lightweight, and means the other drones only receive intelligent, actionable information. The team is also developing network protocols and new ways to configure radios on UAS to help alleviate the bandwidth challenge.

Javier Alonso-Mora, assistant professor at the Delft Center for Systems and Control at Delft University of Technology, was part of a team at MIT that developed a decentralized control algorithm for drone swarms in 2016. The planning algorithm can handle stationary and moving obstacles with less computation than if it were centralized.

There’s no need for every robot to send a complete map of the obstacle-free space around them to every drone in the swarm. Instead, they’ll only share these maps with their neighbors, Alonso-Mora said. From there, the drones can determine where each neighbor’s map intersects with their own, and then pass that on to the next neighbor. Less information is shared, but each drone still knows where it can safely fly without risk of a collision.

The challenge becomes creating a robust, reliable communication system and determining how much information the drones need to pass on and when, and how they reason while out in the environment, especially if they’re faced with an obstacle, Alonso-Mora said.

“They can communicate with their neighbors to reach a consensus and decide how to collectively move,” he said. “They don’t need a centralized unit. If the centralized unit fails the whole system will fail.”

It’s also important to keep in mind that teams of robots or swarms of drones aren’t necessarily going to be right next to each other during the entire mission; chances are, some will go out of communication range from time to time, said Sanjiv Singh, a research professor at Carnegie Mellon University’s Robotics Institute and founder of the startup drone company Near Earth Autonomy.

Singh sees a loose infrastructure where every system in the swarm is assigned a specific task. There’s no team leader, and every drone constantly communicates their location, with the others listening if they can. One drone may go out of listening range for a while, but will return once its task is complete. When part of the mission fails or there’s an unexpected change of plans, the systems will come together and re-coordinate.

“The main challenge is dealing with complex tasks where things are changing, not going as expected or even failing,” he said. “What we want in these teams is to not have a single point of failure. Responding to the world when it’s not being cooperative is the biggest challenge. Let’s say a battery dies in one of the robots unexpectedly. There needs to be a way for the team to reorganize and complete the entire task, even though the world didn’t cooperate.”

The Future

On the entertainment side, Intel’s Cheung would like to see a world where drone light shows aren’t just for large, major events like the Super Bowl. Dancing drones could serve as the entertainment for conferences or as a way to engage fans after professional and local sporting events. Companies could use the shows to brand the night sky with their logo, or as a digital billboard to advertise their products.

“I see a lot of potential with drone fleets,” she said. “We can see the drone light shows going from 300 to 500 to even 1,000 or more. We see a future where we can fly in different environments, not just in an outside stadium but maybe even inside.”

Cheung also expects entertainment drones to become more durable in the future, have the ability to fly for longer periods of time and to wow audiences with even more entertaining shows.

And then there are the slightly more complex applications that require some type of communication among the drones, such as oil spill monitoring, infrastructure inspection and crop assessment. Swarms have their place in these industries as well, and Smith said companies could begin adopting them over the next few years. Of course, for swarms to become mainstream, they need to be easy to implement and use, and he expects these initial solutions to come from one or more of the major players in the industry.

“There’s a spectrum of possibilities. The idea of robots cooperating with each other seems to be very compelling,” Singh said. “What you get, ideally, are simpler systems that are able to team with each other. People are always teaming together. We’re not individual agents. We’re always helping each other or have networks or hierarchies. If you have a team of robots that can talk and plan together, it will make them more efficient doing what they’re already doing.”