Increased deployment and enhanced software relieve burdens on data capture and centers.

As drones are increasingly deployed for Intelligence, Surveillance and Reconnaissance (ISR) missions, military and law enforcement clients are adapting more sophisticated software to manage the overload of data sophisticated reporting is generating.

Not only is latency between real-time events and video data capture shrinking, but ISR software now allows operators to correct atmospheric distortion of ground images so targets can be identified more accurately. Software using AI also enables users to improve reliability of video images and combine data from multiple sources so they can be analyzed in a single display.

Evolving ISR software is attempting to address the deluge of data that is overwhelming command center operators in the defense and public safety fields as they try to analyze thousands of hours of video footage drones are collecting every year.

“You can deploy so many sensors now that you have a complete overload of your operators,” said Scott Richardson, vice president of product for CarteNav, a firm based in Halifax, Canada, that develops ISR mission software. “That is definitely one of the problem states.”

IMPROVING LATENCY ON ISR SYSTEMS

Haivision, a Montreal-based company providing video streaming technology, produces video encoders that take the raw data from the sensors on unmanned aerial vehicles, compresses it in real time and sends it to command center operators. Operators normally need to scan the video footage, and they also need to analyze the metadata that ISR applications provide, such as the latitude, speed and positioning of targets.

“It can be a lot of data and so one of the challenges is that when you’re dealing with satellite or radio communication, some of those links have very little bandwidth available,” said Mark Hochler, marketing director, products and solutions, for Haivision.

To process that data efficiently, Haivision’s software system, Kraken, manipulates it by filtering out unnecessary information and decimating it—having it appear only in sporadic frames within the video. “That uses up less data and it won’t overwhelm your bandwidth and it won’t overwhelm your system at the back of the command center,” Hochler said.

Another goal for defense clients analyzing ISR data is achieving a low bitrate—the number of bits processed in a given unit of time. “It’s a very small amount of room that they have to squeeze the data and the video,” Hochler said.

CarteNav had also developed a number of tools that enables track filtering and track alerting. The software features allow operators to identify desired targets that meet conditions they’re searching for so they can declutter the data stream.

For instance, if a command center operator is looking for ships that are moving at a certain speed but receives a video image of hundreds of ships, the software can eliminate those that aren’t moving,” Richardson said. The filtering tool is available on CarteNav’s two main software products: the AIMS-ISR and AIMS-CR.

In ISR and public safety applications, reducing the latency as low as possible is another critical objective, which enables operators to make split-second decisions based on the data they receive. Haivision’s Makito encoder now can bring latency down to 50 milliseconds.

“The time it takes to process the video, to encode it and to stream it—if it takes over a second, that can be way too long,” Hochler said. “Every millisecond is critical.”

CORRECTING DISTORTED VIDEO IMAGERY

When UAVs fly ISR missions, data transmitted on video can often be difficult to analyze because of blurry atmospheric conditions. ISR software can now improve the video and sensor data quality so footage is easier to interpret.

One company specializing in software that corrects video distortions is Cubic Mission and Performance Solutions, a San Diego firm that provides networked ISR capabilities for defense, intelligence, security and commercial missions.

“As we like to say, ‘We enable people to make life-and-death decisions,’” said Randy Fields, vice president and general manager for Cubic Digital Intelligence, a division of the company that specializes in ISR software.

A UAV, for example, may capture images that show clouds blocking targets a defense client is tracking on the ground. Using AI and machine learning, Cubic’s MotionDSP software allows “people to basically turn the clouds on and off,” see what is beneath them and decide whether to pursue that target, Fields said.

Besides correcting atmospheric conditions, the software also removes shadows next to buildings so operators can identify the location of targets on the ground.

“Maybe there’s a situation happening with special forces or maybe there’s an issue that’s taking place in a law enforcement situation,” Fields added. “Our software enables us to remove shadows and enhance the fidelity of the data so that the data that’s coming from the ISR is the best decision-making quality data that’s possible.”

The software retains the raw data, keeping the “chain of custody” intact, because the filters are added as targeting occurs in real-time. This allows clients who need the untouched images to access them, said Jimmy Zinter, senior director of ISR and digital solutions for Cubic Digital Intelligence.

As technology elements become smaller and smaller, the challenges UAVs face is that they are more impacted by wind and jittery movement, which can also blur video images of targets. Cubic’s stabilization filters, however, allow operators to remove distortions caused by the drones’ motion on the video so operators can more clearly identify targets.

ISR operators can also use the MotionDSP software to overlay images of ground conditions that may have been damaged during natural disasters such as hurricanes or wildfires. The photos of the terrain are juxtaposed over existing land coordinates, such as Google Maps, and are geo-rectified, which means it exactly aligns with the existing land conditions.

“First responders or the Department of Defense could use this,” Fields explained. “You go in with what just happened with Hurricane Ian, and say, ‘OK, the last satellite shot we had of this area was 24 hours before the hurricane. Now, we flew a new map and we can overlay the latest data right on top of the old data so that way we can make decisions using the latest and greatest data that is geo-rectified.’”

Cubic’s software suite was introduced about two years ago and is commercially available to any user. “This was not purpose built necessarily for one customer,” Zinter said. “It’s built because it was intended to solve a problem that we’ve seen in slow motion video, regardless of where they sit, for a very long time.”

FROM AI TO AUGMENTED REALITY

As ISR software evolves, it is becoming more dependent on machine learning and AI to add more sophisticated capabilities for analyzing data and to help operators sift through the deluge of video footage sensors are generating.

AI enables ISR software to classify objects, such as tracking patterns of ship identification on video footage captured by drones. “If we can actually have it so when the camera looks at it, there’s AI determining if this is a fishing vessel or a jet ski, that saves the operator a lot of time and a lot of manual work and can help you identify the things you want to find,” Richardson of CarteNav said.

What AI also brings is the ability—beyond detection and classification—to fuse data from multiple sensors into a single screen, Richardson said. AI-enhanced software could enable operators to merge a range of data, including sensor positions, weather, tide conditions and known objects, together to analyze in one image.

CarteNav is working with partners that develop AI technology to incorporate the technology into its software products. “It would just be essentially another sensor to us,” Richardson said. “We feed it information and get returns back and then do whatever customers have asked us to do with it.”

Cubic has already integrated AI and machine learning into its software, adding the capability to correct hazy video footage with the click of a button. Applying AI improves the quality of detection of objects command operators are tracking on the ground by 40%.

“The ultimate challenge is to find out what’s going on and get the ground truth,” Zinter said. “And this allows them to get that ground truth to execute their mission.”

Another feature Cubic has added to its software is augmented reality, which enables operators to overlay video on top of a geo-rectified still image. The software allows operators to home in on a specific target within the video display, while also presenting the context of the surrounding environment on the still photo.

“That helps you determine things that you may otherwise not be able to see just from the video,” Zinter said. “It allows you to see where to go so you don’t follow something the wrong way. It becomes a very powerful tool for our analysts to use and really allows users to see not only the video but what’s around it.”