Companies such as KVS Technologies and Quantum Systems are leveraging artificial intelligence and machine learning techniques to deliver superior UAV-based inspection services. Meanwhile, Delair spin-off Alteia has created a software platform for implementing artificial intelligence and machine learning in visual data processing at enterprise level.
Developers are working to integrate artificial intelligence and machine learning tools across a range of physical industries. KVS Technologies is a Norwegian company that builds its own smart drones to inspect and monitor power grids. “Our first priority has been to introduce real-time object detection onboard our drones,” said Christoffer Apneseth, the company’s chief technology officer. “Prior to artificial intelligence, everything was based on GNSS coordinates,” he continued, “which are not always very accurate. Trying to get photographs of masts and pylons, we could find ourselves off by several meters in both the X and Y dimensions. That improves if you introduce RTK real-time corrections, but a mast can still appear to one side of the image, or with part of it cut off.
“With object detection,” Apneseth affirmed, “we can process the images in real time onboard the drone, with a trained machine-learning network to detect the top of the mast. We can use that to point the camera to exactly the right location, and get the mast spot-on, in the center of the image. This was our first real implementation of an AI-enabled function.”
BEYOND MANUAL DIGITAL INSPECTION
Visual data plays a central role when assessing the condition and likelihood of failure of key infrastructure, such as buildings, pipelines, electrical grids, solar arrays, dams, roads and bridges. “What we are focusing on is developing strategies that use machine learning to detect faults,” Apneseth disclosed. Visual assessment of digital images by human inspectors is step one in the training of a machine learning network. “We start by collecting visual data very consistently, from one inspection to the next, and from one mast to the next. We always keep the same distances, angles, etc. That hugely simplifies the training of these networks.”
KVS Technologies collects not only high-resolution images but also, simultaneously, 3D LiDAR scanning data, “because some things are easier to find in the 3D point cloud, and some things are easier to find in an image,” Apneseth observed. “And we can combine the two and color the points, classifying the point cloud, with different colors for ground, masts, buildings, etc. We are now developing the algorithm to find the optimal way of using LiDAR data and the photos in combination, for the best possible detection.”
A number of tools are available commercially to help people train machine learning networks. The techniques are fairly standard: human inspectors examine photographs and other imagery and then insert a marker when they see something of interest or concern. “If there’s a tree on the line, a skewed crossbar or a missing insulator,” Apneseth explained, “the inspector will create a ‘ticket’ with a fault category and mark the image. We can show that, along with other images, to a machine-learning network, and it will learn to detect the fault.”
All kinds of faults can be identified and placed into categories. Apneseth iterated: cracked or missing insulator disks, loose stabilizers, crooked crossbars, trees that are too near the power lines, even woodpeckers holes in the masts. On the other hand, to develop accurate algorithms, one needs to show it huge numbers of images. “That has been one of the challenges,” he said. “It’s easy to get a lot of images showing healthy conditions, but you don’t have so many images showing the faults that you are looking for.”
One solution has been to rig faults; that is to stage fault scenarios, sometimes on unused parts of the grid, and then capture imagery of those faults for training. Another strategy has been to use simulations. “We have invested quite heavily in a very realistic simulation environment,” Apneseth noted, “where we can simulate the geography, terrain, trees, roads. We build up digital twins of the network components, masts, power lines, and we also have digital twins of the sensors. The camera and the laser scanner that we have onboard are also modeled, so we can say the drone is flying at 10 meters per second at 20 meters above the power line when the camera fires, and we get an image very similar to what you would get in real life. We can create endless variations in terms of different light, snow on the ground, and of course we can generate the different faults, removing random insulators from masts, inserting a broken crossbar. Like this, we get a huge data set of fault scenarios that we can combine with actual images of real faults.”
PREDICTING FAILURE
Another important capability is being able to compare images of the same elements over time. “For an inspector looking at pictures manually,” Apneseth observed, “it’s not so easy to remember images of the same objects from, say, a year before, to see trends and then try to predict something. With machine learning, the algorithm can follow the development of a fault over time, anticipate problems before they become critical.
“It is perhaps not surprising to find that the most severe faults are also the easiest to detect,” he stated, “things like serious physical damage or a tree on a line. These are faults that we think can be detected in real time onboard the drone itself. We are working on algorithms that will process all the images as they are captured by our drones in operation. These are then uploaded to our cloud system, where we can further process the images.”
GOOD BUSINESS
“It has been non-trivial for KVS Technologies to develop all the capabilities and technologies that allow us to do this,” Apneseth said. “But we believe that we work in a field that will benefit greatly in terms of safety, environmental impact, data quality and economy.”
The company’s clients agree. Agder Energi Nett is a large distribution system operator with 4,000 kilometers of high-voltage power lines in southern Norway. “KVS Technologies won a tender against several other bidders offering similar services for inspection and laser scanning,” Manager for Maintenance and Analysis Håkon Skavikmo said. “Most of the other candidates use helicopters, but we placed a strong emphasis on levels of noise pollution and CO2 emissions, as well as the quality of the data collected.”
Even just 10 years ago, Agder Energi Nett crews were still walking along power lines to do inspections. “This was very costly and time-consuming,” Skavikmo recalled. “Today we have significantly lower costs and we save time. KVS Technologies sends us data from their drone inspections on a daily basis, and this includes reports on issues that need prompt action. We get good images and laser scanning data with a high point density per square meter, which means higher quality. We expect to continue to use drones for data collection and analysis as long as they remain competitive in terms of quality and price.”
BETTER DRONES
Building its own UAVs wasn’t KVS Technologies’ plan from the start. “We pivoted,” Apneseth divulged. “Our strategy initially was to use third-party drones and to integrate our own command and control systems, communications and payloads. We had a fixed-wing and we had a multicopter, but based on our experience in initial operations, we found that we faced more and more challenges with operating needs.”
The point of failure, he said, was motors. “Very few companies fly like we do. We may fly a drone for six hours a day, and it’s not kind of hovering around a windmill, slowly taking pictures, around a building or flying in a flat pattern above a field. We have to continuously follow the terrain, and in Norway it’s very hilly, with a lot of climbing and descending and sharp turns. We were just wrecking the motors on our commercial systems, so we decided in 2020 that if we wanted to be in control of our destiny, ready to scale internationally with a reliable drone, we needed to do this ourselves, and that’s what we did.”
The first KVS Technologies in-house-engineered drone is a coax that’s heavy on redundancy—here, rotors are mounted atop one another on concentric shafts, turning in opposite directions. “It’s more resilient than the third-party systems,” Apneseth noted, “meaning better redundancy options for the power system, better redundancy options for the GNSS, better redundancy on communication channels, and so forth. Our system is kind of physically over-dimensioned, compared to some similar drones. It’s got a cruising speed of 10 meters per second and a five-kilogram carrying capacity.”
The KVS Technologies UAV is not yet for sale, but it works well within the company’s overall concept of operation, with supporting systems, cloud platform and data analytics, and customer applications all included.
SAFE SITES
A large proportion of workplace fatalities occur on construction sites. Machine learning strategies like those being developed by KVS Technologies can be extremely valuable in such settings; just as algorithms can be trained to recognize and signal faults along power lines, they can also be taught to recognize safety hazards, based on visual data coming out of construction sites.
Quantum Systems is one company using artificial intelligence and machine learning techniques that are entirely applicable to safety management in the construction industry. The Munich-based company makes the fixed-wing VTOL Vector UAV, as well as its flagship Trinity F90+, with its 2.4-meter wingspan and 100-kilometer flight range. For starters, said Florian Seibel, the company’s managing director, “our UAVs use artificial intelligence-assisted navigation, enabling a series of features, like visual-based return, e.g. in a GPS-denied environment, or automatic target detection.”
Using high-resolution visual images, trained machine learning networks can detect common safety risks. Highly relevance in today’s construction sector is the ability to recognize people, their specific features and even what they are doing. “Artificial intelligence enables the tracking of objects using object detection,” Seibel said. “This can include vehicles, machinery, but also persons.” Trained machine learning algorithms can detect missing hard hats or gloves, essentially serving as an extra pair of eyes to help create a positive safety culture on a construction site.
Seibel said Quantum Systems works with neural network software similar to YOLOv5. This is a family of pre-trained object-detection architectures and models. The company integrates hardware like NVIDIA’s Jetson TX2 series computer-on-a-module, enabling onboard edge computing. “With this, we can control a gimbal with a camera, to follow the detected person or object in real time,” Seibel continued. “This is a robust tracking function, with rapid re-enabling of tracking if tracking is lost.”
DIGITIZE IT
For a company with physical assets that need to be inspected and maintained, machine learning, as applied to visual data, is clearly a game-changer. But, more and more, artificial intelligence and machine learning are also seen as key assets in the broader drive towards industrial digitization. Alteia, a spin-off of Delair, the French drone maker, has created a platform enabling large companies to easily implement artificial intelligence and machine learning solutions.
Alteia is not a UAV company per se, but what it’s doing is certainly relevant to anyone using UAV-derived imagery. “We are a software provider; we don’t deploy any drones in the field,” said Baptiste Tripard, Alteia’s chief marketing officer. “Our platform accesses data provided by our clients, and it’s not necessarily drone-based. We can include video streams from construction sites, where you have cameras on top of cranes.” Other capabilities include smartphone captures, camera streams in manufacturing plants, helicopters, ground sensors, aircraft, drones and satellites—or, as Tripard put it, “any type of visual support, but also standard databases or Internet-of-things data. Our tool set enables people to leverage visual information across all their organization, build artificial intelligence models on top of it and then deploy those applications.”
Industrial digitization is about getting data and processing networks to converge across all departments and geographical boundaries of an organization. Enedis, a large utility operator in France, is using the Alteia platform, Tripard said, “as a baseline to deploy all of their artificial intelligence applications that are related to their network. This becomes a single source of truth for all their visual data that is collected around the organization. They can then use the different features that we offer, either to process information with our pre-built, artificial intelligence applications, or they can develop their own applications using our software development kit and application programming interfaces.”
Alteia’s clients are mainly large enterprises, like BASF, the German multinational chemical company and the largest chemical producer in the world. “They’ve integrated our platform as a base,” Tripard said, “on top of which they are developing their entire artificial intelligence strategy around visual data. Artificial intelligence can be many different things. It is certainly a new type of business process, and as such it requires new management methods, and this needs to happen at the enterprise level, for a company to ingest newly created processes. Operating drones is also a business process. Companies will have fleet management software and ways to program drones. All of that needs to be taken into the organization.”
A PLATFORM IN THREE LAYERS
Alteia platform works in three layers, Tripard explained. “First, we have what we call a unified data model, across all the different data sources. You have a baseline, and all the data speak the same language.”
The second layer involves the training of machine learning models, similar to that done by KVS Technologies and Quantum Systems. “People doing digital inspection,” Tripard said, “visualize their content on a user interface and then manually annotate the images. Our platform has a library of pre-built machine learning models that are already trained for different types of equipment and features; the model already kind of knows what it needs to detect, so you’re not starting from a blank page.” Users can further tune the pre-built models to suit their own needs, by validating or invalidating the models’ predictions.
“So there is this phase of incremental learning,” Tripard said, “where people can use our interface to validate or invalidate model predictions, and this is then fed back into the model. In this way you can push performance to whatever level of accuracy you want. You can easily train a model up from 80% accuracy to 90% or 95%, depending on your business needs.”
The third layer involves when a detected fault leads to action on the ground. “Once you have your model working,” Tripard added, “detecting pipeline rust with 90% accuracy, what do you do with that? It has to be connected to your existing IT ecosystem, so you can create a work order to get out there and solve the rust problem. So this last piece is connecting the machine learning process to other processes in the organization to generate a business action.”
UAVS ARE PART OF THE PROCESS
“Drones need to become smarter,” Tripard said. “What we see as a trend is more and more edge computing, where machine-learning algorithms are deployed at the edge, at drone level, allowing you to make real-time decisions.” This is the approach already alluded to by both KVS Technologies and Quantum Systems. “Let’s imagine you’re inspecting a pipeline; instead of sending imagery to the cloud and running a crack detection algorithm there, you can deploy the algorithm in the drone itself, at the edge. The drone only pushes its prediction to the cloud, if it’s seen a crack or not. That, to me, is going to be the key to the scalability of the whole model, having the ability to do some of the processing at the edge in order to avoid unnecessary bandwidth problems with all the data transfers.”
Artificial intelligence and machine learning are making a place for themselves in the world of infrastructure inspection, and are becoming ever-more-closely linked to UAV operations. UAVs, in turn, if they’re smart, also appear to be here to stay, Apneseth of KVS Technologies said. “Our own power line inspection market is already there, and, while it is still served primarily by helicopters, every customer we talk to believes they will be using drones in the future. They can see the obvious benefits in terms of safety, environmental footprint, efficiency and data quality.”
Remaining UAV regulatory issues, Apneseth said, are likely to be resolved soon. “We will have systems that are certified or have gone through design verification processes, so that it will be possible to roll this out in all countries, in a harmonized regulatory environment. That will be a key enabler, making it much easier for companies such as ourselves to operate globally, with the same drones and the same operational models, and all of that will really support more widespread adoption.
