The critical overall requirements for unmanned aerial vehicles (UAVs) operating in the energy sector are, naturally, reliability and productivity. Reliability takes on added significance in sectors that represent millions of dollars of investment. No margin for error can be tolerated in such complex environments. The distance from an in-flight UAV to the industrial asset that it is observing or inspecting obviously has critical importance for safety, data precision and cost-effectiveness.
The advantages that UAVs offer, on the other hand, are abundant and attractive: more effective inspection, surveillance services and asset management processes, at costs that are a fraction of traditional methods such as helicopters or deploying an army of inspectors on foot.
As always, balancing device size, weight and power (SWaP) with cost constitutes a significant challenge. But savings in any of these factors cannot come at the expense of precision. Data gathered on infrastructure maintenance needs is only useful if it can be quickly and easily geo-referenced, with maintenance crews efficiently dispatched. For this, precise position and orientation of the inspection device must be provided at all times.
ROLES PLAYED
An early provider of UAVs specialized to the energy sector, the Belgian company Airobot first produced an anti-collision system in 2015 to enable a drone to carry out close inspections. Lessons learned in the early going informed current offerings of flying robots equipped with artificial intelligence (AI) for automated, accurate data collection.
The first mission was to gather data concerning defects or damage to large outdoor structures. The Ranger was installed aboard a drone employed to take photographs of high surfaces and inaccessible areas. It enabled safe operation and furnished further data, which in turn supported calculation of the size of the photographed defects.
The Ranger product kept the drone at a set distance, usually no less than 3 meters from any structure. The remote pilot could control up and down movement of the drone, but not approach any closer. The product saw early use in several industrial sectors with large outdoor installations that required close monitoring, including oil refineries and storage silos.
Airobot’s technology employed a Septentrio AsteRx-m UAS GNSS module to ensure accurate and robust location information when executing waypoint flying. The sensing technology, a combination of LiDAR and ultrasound, enabled the UAV to detect all obstacles in its surroudings, so it would not collide with them. These combined technologies permitted safe and efficient horizontal flying in professional working environments. Ranger eliminated the need to hire and deploy additional expensive equipment to safely and accurately inspect or survey inaccessible areas.
The company received a special prize for this work from the Galileo Masters European Satellite Navigation Competition: “most innovative use of high-precision GNSS positioning in its project: ‘UAV Flight Path Learning through GNSS.’”
The GNSS receiver was designed to focus specifically on providing accurate and reliable positioning in challenging conditions where standard GNSS receivers could not always provide trusted accurate positions. Airobot enhanced the product by incorporating real-time kinematic (RTK) GNSS equipment and procedure, but this iteration was “too early for the market” back in 2015, recalled Airobot Managing Director, Sales and Business Development Jan Leyssens. Nevertheless, the company persisted in its efforts to increase the accuracy of drone location on its missions.
“If you take a picture, you need to know exactly where the picture is taken and be able to calculate the size of the detected defects. Otherwise the photo is not worth anything,” Leyssens said. “We did add-ons to existing drones, then decided we needed to do a full-on integration, building a motherboard, an Aircore.”
Today, the Aircore rides onboard Airobot’s Mapper and Phantom 4 UAVs, designed for accurate data collection and featuring a multi-constellation RTK/PPK GNSS receiver supporting GPS, GLONASS, BeiDou and Galileo to deliver centimeter-accurate results in all situations. A 20.1MP Sony UMC-R10C camera with an Exmor APS-C HD CMOS sensor takes high-quality images with a high ground resolution, suitable for demanding applications.
Another company, Delair, looks at the energy sector from the point of view of at least two different types of industries: utilities and oil and gas. For both, drone and data solutions make infrastructure planning and inspections faster, safer and more accurate by helping create digital models and maps to analyze. “UAVs offer more effective inspection and surveillance services that cost a fraction of traditional methods such as helicopters or deploying an army of inspectors on foot,” said Paolo Guzzini, product manager, power & utilities, Delair.
Founded in France in 2011, now with offices around the world, Delair enables its customers to create “digital twins” of their assets and properties.
In just one example of energy sector use, Delair drones map power line corridors in remote areas. Inspection of these lines is essential for electric grid reliability and resilience, to guard against vegetation encroachment and damage among other hazards. Automating the process with beyond visual line of sight (BVLOS) drones and software analytics with machine learning reduces costs and improves inspection reliability.
A typical inspection gathers photographic and LiDAR data. Drones are programmed with a predetermined flight plan, capture comprehensive data and deliver processed results within a day, at a cost far below that of a helicopter or ground crew. In one instance the drone flew 70 kilometers carrying a 21.4 megapixel photo camera, capturing a 200-meter-wide corridor at 0.15 meters vertical and 0.2 meters horizontal accuracy. A LiDAR camera captured a corridor of similar width at an achieved accuracy of 4 centimeters horizontally and 2 centimeters vertically. Delair’s post processing software classified the LiDAR point clouds with 85 to 90 percent reliability to identify potential vegetation encroachments.
TECHNOLOGIES INTEGRATED
The Airobot Mapper full solution carries an onboard flight controller, two real-time processors for collision avoidance, a compute modeler, wireless link and a Septentrio RTK GNSS receiver—all aboard an AceCore Zoe platform from the Netherlands, which can fly in high winds and some rain. This enables data collection and geocollection of data. One new application is that of inspection of large solar parks, where Airobot also prototyped a processing chain with AI to detect dirty panels.
Solar panels are naturally subject to bird droppings, which cause the panels to heat up at that location, leading to damage. If spots can be detected early and regularly, and personnel dispatch to clean them, the damage and replacement costs to expensive equipment can be avoided. The drones, equipped with hyperspectral cameras (even thermal cameras are not sufficient; once they detect a hot spot, the damage is already done) make such frequent and regular inspection feasible.
For infrastructure such as wind turbines, the company combines RTK GNSS with its Ranger collision avoidance technology to accurately steer the drone in a very stable way. Every photo taken can be accurately georeferenced, to make a reliable 3D model of the full installation, and to spot vulnerabilities.
Delair hardware platforms are equipped with either radio communication links or cellular connectivity to enable longer-range operations, including BVLOS. To capture the necessary data, the drones (depending on the model) house sophisticated imaging equipment: RGB or multispectral cameras; LiDAR sensors for laser mapping and surveying; and advanced geopositioning technology to enable centimeter-level accuracy for precision mapping. The company also offers an open-payload model, allowing users to add custom sensors or other equipment. And, of course, a long-life power supply is required to support flights of up to an hour.
As important as the hardware, Guzzini said, is the software solution to manage and analyze the collected imagery and data, including photogrammetry software and industry-specific analytics for identifying issues such as vegetation encroachment, pipe leaks and other safety hazards. The company’s software offering is based on its foundation delair.ai platform, a cloud-based platform to manage, process, view, analyze and collaborate around aerial data. “It is the key to unlocking the power of drone-collected data,” Guzzini said. “With it, we can quickly access deliverables that photogrammetry and analytics tools generate, such as 3D point clouds, digital surface models (DSM), digital terrain model (DTM) and orthomosaic, as well as specific images and data on the types of assets the clients are monitoring.”
The Spanish company GeoNumerics supplies the mapping software muscle required by drone operators to complete their missions. “From experience with our partners and customers, we know that they employ GNSS receivers, inertial measurement units (IMUs) and magnetometers for navigation,” reported Chief Scientist Ismael Colomina. Gathering data for analysis may require a variety of sensors, but Colomina is principally concerned with post-mission trajectory determination for geodesy, photogrammetry, and sensor orientation and calibration for precisely georeferencing of the collected data. “Less frequently, we find they are using baro-altimeters. In some cases, we recommend the use of stable clocks for more precise geolocation.”
NAVIGATION-CHALLENGED
“Flying the drone so close to obstructions and obstacles, which are the same objects that we need to collect data on, is the most challenging,” Leyssens noted. “We use an all-constellation receiver to avoid GNSS signal outages. And it’s very good for multipath [reflected GNSS signals] which is abundantly present.” (Figure 1 shows the flow of positioning data aboard the Airobot platform.)
Each energy sub-sector brings its own particular challenges with the type of structures involved. Wind energy and electrical towers are especially difficult. “For collision avoidance in close-range flying,” Leyssens continued, “a lot of applications use optical technologies, but the problem with wind turbines is that the blade is so wide, there are not so many features of it. And high-voltage towers have open-truss structures; the background is so much bigger than the foreground. These are extremely difficult for optical [camera] technology to deal with.”
Airobot now routinely uses a combination of ultrasound and LiDAR. Ultrasound enables the drone to see everything in front of it, although the technology has a range limitation of 7–10 meters. LiDAR has longer range, but with its narrow field of view, cables and open-truss structures are easy to miss. A point-cloud scanner is too costly for many applications.
“We are integrating hyperspectral cameras,” Leyssens said. “We are looking into automatic detecting of corrosion and other things, like coating degradation, that are not visible to the naked eye or photographic cameras. They are visible in other parts of the light spectrum. Rust, for example, is very hard to distinguish from dirt or bird droppings; more information from different spectral bands will enable correct identification of the area.”
Airobot has been selected by US UAV manufacturer General Atomics to collaborate on a project using Airobot’s expertise in AI for processing hyperspectral imagery in development of the MQ-9B SkyGuardian Remotely Piloted Aircraft (RPA) for Belgium. This followed that country’s defense ministry acquisition of General Atomics SkyGuardians to meet the nation’s RPA requirements.
According to Guzzini, “Long-range flying, including BVLOS, is often required in energy use models and that presents navigation challenges. Delair drones are designed for long flights and intuitive navigation flows and tools assist the user in navigating the drone safely and efficiently. Our drones are lightweight and modular from an assembly standpoint, which makes moving them between jobs sites easy. And they are equipped with highly efficient takeoff and landing technologies that allows them to be launched in challenging environments that sometimes don’t offer ideal deployment conditions or terrain.
“From a hardware point of view,” Guzzini continued, “our systems are developed to be open, modular and seamlessly integrated for optimal efficiency, safety and usability.”
Guzzini emphasized the ultimate value found in integrating aerial data collection into further steps of processing and analysis, critical to extracting valuable business insights for energy companies. This optimally occurs throughout the life cycle of an asset, from planning to operations and maintenance. “The delair.ai platform is the foundation of this data management and analysis, bolstered by a range of use-case-specific applications from both Delair and specialized third-party suppliers that integrate with our solution.”
Colomina of GeoNumerics outlined the requirements for his clients: “As for navigation, moderate accuracy and high reliability of the navigation solution is required. While navigation accuracy requirements are at the few-meters level, positioning requires higher accuracy, in some cases at the few-centimeters level, however enjoying the luxury of post processing.”
GeoNumerics obtains both navigation and positioning requirements by:
• a sufficiently redundant sensor configuration of the navigation/positioning payload
• rigorous sensor modelling
• an effective outlier detection strategy
• an efficient state estimation algorithm and software.
These four tactics combined furnish a sufficient-and quantifiable-reliability level. One application is the management of GNSS signal reception problems generated by intentional/unintentional GNSS jamming.
Where jamming may be encountered, geodetic/survey-grade GNSS receivers are required, possibly aided with IMUs if not only position but position-and-attitude (orientation) is also of interest.
“For a number of reasons,” Colomina said, “the main one being that autopilot vendors and orientation software vendors are different communities, the navigation task is performed by the autopilot’s hardware and software, and the orientation task by different, unrelated ones. This is clearly suboptimal in that neither navigation benefits from the orientation payload sensors nor orientation benefits from the navigation sensors and previous results.”
Such operations must take at least two additional challenges into account:
• short duration and low dynamics of drone flights, particularly multicopter drones
• the fact that the final accuracy the end user is interested in depends also on the imaging sensor that taps into the navigation/positioning solution.
Short flights and low dynamics make it difficult to calibrate IMUs. The transfer of accurate orientation from the motion sensors to the imaging sensors requires properly calibrated imaging sensors which, in turn, adds requirements on the pre-calibration of the imaging sensors, on the stability of calibration and on the simultaneous orientation and calibration task, that is to say self-calibration.
“Lastly,” Colomina added, “a recurrent problem that we face is the synchronization errors between the various components of the positioning/navigation/timing (PNT) payload and, above all, between the PNT payload and the imaging sensors.”
CONCLUSION
The real estate mantra of location, location, location holds true for inspection and maintenance in the energy sector. Data is only as good as the precision with which it can be pinpointed to a spot within large, complex installations. And that positioning can only be obtained with precise data on UAV location and orientation, coupled with sophisticated algorithms to then determine the exact location of the observed site, in some cases only inches or centimeters in size
