AI, ML, CV Collaborate to Fuel Construction Success.

At opposite ends of Illinois, two leaders in emerging construction technologies are collaborating to demonstrate how AI, ML (machine learning) and CV (computer vision) can validate construction’s digital-data mandate: Save time, save money, save lives.

For the fourteenth consecutive year, U.S. construction valuation has grown and is now at $1.98 trillion, noted Mani Golparvar, professor of civil engineering, computer science and technology entrepreneurship at the Grainger College of Engineering/University of Illinois. But, he’s reported, 53% of construction projects are behind schedule, 66% are over budget and nearly all carry cost overruns.

Golparvar estimated that $280 billion in potential added value exists through improved coordination. “As an industry, we’ve significantly improved the way we plan projects. But the way we plan the job, execute, monitor our execution and use whatever we monitor to update a plan—these four problems all contribute to why projects are behind schedule and over budget. If you’re looking to these issues, 80% are preventable: Project team members do not have visibility early enough to be able to come up with a remedy.”

To that end, Golparvar is founder and chief strategy officer of Reconstruct, a Visual Command Center that uploads schedules, 2D and 3D models, and reality capture so users can track progress as well as coordination and communications problems. Reconstruct says its benefits include a 30% reduction in time reporting field progress, and 25%-40% in improved schedule management and proactive risk mitigation. Eighty-five companies and several universities now partner with it.

Burcin Kaplanoglu mirrored Golparvar’s remarks. Speaking from the Oracle Industry Lab he heads just outside Chicago, he noted that U.S. infrastructure is graded as a C- by the American Society of Civil Engineers. At the same time, he said, “we [would] need to hire 500,000 people a year, on top of regular hiring, to meet demand.”

Consequently, he added, “we need to automate processes, gain efficiency and improve safety with technology. Talking to your data is going to have tremendous benefits for the construction engineer. Getting that response really quickly is going to change how we interact with technology.”

Golparvar and Kaplanoglu recently joined forces around a Reality Mapping Experiment (see Constructing the Future sidebar, pg. 46) at the Oracle lab to explore what combination of tools, processes and teams can best conduct reality capture and mapping on-site, reducing time and cost and ensuring safety.

That’s of great value to an industry facing an ever-growing tech stack, as Jennifer Suerth, senior vice president of Pepper Construction, which built the lab, said in an “Oracle TV” video. “Autonomy allows us to continue to get the work done but be efficient with the resources we have.”

Defining Terms

Golparvar and Kaplanoglu offered quick definitions of ML, CV and AI.

Machine learning allows computers to “learn” from data to improve and streamline processes. “Machine learning is continuing to improve the construction industry,” Kaplanoglu said. “We need to figure out how to optimize and how to forecast better.”

“Computer vision is a form of AI,” Golparvar said. “It involves the process of analyzing pictures and videos and generating actionable insights from them, doing image processing and understanding the geometry of the scene.” Recognizing objects by automatically analyzing them can track progress, detect and recognize anomalies, and coordinate and optimize operations.

“To solve that coordination issue requires everyone to be on the same page in terms of what was there versus what should be there,” Golparvar continued. “In construction, we’ve really taken advantage of design information that we call BIM, building information models, 3D representation of design. What if you tie them against the schedule? At any snapshot in time, you can click on any two points, you can measure the length, area and volume. The picture shows you what was promised to be done; the picture in the background shows you what has been achieved on the job. And the delta between the two—what needs to be coordinated in terms of quality, safety and progress—can be color-coded in red and green: what is on the schedule, what is behind.”

Generative AI is the third leg of this digital triad. AI involves machines mimicking human cognitive functions, but there’s more to that than large language models. “GenAI is going to help us do project management and planning,” Kaplanoglu said. AI can inform the entire construction cycle, from estimating preliminary costs, to tracking quality and productivity, to the more tactical tasks of ordering and payment.

Constructing the Future

Kaplanoglu ventured predictions for AI, ML and CV in construction:

• AI use cases will go beyond predicting outcomes to driving efficiencies, on and off the job site

• Machine learning and computer vision will further streamline processes

• Robotic scanners and autonomous drones will increase productivity and safety

• Stronger data practices will help organizations demystify analytics and make better, faster decisions

Unified, intelligent clouds will unite software, platforms and infrastructure within end-to-end solutions, easing management and accelerating change.

Inside the Oracle Lab

Oracle is a major player in the world of project management and design, and in 2018 Kaplanoglu co-founded the Oracle Industry Lab to advance the company’s innovation in construction and engineering. He explained the rationale. “When you’re running day-to-day operations, it’s very hard to stop what you’re doing and try these new technologies. So, by creating a testbed where we bring the technology and they bring their problems, we are creating a neutral space to try and learn from it.”

To realize this, writing a data strategy around machine learning was followed by building an ecosystem “because our products do scheduling, safety, risk management, cost. We had almost 2,000 engagements in the first two years,” he said. The pandemic only spurred a concentration on autonomy. “All of a sudden, we did remote site monitoring, cameras and drone flights, route inspections. We could work with architects and designers remotely; they didn’t have to come to the site. Everything was video.”

The original “sandbox” gave way to a much larger, industry-benchmarking construction space for investigating realistic job site situations. It opened in April 2022—“on time, on budget.” With analogs now operating in the United Kingdom and Australia, it has passed 13,000 engagements. “I think construction really got used to digital imagery, 3D point clouds, using drones, capturing data,” Kaplanoglu said.

Construction has some relatively unique characteristics, from a constantly changing environment to wind and weather exposure. “It’s like building a prototype each time,” Kaplanoglu added. Data can bridge that. For example, Komatsu, the massive construction and mining equipment manufacturer, is using Nvidia’s Jetson edge AI platform to impart intelligence to trucks, excavators and the like. “AI and machine learning are critical to providing real value to the construction space,” a Nvidia spokesman told Inside Unmanned Systems.

“In terms of project management,” Kaplanoglu said, “there’s tons of opportunities when we want to use machine learning, computer vision and GenAI.”

The Reality Mapping Experiment

The need to test construction innovation, validation and measurement in real-world situations led to a collaboration between Oracle’s. Smart Construction Platform ecosystem and founding partner tenant Reconstruct’s visualization tools. The goal was to create guidelines and tool utilizations best suited for particular use cases, improving documentation of and guidelines for progress, quality and assessment of as-built conditions through a structure’s lifecycle.

“The Report explains costs, time spent on different tools, and what kind of results you should be expecting from each,” Golparvar explained about the 2023 document.

Pepper Construction and Clayco offered project expertise on the construction side, with leading drone company Skydio and 3D measurement company FARO Technologies providing technical expertise.

Eight different ways of capturing and mapping data were scanned across resolutions, speed, cost and deliverables. These included manual and autonomous drones, 360 and smartphone cameras, and various stationary and mobile LiDAR techniques. Capture data was processed into Oracle’s platform and then into Reconstruct, which could process them for a consistent viewing experience. Post-processing, photogrammetry and 4D simulations could be viewed for actionable and exportable results.

Construction is said to be the number one civil market for drones, which can take pictures quickly in hard-to-access areas while their sensors allow for obstacle avoidance. For the experiment, a fully autonomous Skydio drone used its “Indoor Capture” vision-based autonomous software over several iterations to, as Solutions Engineering Manager Colin Romberger put it on Oracle TV, “get total coverage…to have the best data to put into photogrammetry software for Reconstruction.”

Kaplanoglu discussed gains from using autonomous drones. “The time savings dropped significantly, because you’re already preprogrammed.” Human drone flights also improved with repetition. “There’s still some autonomy, like it avoids obstacles and you can do flight plans.

“But the biggest drop happens when it’s fully autonomous.”

An AI Construction Cycle

“You always want to know about what you’re trying to solve, what kind of tools you have and what kind of resources you can leverage to solve that problem,” Golparvar said.

He and Kaplanoglu enumerated how AI and associated technologies can self-direct and break down planning and actions.

Site selection already involves machine learning, Kaplanoglu said, “because you have certain parameters to make sure you comply with, like height restrictions, zoning. You can use machine learning to pick the optimal location.”

Design can use machine learning to investigate bigger datasets and find helpful patterns. Golparvar: “You want to capture the architect’s intent and transform it into a document that can be used as a base of design.” GenAI can provide and reconcile design alternatives, incorporating everything from routing mechanical systems to complying with local codes. “The way it works,” Kaplanoglu added, “you sketch things and then people try to visualize it and build models.” Learning from them can offer significant time savings. “I’m talking about doing this in an hour versus doing it in months.”

Construction has its own mini cycle. “You define your scope, you hire engineers and architects, and now you need a contractor to build it” Kaplanoglu said. “You send an RFP for a contractor to build it.”

RFPs, however, can be limited and schedulers may cut and paste. Kaplanoglu offered a solution: “We can upload an RFP document to Oracle Cloud infrastructure. It reads the document and gives you a summary and then it asks you questions. It’s going to show you an early-stage prototype, and it builds a schedule for you. It’s going to be a good template for you to build your own schedule.

“Now something that would have taken you three days is going to take you maybe two to three hours.”

Operations can use computer vision. “How many times does someone show up at their location and they don’t even know what the specifications are or what they’re supposed to do,” Kaplanoglu said. “You can take a picture or video and then computer vision can tell you, ‘It’s this manufacturer, this model, or vision can process the video or image and say, ‘Oh, there’s rust in this corner.’”

Predictive maintenance can significantly reduce costs and downtime. Safety also can be empowered by AI. “Many of the AI tools that we have are completely focusing on offering better awareness for our workers,” Golparvar said.

Data Integration

Skydio has participated in the Oracle lab’s work, and Kaplanoglu offered an example of its drones adding value. “We have a great relationship with Skydio. Instead of taking images that humans need to tag, locate and input, that data goes to our product, and then our Vision Services recognizes the rust and registers it to our work order system. But it doesn’t just register; it tells you the lat[itude] and long[itude], the location, the severity of the damage, it helps you create a ticket.

“Basically, we automate the whole thing.”

Imprecise capture, incorrect inputting and lack of updating can undermine results. Still, “the industry has significantly improved its use of cloud,” Kaplanoglu said. “The data is more accessible, you can put more line, compute is easier. There are a lot of benefits to it, and our industry really took that to heart.”

Golparvar advocates a step-by-step process. “We do a maturity assessment on the readiness of a company for adopting and adapting AI-driven products, and what we can do to make sure that these products fit into the existing workflows. Because people are resistant to change.

“We also need to demonstrate the return on investment, at the project level and the individual level”—quality, speed, time and money saved. “So, the best strategy is to kind of introduce one next step to that project team so we can take them from where they are to that future where many of their steps will be fully automated, and make sure they see how AI is verifiable, to have that element of trust.”

Kaplanoglu is bullish in terms of technology adoption. “Machine learning, computer vision and GenAI are all going to have a big impact in the next three to five years.”