By Lauren Butler, Journalist, Infrastructure magazine
Digital twins – virtual replicas of physical objects and systems – have huge potential across many industries, including infrastructure, to improve efficiencies, cost and safety. The data and digital specialist arm of the CSIRO, Data61, is using sophisticated algorithms to allow manufacturing and other industries to create the 3D representation part of a digital twin in its new Melbourne-based Mixed Reality Lab.
According to the International Data Corporation (IDC), 30 per cent of the top 2,000 global companies will use data from digital twins of Internet of Things (IoT) connected assets to improve productivity by up to 25 per cent.
CSIRO’s Data61 Senior Software Engineer, Matt Bolger, said that the new Mixed Reality Lab is made possible by a set-up of industrial and consumer optical cameras and sensing equipment, which capture detailed information about a physical object and the space surrounding it. The new technology blurs the lines of what is physical, digital and biological to improve decision-making and planning.
A focus on the technical
The new Mixed Reality Lab is the result of Data61’s research expertise in machine learning, computer vision, computational modelling and IoT, as well as CSIRO’s patented Stereo Depth Fusion technology, which uses depth estimation to significantly speed up reconstruction and therefore the time needed to create a digital twin.
According to Mr Bolger, the work happening at the Mixed Reality Lab primarily focuses on measurement and visual inspection problems. Its camera systems, projections and algorithms can be tailored to various use cases, making it highly valuable across several major industries, including potential benefits for the infrastructure sector.
The team in the Mixed Reality Lab are primarily in software development and engineering. Data61’s software platform Workspace in particular specialises in modular software development. Mr Bolger said they’re looking at different vision-based systems for doing 3D reconstruction of objects.
“That could be something being manufactured in a factory. It could also be some large object outside, so we’re looking at the different techniques that could be used where we can use different types of depth-sensing cameras, or traditional 3D cameras,” Mr Bolger said.
“We can then build arrays of these systems up, where we can be scanning large objects. So we’re taking a lot of different photos of objects from different angles, analysing that, and we can start creating a 3D reconstruction of the object.”
The lab is an amalgamation of multiple areas of technology, taking machine learning to the next level. Data61 looks into the applications of deep learning, as well as machine learning and artificial intelligence (AI).
As well as objects, the technology can be applied to humans to analyse movement, using deep learning and biomechanical modelling to make predictions and improvements to various environments, including
The Lab can take relatively low resolution video feeds from far away and use machine learning to identify where people are. This allows them to track their skeletons, which act as biomechanical models.
“In terms of factory scenarios or work scenarios, this allows us to use deep learning to start identifying foreign objects in a scene. This provides ways for the computer to understand what’s happening in a work environment. That’s feeding into simulation systems or some sort of situational learning system,” Mr Bolger said.
“That machine learning then starts to feed into analytics. Anything where you can start building up your training set and start building these big data sets, that can start being fed into potentially using machine learning and deep learning as a technology for predicting future things, or helping you identify things much more quickly without user interactions, or tagging and identifying things.”
Applications in the infrastructure industry
Using this technology, technicians can potentially create digital twins to provide infrastructure industry professionals with information to make more informed decisions about how infrastructure is built, used and maintained.
One key area of information that could be provided by this technology is in relation to defects.
“We can analyse defects and then present an overlay of those defects back to the users, viewed through augmented reality. That’s really just scratching the surface on scenarios and how they could be applicable,” Mr Bolger said.
Digital twins are expected to significantly reduce costs in manufacturing, making a valuable contribution to the economy. With a thriving manufacturing sector central to the government’s plans to grow the economy and create new jobs, this application is critical to growth in both manufacturing and infrastructure.
At the moment, Data61 is also undertaking work in areas from bushfire management and flood modelling, right through to traffic flow.
“There are lots of pieces of technology coming in that are probably applicable to that infrastructure area. There’s this concept of monitoring everything, analysing it in real time and then making predictive decisions,” Mr Bolger said.
“Our team’s focus is in the lab at the moment, but I think a lot of these technologies are very applicable.”
Within the lab, the team is specifically looking at mixed reality and how it can be analysed and turned into real-time feedback. This could have a huge impact on inspection information, highlighting defects, or more general situational awareness.
“This includes things like tracking robotics and automatic vehicles, as well as humans moving around the space. Maybe identifying hazards where action needs to be taken. There’s a lot of side pieces to the puzzle that we’re also looking at in addition to the core scanning inspection problem in the lab,” Mr Bolger said.
It all comes down to scalability
Mr Bolger said that Data61 has been in recent conversations with players in the construction industry to translate what it’s currently doing to be leveraged in the infrastructure sector. With some of the manufacturing scenarios Data61 is currently working on being large scale, it is feasible that the technology could be applied to large infrastructure projects to deliver exciting outcomes.
“Some of the technologies can be used to create a representation of an object; take very accurate measurements of some large building or something. Then periodically we would do the same thing, and compare these differences over time. So that’s one way we see it,” Mr Bolger said.
“I think particularly around the health and safety monitoring, there’s probably a lot of similarities to factory scenarios as well. With the concept of situational awareness, you can imagine a system that through cameras, or some deidentified sensing system, can monitor people in a work area, or maybe it’s visitors to a certain area.
“Pulling statistics from that and grabbing these data sets means that it then can be used to set analytics and start doing predictive work and those sort of things. I see a lot of those technologies applying to the infrastructure space as well.”
Amongst its other benefits, the modular aspects of the software and technology have multiple uses. Much of this comes down to its scalability.
“The background of our team is that we’re a software development and engineering team with a specialty in modular software development. We can facilitate collaborations between different research groups in a software context and build these systems so you can leverage something you create for project A and use it somewhere else in project B.
“This work we’ve done in the lab over the last few years is sort of like a hardware extension of that as well. So we’re facilitating the modular development of the software and making things that can be reused and scaled, but then sort of bringing in the hardware components of that as well,” Mr Bolger said.
These hardware components include several types of camera systems, each of which come from a different vendor, or which might use a different type of technology. For this reason, the types of data they produce are different.
“We’re developing systems that bring that into a common platform where you can swap out cameras or a different type of sensor for something else, or bring in your data from some other form and we can bring that into a common platform,” Mr Bolger said.
“We can then reuse the other components in the overall pipeline as though you’re processing the data. You can build a sort of scalable computing network systems as well.”
Collaboration with industry
Since the lab’s opening, the Data61 team has had a number of local companies and universities reach out to collaborate, and one of the goals is to have this technology successfully implemented across industries.
“There’s success on many levels. Part of the lab is providing a facility where we can bring in local industry here and start looking at how these sort of technologies apply to their challenges. Or what other sorts of complementary research and development is happening out there that could be part of this as well,” Mr Bolger said.
These collaborations, as well as the facilitation of several areas of research, will help Data61 to deliver exciting commercial projects and continue the research projects that are underway. With the technology rapidly developing, it’s only a matter of time until we see this implemented on major infrastructure projects.