Written by Yuhan Yan
In 2019, Microsoft announced its AI for Cultural Heritage program, aiming to preserve and treasure people, places, and historical artifacts through AI technology and collaboration with cultural nonprofit organizations.
““Preserving cultural heritage isn’t something that is solely nice to have or nice to do, it’s sometimes imperative to the well-being of the world’s societies.””
According to Poonam Gandhi, a professor at Flame University, culture and heritages shape human society and provide a sense of identity. Specifically, cultural heritage can stimulate tourism, which benefits local economies, educates people with culture and history, and enhances social inclusiveness.
While the importance of preserving cultural heritage is obvious, people are facing difficulties in the preservation process. So, what can AI and other modern technologies actually do in the process of preserving cultural heritages that makes Microsoft and other organizations invest in them?
Antiques Restoration
Due to natural disasters or man-made damages during the war, a great deal of precious historical sites and artifacts have been damaged beyond repair. It is difficult to repair damaged sites, as it may take several generations of individuals working on these projects due to time, cost, and the lack of information on how the original work actually looked. Rebuilding cultural heritage sites and artifacts is similar to solving a big jigsaw puzzle without knowing what it should look like – everything starts from scratch and guesses.
RePAIR
The ancient Roman city of Pompeii, which was destroyed and buried by the eruption of Mount Vesuvius in A.D. 79, is now only left with the incomplete architecture and thousands of irregular fragments of ancient artifacts, such as vases and frescoes.
Figure 1: The Pompeii Archaeological site at sunset. Source: Buena Vista Images/Getty Images.
The RePAIR (Reconstructing the Past: Artificial Intelligence and Robotics meet Cultural Heritage) project is run by a group of scientists at the Italian Institute of Technology (IIT) to help with the time-consuming and labor-intensive physical reconstruction process of Pompeii’s restoration. The technologies involved in the project are robotics, 3-D scanning, machine learning algorithms, and artificial intelligence.
Figure 2: A video showing a robot working with broken fragments found at Pompeii. Source: YouTube.
These artifact fragments are scanned using high-tech computers, which use machine learning algorithms to predict their original configuration. Throughout the process, the AI is guided by humans to ensure that the pieces are being accurately reconstructed.
A team of archeologists from the University of Lausanne, along with other professional art historians and scholars, input their current knowledge about Pompeii into the AI algorithm, then let it evolve and study the information by itself based on those examples, and apply it to a larger scale. The computer software compares all fragments in pairs and evaluates their degree of similarity based on the shapes, the fitness, and how the illustrations on the pieces match up, which helps draw a picture of what the original complete artifact looked like.
The second component brings robots into the workplace. The robot should be able to scan those pieces on its own, and move the fragments together as a complete one based on the paired results generated by AI. In the end, it will reassemble the pieces together into a finished “puzzle.” Designed by Nikolaos Tsagarakis and his colleagues at the Humanoid & Human Centered Mechatronics Lab at IIT in Genoa, the robot looks like an arm that has a similar size as the average person, weighing 25 to 30 kilograms and lengthening 80 to 100 centimeters. A prosthetic hand with a glove is attached to the arm to grab and move the fragments with extreme care, and gather information about them while holding them. The robot saves a large amount of time and human resources. Antonio Bicchi, a senior scientist at IIT in Genoa and the chair of robotics at the University of Pisa, is in charge of the design of the robot’s hand, and says the ideal version of it in the future will also be able to allow humans to wear the glove and gather data during the handling process.
Figure 3: An illustration of the robot’s working process. Source: Courtesy IIT.
This project and its techniques are still at the experimental stage. The two experimental subjects are the frescoes from the ceiling of the House of the Painters at Work in the Insula of the Chaste Lovers, and the frescoes from the Schola Armaturarum. At the end of the project, the finished frescoes will be exhibited to the public for visitors to experience ancient Pompeii and so that researchers can conduct further studies. This is the first time in the archeological field that AI will be used at such a large scale, and that robotic hands will be responsible for such a large amount of pieces.
Figure 4: Pieces of a Pompeii pottery artifact mid-assembly. Source: Courtesy IIT.
Therefore, the progress it can contribute to cultural heritage restoration and human history is significant and meaningful. “If the project succeeds, the technology can be used to reconstruct a variety of fragmented cultural artifacts whose restoration has remained ‘out of human reach’, ” says project coordinator Marcello Pelillo, a professor of computer science and artificial intelligence at the University of Venice.
The RePAIR scientists aim to install the robot in Pompeii by the spring or summer of 2022. And hopefully, this project will succeed and benefit other cultural sites in the world that require similar attention.
Scan4Reco
Another EU-funded project called Scan4Reco is also a great example of using artificial intelligence-assisted technologies for cultural heritages’ preservations.
Figure 5: Image of a robot detecting the deep damage of an artifact. Source: Scan4Reco.
This project uses in-depth scanning cameras to create a high-resolution 3D replica of an artifact to transfer it into digital files. Then, more information about the object’s underlying layers which cannot be witnessed directly by the human eye is added to the research by collected sensor data.
“Very specialized sensors used in the conservation domain carry out non-destructive diagnostic analysis of the different layers that is equivalent to the more destructive chemical ones,” explains Dr. Anastasios Drosou, deputy project coordinator.
The sensor data is combined with predictive modeling technologies using AI, whose deep learning algorithms simulate the artifact’s appearance as it ages. Therefore, it can suggest conservators with appropriate treatments to protect the artifacts before their conditions worsen, or to minimize the existing damages and keep their current state. According to Dr. Drosou, when the system detects a scratch or some color fading in a painting, the system would advise conservators to apply a particular chemical on the scratch, or paint or cleaner in the faded area.
Digitization
Laser scanning and photogrammetry are additional technologies that play a key role in the increased progress of digitizing cultural heritage. Having digital cultural heritage will help record history, establish a digital library for researchers and public education, as well as enrich cultural vibrancy.
The Mogao Caves, located in Gansu province in northwestern China, contains over 2,000 sculptures and 45,000 square meters of unique and fragile painted murals, all of which are between 600 and 1,600 years old. Currently, archeologists are facing the challenge of preserving these ancient works of art in their current condition.
Figure 6: A Buddha statue from the Tang period in Cave 148 of the Mogao Caves. Source: Krister Blomberg/Wikimedia Commons.
These caves house a great deal of history regarding the Silk Road, as well as related commercial trades and cultural exchanges across countries. However, due to thousands of years of natural disasters and damages, as well as the overloaded tourists after its development and reopening as a historical visiting site, the damage to the painted statues and the fresco degradation of Mogao Cave has dramatically accelerated.
To better preserve this UNESCO World Heritage Site, researchers and scientists in China used Laser Scanning and Multi-baseline Rotating Photogrammetry to turn the statues and frescos into digital formats, and store them in a safe data library for future studies and possible virtual exhibitions. This would allow people appreciate the caves on their mobile devices with the similar experience as if they were visiting the site in person, but ensure that the caves were not being damaged further.
Photogrammetry: “a sophisticated process by which information is extracted from photographs to create accurate three-dimensional maps and models. Using ultra-high-resolution aerial photographs, this practice combines UAV-mounted overhead sensors with powerful GIS mapping systems to create dynamic, measurable documents for a number of real-world situations and uses.” (Source: Science Direct)
Laser Scanning: “an automatic, direct measurement of 3D points that can accurately measure and collect data from objects, surfaces, buildings, and landscapes. Laser scanners collect information in the form of point cloud data, which consists of millions of 3D coordinates (XYZ coordinates).” (Source: Science Direct)
Another historical site that has been damaged and needed further protection and repair is the Notre-Dame Cathedral which caught on fire in 2019. The government is planning for its reconstruction, with a hope to finish by 2024.
Figure 7: A video showing how laser scanning revealed Notre Dame’s building structure. Source: YouTube.
Thanks to the digital scanning that was conducted prior to the fire, the team was able to create a 3D BIM model of the Notre-Dame Cathedral as it existed before. Then, by using recent reality capture scans to create the post-fire 3D model using a 3D design software called Autodesk, and comparing it to the pre-damaged 3D model, the Cathedral’s rebuilding plans were more accurately solidified.
Since the majority of the damage on the Notre Dame Cathedral was its roof, which is made of wood, its cultural tragedy rang the alarm for other historical buildings with wood structures, as their fragility was exposed. Performing 3D scanning and modeling to have digital forms of the heritage sites on file would provide a strong risk management preparation plan for future possible accidents.
Figure 8: The wood frame of Notre Dame’s Cathedral’s roof. Source: Notre Dame De Paris/Maurice de Sully association.
Conclusion
AI and other advanced technologies can help with the restoration and digitalization of tangible cultural heritage. They work as assistants to archaeologists and historians in preserving work that cannot be achieved by humans alone, or projects that are overwhelmingly time-consuming. Additionally, they work as tools for people to utilize and advance their working process to a higher level.
This article presented projects and examples that have exemplified great progress in cultural heritage preservation. As these technologies continue to develop and experts in the field become more knowledgable and adept at using these technologies, there is a positive prospect that continually enhancing technologies will deliver further achievement in cultural heritage preservation and digitalization by human inventions and under human supervision.
+ Resources
“AI for Cultural heritage.” Microsoft. Accessed February 25, 2022. https://www.microsoft.com/en-us/ai/ai-for-cultural-heritage.
“An Introduction to Drone Photogrammetry.” Mapware. Accessed March 4, 2022. https://mapware.ai/an-introduction-to-drone-photogrammetry/?gclid=CjwKCAiAjoeRBhAJEiwAYY3nDBfZ-wtHG1wd1Xv4sfiGfTEjObRb2Pm-CkaRFOXQycgG5RP7R-1R0BoCm-cQAvD_BwE#whatisutm_term=photogrammetry&utm_campaign=Traffic_Clicks_Global+%5BJoe%5D&utm_source=adwords&utm_medium=ppc&hsa_acc=7859616296&hsa_cam=15446499960&hsa_grp=136083379128&hsa_ad=566158081175&hsa_src=g&hsa_tgt=kwd319349969229&hsa_kw=photogrammetry&hsa_mt=p&hsa_net=adwords&hsa_ver=3.
Brandon, Elissaveta M. “How AI and Robotics are Reconstructing A 2,000-year-old Fresco in Pompeii.” Fast Company. December 23, 2021.
“Cutting Edge | Protecting and Preserving Cultural Diversity in the Digital Era.” UNESCO. October 28, 2020. https://en.unesco.org/news/cutting-edge-protecting-and-preserving-cultural-diversity-digital-era.
Denbo, Seth. “Digitalizing Cultural Heritage: Historic Preservation and The Mogao Caves in China.” Perspectives on History. December 1, 2016. https://www.historians.org/publications-anddirectories/perspectives-on-history/december-2016/digitizing-cultural-heritage-historic-preservation-and-the-mogao-caves-in-china.
“‘Digital Surrogates’ Assist in the Preservation of Cultural Artefacts.” CORDIS. February 19, 2019. https://cordis.europa.eu/article/id/248968-digital-surrogates-assist-in-the-preservation-of-cultural-artefacts.
Gandhi, Poonam. “World Heritage – Why Preserve, Conserve and Protect?” TravelWorld.com. April 20, 2021. https://travel.economictimes.indiatimes.com/speakeasy/world-heritage-day/4891.
“How Technology Is Preserving Our Cultural Heritage.” Reader’s Digest. Accessed February 25, 2022. https://www.readersdigest.co.uk/inspire/life/how-technology-is-preserving-our-cultural-heritage.
Horgan, Rob. “How Digital Technology is Aiding Notre Dame Rebuild, Two Years since Tragic Blaze.” New Civil Engineer. April 15, 2021. https://www.newcivilengineer.com/latest/how-digital-technology-is-aiding-notre-dame-rebuild-two-years-since-tragic-blaze-15-04-2021/#:~:text=In%20a%20bid%20to%20aid,existed%20before%20the%20catastrophic%20event.
Ibaraki, Stephen. “Artificial Intelligence For Good: Preserving Our Cultural Heritage.” Forbes. March 28, 2019. https://www.forbes.com/sites/cognitiveworld/2019/03/28/artificial-intelligence-for-good-preserving-our-cultural-heritage/?sh=bf2912f4e960.
“Laser Scanning Reveals Cathedral’s Mysteries | National Geographic.” National Geographic. June 23, 2015. Video, 4:10. https://youtu.be/jAi29udFMKw.
“Laser Scanning.” ScienceDirect. Accessed March 4, 2022. https://www.sciencedirect.com/topics/earth-and-planetary-sciences/laser-scanning.
Pinkowski, Jen. “Pompeii’s Ruins to Be Reconstructed by Robot.” Scientific American.December 6, 2021. https://www.scientificamerican.com/article/pompeiis-ruins-to-be-reconstructed-by-robot/.
“RePAIR Project.” Pompeii Sites. September 2, 2021. Video, 4:49. https://youtu.be/v9rpoLL5eLM.
“Reconstructing the Past: Artificial Intelligence and Robotics Meet Cultural Heritage.” CORDIS. Accessed February 25, 2022. https://cordis.europa.eu/project/id/964854.
“The RePAIR Project Begins – Robotics and Digitisation at the Service of Archaeology.” Pompeii. Accessed February 25, 2022. http://pompeiisites.org/en/comunicati/the-repair-project-begins-robotics-and-digitisation-at-the-service-of-archaeology/.
Werkheiser, Greg. “Artificial Intelligence: Cultural Heritage Required.” Cultural Heritage Partners. Accessed February 25, 2022. http://www.culturalheritagepartners.com/artificial-intelligence-cultural-heritage-required/.
“What Is Laser Scanning and How Can It Be Used?” TopoDot. June 11, 2020. https://blog.topodot.com/what-is-laser-scanning-and-how-can-it-be-used/.
Wiggers, Kyle. “Microsoft Launches AI for Cultural Heritage Program.” Venture Beat. July 11, 2019. https://venturebeat.com/2019/07/11/microsoft-launches-ai-for-cultural-heritage-program/.