ABSTRACT

Photogrammetry is a useful way to quickly and accurately construct 3-D images online. Journalists and digital artists are applying photogrammetry to investigative journalism and cultural heritage preservation. Unfortunately, there has been very little research done on how to authenticate these 3-D models. The goal of our research is to build upon current open source frameworks to create a new workflow that expands provenance and authentication to 3-D photogrammetric models for journalists. Existing frameworks focus on images, but our work expands upon provenance methods built by Starling Lab for Data Integrity at Stanford by designing an interface that journalists could use to verify photogrammetric models. We implemented a framework to associate images with a 3-D photogrammetric “fingerprint” of the location where the photo was taken and a method to present the digital signature in conjunction with the 3-D model. Our research is just the beginning, and a lot of work must go into tackling the problem of 3-D digital authentication. All in all, the rise of 3-D methods and media necessitates a revolution in techniques to preserve the authenticity of 3-D digital records.

The Stakes

On February 22, 2023 two civilians were killed in Nablus by Israeli security forces. Israel stated that the troops were fired at, but videos that emerged from the chaotic event indicated otherwise. In the article, “3-D analysis shows how Israeli troops fired into group of civilians,” Washington Post Journalists Miriam Berger and Even Hill used three videos of the event, audio, and then thousands of images of the site taken after the shooting to reconstruct the scene utilizing the process of photogrammetry. 

Photogrammetry uses photographs taken of a real life environment to recreate in a digital 3-D structure a model, like the street in Nablus where the shooting occurred. The photogrammetric model combined with additional illustrations allowed the Post’s digital forensics team to simulate the perspective of both a civilian hiding on the street and the shooter in an Israeli Wolf armored vehicle. They were then able to analyze the view of the shooter of the civilians, and cast doubt on the statements made to The Washington Post by the Israeli government about the event. 

As the Post indicated: “The reconstruction shows that, while responding to what they claimed was a gunman, Israeli forces fired at least 14 times from inside their armored vehicle as it moved down a street and then came to a halt next to a short wall behind which the civilians huddled. The Israelis continued firing even after those people would have been visible from the vehicle’s windows, the analysis shows.”

The article created a unique reader experience. By combining comprehensive investigative journalism, illustrations, and highly technical 3-D models, viewers can see for themselves how the event unfolded. While interacting with the 3-D model, the reader sees the civilians crouching behind the street wall and observes through the inside of the armored vehicle what would have been the soldier's perspective. The article explains how, “The 3-D reconstruction shows that the man who ran from the sidewalk and ducked behind the wall would not have been clearly visible from either window,” enabling questioning of the motives that made the soldiers continue to shoot.

This new area of visual storytelling and digital forensics utilized in the Nablus article is being pioneered by The Washington Post. The newspaper has been using 3-D constructions based on collected video and photographic evidence from bystanders to enhance their journalistic abilities. In addition to the Nablus article, Post journalists have also applied these digital reconstructions to other critical events, like the Uvalde School shooting, examining deficiencies in the response to the school shooting, or tracking police brutality in protests in Iran. 3-D reconstructions offer a powerful tool to bolster their reporting and  better authenticate events on the ground.  

Another application of photogrammetry is its use to preserve cultural heritage sites in the war between Russia and Ukraine. As they are destroyed, capturing images of buildings and monuments with 3-D scanning technology allows for the photographic recreation of these unique places. 

Pixelated Realities, a digital artistic non-profit based in Ukraine, partnered with photographers, 3-D artists, and people in Ukraine in order to capture necessary data that will allow the creation of  complicated photogrammetric models where scenes of war can be reconstructed. Since the war began in February 2022, artists at Pixelated Realities have deployed tools from Lidar scanning to photogrammetry to capture structures, art, and cultural sites across Ukraine from Kharkiv to Kiev. The NGO is working on an ongoing project “The Museum of UA Victory,” and hopes to use 3-D Structures to capture audiences' attention around the world. The Pixelated Realities mission statement statement explains:

Three-dimensional reconstructions, whether of a devastated home or the rusting shell of a burnt-out car, have the potential to be more emotive than traditional war photography. We believe that empathy between us is important and make a step to develop amicable relationships between usual people of Ukraine and Europe by showing hardships which we now see every day in our previously peaceful country.

An even more abstract application of 3-D authentication technology may lie in unfolding virtual worlds. In order to successfully construct Meta’s dreamed-of digital world, 3-D environments, tokens, interactables, and avatars will be central to the experience. A plethora of authentication problems arise within this virtual space, whether it's how to credit artists for their 3-D work, prove authenticity for digital goods being traded or make sure the relationship between an avatar and user is secure. 3-D authentication tactics will be needed for all of these features and will be key to forming trust between the virtual and real worlds.

To date, resources and research dedicated to tackle the problem of authenticity focuses largely on images. Applying the concept of provenance – surfacing metadata, tracing the digital history of a photograph, and utilizing new tools from Web3 like cryptographic signatures – creative solutions are emerging to track and present digital media provenance in an effort to combat mis-and disinformation online. Because of its rise in popularity, photogrammetry should be included in the development of provenance protocols. As displayed by The Washington Post’s Digital Forensics reporting and Pixelated Realities Ukrainian cultural preservation work, photogrammetry and reconstruction are powerful journalistic and accountability tools for the future. 

However, unlike projects spearheaded by Adobe and the Content Authenticity Initiative and the adoption of technical C2PA standards designed in large part to standardize photographic authentication, standards for photogrammetry have yet to emerge. Most digital models, like the Washington Post’s Nablus investigation or the The Museum of UA Victory, are viewed through a website. Usually, text appears on the page or next to the model describing who made the model, where it was made, and other information that can bolster the reporting, as was the case with the Nablus story. Currently, it is up to the discretion of the website or journalist organization to determine what information to disclose. Little work has been done to unify standards and best practices for the authentication methods of photogrammetric models. The rise of 3-D methods and adoption by media, necessitates a revolution in techniques to secure the authenticity of 3-D digital records.

What Should We Do?

The applications of photogrammetry could have a far reaching impact on digital media. To date there is minimal research on authentication methods of these 3-D visuals. Most of the research focuses on mapping the accuracy of models to real life and how to get better quality photogrammetric models. This does not address the importance of what those photogrammetric models are being used for and how to create secure ways to authenticate the models if they are to be shared around the internet. The crucial role these 3-D models could play in the future means equally secure methods of verification and authentication must be developed.

Creating provenance methods for 3-D models introduces a host of problems that are difficult to tackle in comparison to capturing end-to-end provenance of just a single image. Photogrammetric models can be constructed from thousands of photos, making tracing all of them a difficult task. During the process of constructing a model, depending on the quality of the source material the model may take some editing afterwards to actually more accurately reflect the real world. Finally, the content file these 3-D models are stored in have a lot less internet architecture built to support them, making it challenging to apply provenance methods currently being used.

The goal of research we conducted as part of Stanford University’s REU program in the summer of 2022 was to build upon current open source frameworks to create a new workflow that expands provenance and authentication to 3-D photogrammetric models for journalists.

As mentioned, existing frameworks focus on images but our work expanded upon provenance methods deployed by the Starling Lab for Data Integrity, a research lab anchored at Stanford University and USC. Our goal was to design an interface that journalists could use to verify photogrammetric models. It is difficult to implement existing methods of image verification to authenticate models, due to how they are stored (usually in .obj files). Building on provenance methodology-using metadata, cryptographic signatures, and registration on a blockchain, we implemented a framework (above) to associate images with a 3-D photogrammetric “fingerprint.” We designed a specific manifest framework for photos used in 3-D photogrammetric models and wrote a script that streamlined the digital signature process. The data embedded in the digital fingerprint and stored on the blockchain, affords users additional context as to the origins of an image.

It starts with a Mesh

After the initial photos are gathered and verified, the process of constructing the final 3-D mesh begins. The verified photos and placed into a photogrammetry program- for our research we used the open source software Meshroom by Alicevision. As seen in the image below, the program stitches together the photos that have been fed to it, matching lighting and data points to create a base layer for a 3-D model. In this image one can see all the camera angles and positions where the photos were taken (little squares), as well as the beginning of the mesh and color reconstruction. 

The finalized image produced by the photogrammetry app usually requires a bit of editing, in apps like Blendr, to make it more accurately reflect the real world. While photogrammetric software is good, depending on the quality of the original photos, various results in the final model can cause inaccuracies. This is another reason verification to the original images is important, because it allows the viewer to check that the model hasn’t been over-edited.

Finally, we built a website framework that allows users to interact with this 3-D photogrammetric model side-by-side with the verified photos. Our template features a small eye “i” icon that serves as a signal to readers that more “content credentials” can be explored. Clicking on the “i” will surface the authentication data embedded in the image. The goal of this framework is to have users compare the original verified photo against the final mode. Users can explore the content credentials, view the 3-D model, and look at the original images. This multifold approach actively engages users in the process of verification and content authentication for the 3-D model.

It’s Early Days

More research is needed to improve authentication methods for 3-D models. While the process that our Stanford team designed is a start, more needs to be done to tackle authenticity for 3-D models. Our process currently requires the authentication of every photo within the 3-D model, which only works well in the context of small models. Larger, more detailed photogrammetric models can require thousands of initial photos to construct a 3-D element, larger than what is pictured about which is about 100 photos. The scale necessary quickly becomes unwieldy when attempting to authenticate every photo. Additionally, there is no way to guarantee that the photos fed into the photogrammetry software are the verified photos themselves, as the software has no way currently to check. Ideas to be tackled in future research include:

• Creating a way to sign and verify the .obj file directly, allowing for the verification of a single file instead of thousands of individual photos.

• Integrating the verification process into a single app, which would simplify the process and make it even more tamperproof.

• Identify ways to make sure the signature on the 3-D model can be tracked across the internet.

• Expand and find more efficient verification methods for larger models.

• Further refine the models for accuracy in order to answer the question of whether they are deterministic.

Having secure, traceable ways to track provenance and manipulation of 3-D models is going to be key moving forward as the possibilities for the virtual world exponentially grow. While the future of photogrammetry is promising, we must make sure to take a step back and build a robust system of trust and authentication in order to prevent exacerbating the misinformation problem.

Adri Kornfein is an Electrical Engineering student at Stanford University originally from Sebastopol, California. She has a passion for energy and the environment, journalism, and utilizing technology for social good. Adri was a summer research assistant at the Starling Lab for Data Integrity, a writer for The Stanford Daily’s Business and Technology desk, and was awarded the Honorable Mention for the Stanford Boothe Award.

Stanford Electrical Engineering students Justin Wei and Na Young Son also contributed to this research.

Stanford Physics PhD candidate Noah Huffman oversaw and managed the research.