Towards Querying Multimodal Annotations Using Graphs

Jonas Bruschke, Cindy Kröber, Ronja Utescher, Florian Niebling In: Münster, S., Pattee, A., Kröber, C., Niebling, F. (Eds.), Research and Education in Urban History in the Age of Digital Libraries. UHDL 2023. Springer Nature Switzerland, Cham, pp. 65-87. 2023
Towards Querying Multimodal Annotations Using Graphs


Photographs and 3D reconstructions of buildings as well as textual information and documents play an important role in art history and architectural studies when it comes to investigating architecture, the construction history of buildings, and the impact these constructions had on a city. Advanced tools have the potential to enhance and support research workflows and source criticism by linking corresponding materials and annotations, such that relevant data can be quickly queried and identified. Images are a primary source in the 3D reconstruction process, with the possibility to create spatializations of additional photographs of buildings which were not part of the initial SfM process, enabling the linking of annotations between these photographs and the respective 3D model. In contrast, identifying and locating respective annotations in text sources requires a different approach due to their more abstract nature. This paper presents concepts for automatic linking of texts and their respective annotations to corresponding images, as well as to 3D models and their annotations. Controlled vocabularies for architectural elements and a graph representation are utilized to reduce ambiguity when querying related instances.