2.1 Object Detection

[6]Object Detection is one of the fundamental tasks in computer vision and maintains its prominence as one of the most active research areas.‍[9] It goes beyond image-level classification by requiring not only the classification but also the localization of objects within an image. Image-level classification, in contrast, involves assigning a label to an entire image, which generally poses less complexity. Classification benchmarks get their complexity from the amount of categories that need to be distinguished, as exemplified by the classic ImageNet‍[10] benchmark, which required differentiation among 1,000, or in some versions, even 21,000 classes.‍[11]

[7]In detection, two-stage methods like the Region-based Convolutional Neural Network (R-CNN) family,‍[12] have dominated the field, offering strong accuracy at the cost of increased runtime. In contrast, one-stage methods like You only look once (YOLO)‍[13] or Single Shot MultiBox Detector (SSD)‍[14] have traditionally prioritized speed over precision, although the performance gap has narrowed with more recent architectures. Recently, transformer-based detection architectures, particularly those based on the DEtection TRansformer (DETR),‍[15] have become the dominant paradigm. However, one-stage and two-stage methods maintain their relevance. Notably, recent versions of YOLO continue to show competitive performances on many detection benchmarks and are very popular due to their easy usage, efficient training and inference times. For our purposes, the complexity and performance of modern detection benchmarks far exceed our demands. The task of detecting images in printed layouts is straightforward and can nearly be considered as a resolved problem within the field of computer vision. Consequently, we favor the YOLO-v8‍[16] architecture over more powerful but slower modern two-stage or DETR-based architectures.

[8]While applying object detection to artworks is less common than in natural image analysis, it is recognized as a well-established practice within digital humanities. Pioneering work in object detection for artworks was conducted by Elliot J. Crowley and Andrew Zisserman. They trained detection algorithms on the photographic dataset Pascal VOC‍[17] and employed transfer learning to apply the trained algorithms for identifying objects such as vehicles or animals in artworks.‍[18] Another line of research focused on the detection of smell-related objects in artworks,‍[19] a topic addressed with the introduction of the ODOR challenge and dataset. Special cases of object detection, such as the detection of depicted faces‍[20] and persons‍[21] have a great relevance for art history, e.g. for studying portraiture, figure painting, or genre painting. Westlake et al. introduced a dataset of annotated persons depicted across a wide range of artistic styles.‍[22] Closely related to that is the application of pose estimation, which typically requires an initial person detection stage. Pose estimation was used to analyze and cluster artworks based on body postures,‍[23] hand gestures,‍[24] and overall image composition,‍[25] or to recognize smell-related‍[26] and sensory‍[27] gestures.

[9]A common challenge for object detection in artworks is the sparsity of annotated datasets. Gonthier et al. addressed this by employing a weakly supervised training approach, using image-level labels before assessing the method on instance-level labels.‍[28] Additionally, they shifted their focus from detecting modern categories to identifying objects with art historical relevance. This thread was picked up by Marinescu et al., who adapted COCO categories to be consistent with historical contexts,‍[29] and by Reshetnikov et al., who compiled a large dataset categorized by art historical themes.‍[30]

[10]Another approach to address the challenge of data sparsity is the use of style transfer to mimic artistic object representations.‍[31] This technique leverages existing annotations from photographic data to obtain an adaptation to the artistic target domain using transfer learning. This strategy is commonly employed for various tasks, including pose estimation,‍[32] emotion recognition,‍[33] or painting captioning,‍[34] and has also been successfully applied to object detection.‍[35]

[11]The issue of data sparsity, at least for person detection and pose estimation, has been significantly mitigated with the introduction of the Human-Art dataset.‍[36] This dataset features 50,000 images of artistic creations like sculptures, paintings, or cartoons annotated with the position of persons and pose estimation keypoints.

2.2 Image Retrieval

[12]The retrieval of similar depictions of identical objects from auction catalogs can be contextualized within several closely related research areas, with Content-Based Image Retrieval (CBIR) being the most prominent. CBIR, often used synonymously with ›Image Retrieval‹, can be defined as the problem of searching for relevant images in a database given a query image based on visual features.‍[37]

[13]The retrieval process can also be framed as a problem of Near Duplicate Detection (NDD) when considering different depictions of the same object as near duplicates. Primarily aimed at identifying digital copies or manipulated images, NDD differs from CBIR in its application focus. While CBIR is a relatively open task that accommodates a wide range of queries and image similarities, NDD specifically targets the duplicate nature of the query and target images.‍[38] Irrespective of whether it is labeled as NDD or CBIR, a retrieval system typically operates in two phases: A preparatory phase, where a database of quantifiable image representations is created, and an online phase, where this database is queried.‍[39] These queries are not limited to image inputs alone; for example, a shared image-language embedding space, as provided by Contrastive Language-Image Pre-Training (CLIP),‍[40] enables cross-modal querying.‍[41] However, in the context of our application, we are focusing on image queries. In this scenario, the online phase entails a feature extraction similar to that during the database creation. This process maps the query input into the same embedding space as the already processed image corpus, enabling the computation of distances between the query features and the previously extracted features to assess their similarity. As computing the feature distance across all stored feature vectors can become intractable for large image corpora, the embedding space is often clustered or indexed to enable efficient querying. Chen et al. categorize methods based on their reliance on off-the-shelf models, originally trained for different tasks than retrieval, versus methods that incorporate an additional training stage dedicated to fine-tuning the models specifically for image retrieval.‍[42] In our application, we employ the simpler approach by using pre-trained models to extract features. Moving forward, we plan to improve our system by integrating more elaborate training schemes specifically designed for retrieval tasks.

[14]The process of fine-tuning models for retrieval is closely related to the field of metric learning. However, the term metric learning is often discussed more from a technical standpoint than from its practical application. Metric learning involves mapping data into an embedding space where similar data points are closer together and dissimilar data points are further apart, guided by a distance metric such as the Euclidean norm.‍[43] Metric learning can be used for many purposes such as classifying samples according to their nearest neighbors in the embedding space or in self-supervised pre-training. However, retrieval can be considered its most natural application as it typically entails the search for the closest query result in the feature space. This way, retrieval can be framed as an instance-level open-set classification, where the query has to be matched to its closest neighbors in the feature space.

[15]Independent of whether models are specifically trained for retrieval is the consideration of which features are used for the distance computation. Before deep learning, approaches typically relied on local features obtained with methods such as Scale-Invariant Feature Transform (SIFT).‍[44] These local features can be aggregated into a global descriptor using methods like Bag-of-visual-Words (BoW).‍[45] Such combinations of classical local features and their global aggregation have been applied in various contexts, including the recognition of CD-covers,‍[46] object retrieval‍[47] and product search.‍[48]

[16]The question of feature aggregation remains relevant in modern, deep-learning-based approaches: Even when employing neural networks, various feature representations can be considered, involving different scales‍[49] (intra-model) or different models‍[50] (inter-model). Effective fusion of different feature levels can lead to a more meaningful feature representation.‍[51] For our application, a comparatively simple strategy is sufficient. We simply use the flattened output of the last layer of a pre-trained Convolutional Neural Network (CNN) to represent the image contents and compute similarities. Future research could explore different scales and more elaborate feature fusion, but also revisit classical feature extraction methods such as SIFT, and analyze their impact on the type of similarity encoded in the query results.

[17]In the fields of digital humanities and computational cultural heritage, research formerly centered around the visual retrieval of similar images or image parts to identify patterns across numerous artworks and uncover relations between them.‍[52] Eventually, projects and works also focused on the development of interfaces to search for images or image parts.‍[53] Besides feature vector distances, works also explored the application of different metrics to capture diverse aspects of image similarity, for example color concepts,‍[54] image composition,‍[55] body posture,‍[56] or even symbolic meaning.‍[57]

[18]The wealth of works which apply computational methods for art analysis and understanding is reflected in several review papers. While Bengamra et al.‍[58] provide a summary of significant computer vision applications for art, they specifically focus on object detection. An overview of datasets and works on the task of recognizing and extracting patterns in visual arts using deep learning is given in a paper by Giovanna Castellano and Gennaro Vessio.‍[59] Another review provides an overview of how computational methods are used for classification, object detection, similarity retrieval or multimodal representations, among others.‍[60] Amalia Foka then presented past computer vision applications for art historical research and future possibilities.‍[61]

2.3 Digital Art Market & Provenance Studies

[19]Computational methods have also been applied to art market research: Utilizing a subset of over 267,000 sale transactions from the Getty Provenance Index and complex network science, Schich et al. studied the history of the art market and collection dynamics to reveal social, temporal, and spatial networks.‍[62] Fletcher et al. studied the art market in London between 1850 and 1914 on the basis of complementary datasets and visualizations.‍[63] Scheithauer et al. suggested a two-step pipeline to analyze the layout and content of auction sales catalogs utilizing object detection and text sequence labeling models.‍[64] A similar approach and goal to the one presented in this paper was announced in 2021. Then, the Fraunhofer Institute published a report which informed about a feasibility study. The study developed AI-methods for image search in auction catalogs, enabling a successful comparison between current and historical images.‍[65] A second feasibility study, which again highlighted the success of the methods, was published in the magazine Museumskunde in 2024.‍[66] In the humanities, the art market has been a research topic for many years. Numerous publications on the art market between 1901 and 1945 in Germany and other German-speaking countries focus on individual art dealers‍[67], auction houses‍[68], or the valuation and price development.‍[69] An overview of relevant literature is given in the bibliographies German Sales 1901–1929‍[70] and German Sales 1930–1945.‍[71]

[20]The relatively new field of digital provenance research studies the impact of digitality on provenance research, focusing on chances and challenges, and uses digital methods for analyzing provenance data.‍[72] Works report on the development of research databases‍[73], the presentation and communication of provenance information and research results online‍[74], the future of provenance research and digital infrastructures in Germany, especially focusing on tendencies and desiderata.‍[75] Special attention has been paid to aspects of incompleteness and vagueness in provenance research.‍[76] Rother et al. study the transformation of unstructured provenance records into Linked Open Data and how computer-based methods can be utilized for a comprehensive analysis of provenance records.‍[77]

4.1 Data Preprocessing

[24]Before we can store the feature representations of objects depicted in the auction catalogs, we must prepare the catalogs and crop the illustrations. First, we parse the PDF-files of the two bibliographies‍[81] and convert the information into a structured, machine-readable format. Table 1 lists the extracted metadata fields describing the catalogs.

[25]In a second step, we filter the catalogs using the metadata and process them further. Due to computational constraints, we initially limit our scope to a smaller number of catalogs, focusing on locations in Switzerland. Additionally, we narrow our selection to catalogs covering sales of ›Gemälde‹ as specified by the keywords in the bibliographies. This process results in a set of 86 catalogs, with an additional 25 catalogs which are included later for evaluation. We plan to extend the dataset in future work, eventually covering all catalogs in German Sales.

[26]Using the links provided in the bibliographies, we download the catalogs from the online services of the Heidelberg Library and convert each page to an image file using the PyMuPDF‍[82] library. To minimize memory use, we store the pages as JPEG-files with 95 % export quality compression. Subsequently, we employ the YOLO-v8x‍[83] object detection algorithm to automatically detect images of objects on the pages, crop the detected images and save them separately. To train the detection algorithm, we manually label a set of 181 catalog pages with the position for 316 depicted objects. We split 33 for validation (containing 60 objects), and train the algorithm with the remaining 148 annotated pages (256 objects). Finally, the cropped object depictions are converted to grayscale and rotated to align with the longer edge, ensuring a consistent representation in terms of print color and orientation. Fig. 5 provides a complete overview of the preprocessing steps described above. 

Figure 5: Complete pipeline of preprocessing steps applied to the auction catalogs prior to extracting features from the depicted objects. [Graphic: Mathias Zinnen / Sabine Lang 2024]

4.2 Feature-Based Retrieval

Figure 6: Process employed to perform an image-based search for identical or similar images in auction catalogs. [Graphic: Mathias Zinnen / Sabine Lang 2024]

[27]Fig. 6 illustrates the process of feature extraction and query employed for the reverse image search in the auction catalogs. To prepare the system, the previously cropped images from the auction catalogs are used as an input to various feature extraction methods to compile a database of feature vectors as shown in the top row of Fig. 6. Specifically, we employ three different ResNet-50‍[84] feature extractors, each pre-trained for different tasks:

1. Classification of 1,000 classes in the ImageNet‍[85] dataset,
2. Detection of smell-related objects in a dataset of historical artworks (ODOR)‍[86],
3. Recognition of 17 pose estimation keypoints in the COCO‍[87] dataset.

[28]The motivation behind selecting these three pre-training schemes is to evaluate whether the choice of extraction method can influence which images are found to be similar. The underlying hypothesis is that the embeddings extracted from the images can be related back to the task the extraction model was originally trained for. Accordingly, pose recognition embeddings would be expected to emphasize the body posture of depicted persons. ImageNet embeddings, on the other hand, have been shown to generalize towards various tasks and put a specific emphasis on texture versus shape.‍[88] The model trained for object detection in premodern artworks was selected to test whether the availability of artistic object representations in the training data supports the models in extracting more relevant embeddings for artwork similarity.

[29]After deploying the application, users can upload arbitrary images to the system. These images are fed to the feature extractor, similarly to the process during the creation of the feature database. Afterwards, we compute vector distances $d\left(v_q\mathrm{, }{v}_i\right)$ between the query vector $v_q$ and the $n$ precomputed vectors $v_i\mathrm{, }i\mathrm{ \in }\left\{\mathrm{1, 2}\mathrm{, }\mathrm{...}\mathrm{, }n\right\}$ and return the $k$ images $I_1\mathrm{, }{I}_2\mathrm{, }\mathrm{...}\mathrm{, }{I}_k$ where the distance between the feature and query vectors are lowest (see bottom row of Fig. 6).

[30]Our approach simplifies the process of image retrieval by using the flattened final feature vector obtained just before the final classification layer of the ResNet-50 architecture, which typically has 2048 dimensions. Incorporating more elaborate strategies and vector representations that account for multiple scales might further enhance performance. This improvement is a potential direction for future work, although it is beyond the scope of this current study.

[31]For efficient querying, we implement FAISS (Facebook AI Similarity Search)‍[89], which performs initial clustering and hashing in the feature space to speed up the search process. The similarity between vectors is computed using the Euclidean Distance formula:

[32]$d\left(v_q\mathrm{, }{v}_r\right)\text{ = }\sum _{i\text{ = }1}^{2048}\sqrt{{ (q_i\text{ - }{r}_i)}^2}$,

[33]where q_i and r_i denote the i-th element of the feature vectors v_q and v_r extracted from the query images I_q and I_r, respectively.

4.3 A User-Interface for Image Search in Auction Catalogs 

[34]In order to test our method and eventually enable its use by provenance researchers, we develop a user-interface with the open source package Gradio.‍[90]Fig. 7 shows a screenshot of the interface and an exemplary search. Users first select a query image either by simply uploading it or by using the drag-and-drop functionality. They can then select one of the three models discussed above to determine the feature embedding used to compute image similarities. Before users initiate the search, they can adjust the number of results on the right hand side of the interface. After the search is started, the most similar images to the query are computed using Euclidean Distance (see paragraph 31–33 for more details). Eventually, the search results are displayed underneath the query with additional metadata, including the DOIs of the respective catalogs.

Figure 7: Easy-to-use interface developed by using the open source software Gradio. [Screenshot: Mathias Zinnen / Sabine Lang 2024]

5.1 Detecting Images in Auction Catalogs

[36]To measure the performance of the image detection step, we apply mean average precision as defined in the COCO challenge (COCO mAP).‍[91] COCO mAP is the most widely used evaluation metric for object detection algorithms to date and realizes a trade-off between precision and recall of detected objects by averaging over multiple confidence and overlap thresholds.‍[92] Using the training and validation splits defined in Section 4.1, we train the detection algorithm for 50 epochs. We then evaluate the model on the 33 unseen validation pages and achieve a COCO mAP of 98.6 %. Exemplary predictions from the validation split are illustrated in Fig. 8.

Figure 8: Exemplary predictions from the image detection stage: most images are reliably detected. However, the algorithm has problems differentiating between different object types / elements as evidenced by detected carpets, sculptures, and ornamented text (bottom right). [Graphic: Mathias Zinnen / Sabine Lang 2024, image source: German Sales 2025]

[37]The detected depictions of objects other than paintings could likely be corrected by integrating another training stage specifically targeted at recognizing objects which are not paintings. However, in our use case we can tolerate these false predictions as they will not produce visual features similar to any query image. Thus, we do not expect any negative impact apart from a slightly increased memory usage. Generally, the examples visually confirm the strong performance of the detection system, highlighting that the detection of images in auction catalogs is an easy-to-solve problem for modern detection algorithms.

5.2 Retrieval

5.2.2 Results

[45]We evaluate our method with $Q\text{ = }18$ images. As queries, we select already available digital images of artworks; these are not included in our corpus of auction catalogs. The distribution of corresponding images $R$ in the corpus for the evaluation images is as follows: 15 images of artworks have one matching target image, two images have two matching target images, and one has three. To enable a convenient reproduction of the evaluation, we include all evaluation images as examples in the app.

[46]Table 2 presents top-1 and top-5 accuracies, along with the retrieval mAP for the three pre-training schemes detailed in section 4.1. All models share the same ResNet-50 architecture but vary in their pre-training. No specific fine-tuning was applied to the models for the retrieval task.

Training Dataset	Top-1 Acc.	Top-5 Acc.	mAP
ImageNet	72.2	88.9	73.3
Arts	77.8	83.3	72.2
POSES	33.3	38.7	29.6

Table 2: The table shows the metrics for our method where the extraction network was trained on the ImageNet, Arts or Poses datasets. While for the top-1 accuracy we achieve best results with the Arts model, the ImageNet model scores the best top-5 accuracy and mAP.

[47]Comparing the ImageNet and Arts pre-trained models, we do not see strong differences. While Arts pre-training shows slightly better performance in top-1 accuracy, ImageNet pre-training yields higher top-5 accuracy and retrieval mAP. From the two models evaluated, we cannot conclude that pre-training within the target domain (premodern paintings vs. photographs) increases the retrieval performance. Instead, these results suggest that the features learned from ImageNet classification are sufficiently generic to capture similarity in artistic representations of reality. Qualitative examples which illustrate successful queries are discussed in section 6.

[48]The decline in performance for the model pre-trained for pose estimation is striking. We hypothesize that the features learned for the body posture estimation are too specialized towards their original application to effectively capture relevant aspects of artwork similarity such as image composition. They also fail to recognize similarities between landscapes or inanimate objects, such as clouds, trees, houses, and abstract pictorial elements. To validate this hypothesis, we conducted another evaluation with only six images featuring at least one person very prominently in the image and reported the results in Table 3. This setting led to an increased performance for all models, particularly for the Arts model, which achieved perfect top-1 accuracy. The pose estimation model also displayed significant improvement, especially when compared to the ImageNet model. This indicates that models trained for pose recognition indeed capture artwork similarity better when persons are depicted.

Model	Top-1 Acc.	Top-5 Acc.	Retrieval mAP
ImageNet	85.7	88.1	85.7
Arts	100	100	100
POSES	71.4	71.4	66.3

Table 3: Evaluation of the models’ performances when the query images are restricted to images of artworks with at least one person depicted. We see an increased performance in all models with the Arts-pre-trained models even achieving perfect retrieval metrics. Specifically the model trained for pose estimation performs considerably better and partly closes the gap compared to the two other pre-training schemes.

[49]Two artworks could not be retrieved by any method (see Fig. 9). This drastically lowers the evaluation metrics. To identify the cause, we experimented with different image rotations and compression applied to the query image. However, neither of these measures could resolve the issue. We also confirmed that the images were correctly extracted during the preparatory image detection step (see section 5.1) to eliminate the possibility that it might have failed.

Figure 9: These two artworks drastically lower the quantitative metrics: Adrian Ludwig Richter’s Hirten am Feuer, (c. 1861) on the left and Heinrich Bürkel’s Der Kochelsee mit den Häusern von Schlehdorf (c. 1863 / 1867) on the right. [Graphic: Mathias Zinnen / Sabine Lang 2024, image sources: Ketterer Kunst 2018a; Ketterer Kunst 2018b]

[50]Further investigation revealed that the images cropped from the auction catalogs (see Fig. 10) were of poor quality, characterized by blurriness and noise. However, as illustrated by the two cropped images of artworks in the second row of Fig. 10, poor image quality alone is not a determining factor for the model not being able to find the images. Although they also have a bad quality, the model is able to detect the works in the dataset.

Figure 10: Crops of the target images of artworks as detected in the auction catalogs. Row (a) shows the images which could not be found by any of the extraction methods. The second row (b) shows detections of an artwork by Max Liebermann with a similarly bad image quality which were found (note also the difference in quality and color between the two identical images). [Graphic: Mathias Zinnen / Sabine Lang 2024, image source: German Sales 2025]

[51]In the following, we present two case studies to illustrate how the presented method can be successfully used to search for identical and similar images. This way, the method not only aids provenance researchers with reconstructing an object’s origin but also facilitates the study of visual patterns.

6.1 Case Study: Retrieving Identical Images

[52]In this section we want to demonstrate the potential and performance of our method to search for identical images in auction catalogs, thus assisting with the reconstruction of an object’s provenance. To perform the search, we use the demo-app which we have introduced in section 4.3. In June 2024, the Munich-based auction house Neumeister offered a work by Johann Sperl (1840–1914) entitled Sommerlust.‍[95] Can we find out more about the painting’s provenance using our method? To this end, we perform a search in our collected data set using the app. We upload the query image, set the number of results to ten and initiate the search. The results appear after a few seconds underneath the query image (Fig. 11). The first three results are identical to the query image, suggesting that the painting was offered in three auctions. The first result refers to an auction which took place on April 24, 1928, in the Kunsthaus Lempertz in Cologne.‍[96] The painting was listed as lot 32, titled Kinder auf der Wiese‍[97] and displayed on panel 13.‍[98] Notably, the name differs from the current title, highlighting the issue of a text-based search. Two years later, on November 14, 1930, Sperl’s painting was included in an auction at Paul Cassirer’s Kunstsalon in Berlin.‍[99] The respective auction catalog shows the work on page 88. Back then, however, it was entitled Kind auf der Wiese – a slight variation of the 1928-title.‍[100] The last result points to an auction at Hugo Helbing’s gallery on March 26, 1927 – the earliest auction date in our list of results.‍[101] The corresponding catalog lists the work as Sommerlust (lot 106) on page 22‍[102] and includes a reproduction of the painting on panel 16.‍[103] The example of Johann Sperl demonstrates the efficiency of our method and is indeed confirmed by the provenance information given on the website of Neumeister which mentions all three provenances.‍[104]

[53]If we look at the calculated feature distance, we notice that the first result (Lempertz) had a distance of 257, the second (Cassirer) a distance of 259 and the last (Helbing) a calculated distance of 448 to the query image. All images show the same content to the query; why do they have a different distance then? Here is a possible explanation: The coloration of the images vary slightly as well as the rotation (Helbing). These deviations from the query image might cause these observed distances, especially to the Helbing example.‍[105]

Figure 11: Search results for Johann Sperl’s Sommerlust: The work appeared in a catalog in (a) 1928, (b) 1930 and (c) 1927. [Graphic: Mathias Zinnen / Sabine Lang 2024]

6.2 Case Study: Retrieving Similar Images

[54]For many visual disciplines such as art history it is not only relevant to find identical but also similar images. Finding these similarities offers insights into reception processes over time and space, prevailing taste, artistic networks and focuses of auction houses and collectors. Can we use our method to address the following questions:

1. In which contexts does the motif of the boat appear?
2. What can be said about the drawing style of Max Liebermann (1847–1935)?

6.2.1 The Motif of the Boat

[55]In order to answer the first question, we choose Andreas Achenbach’s (1815–1910) painting Fischerboot auf stürmischer See (1895) as a query image and initiate the search (Fig. 12). The first result is identical to the query, therefore we disregard this image in our analysis.

Figure 12: The first search results (excluding the first / identical) for Andreas Achenbach’s Fischerboot auf stürmischer See: The painting was offered at auctions in (a) 1936, (b) 1931 and (c) 1928. [Graphic: Mathias Zinnen / Sabine Lang 2024]

[56]The second image (a) shows a boat close to the shore, embedded within a mountain view. It was painted by Max Buri (1868–1915) and offered as Brienzersee (1894) in an auction hosted by the Gallery Neupert in Zurich on April 4, 1936.‍[106] Similar to the query, the boat is shown at a central position; however, the stormy water and atmosphere is replaced by a calmness and tranquility. In addition, Buri’s painting is devoid of any human life. The third result shows a similar visual pattern: A boat floats on a calm lake at a central position, surrounded by a deserted landscape, mountain range, small houses and shore. The image was painted by Otto Frölicher (1840–1890) and offered as Barken in an auction catalog of G. & L. Bollag in Zurich in 1931.‍[107] These first results suggest a content similarity, possibly influenced by the motifs of the boat and / on water. The last result discussed in this case study stems from a catalog of the Kunstsalon Dr. Störi in Zurich; the respective auction took place in March 1928.‍[108] The image shows a painting by Guillaumin Armand (1841–1928) and is entitled Der Kran.‍[109] We see a crane vessel in the middle ground, another one is visible in the background. The right side of the image displays piles of sand and two standing figures who turn their back towards the viewer. Compared to the first results the image conveys a sense of motion and liveliness, this is accentuated by the painting style (the reproduction suggests visible brushstrokes and different color regions). In addition to the content similarity, the last example thus suggests a similarity based on the mood and effect of the image. This concise study suggests that the motif of the boat mainly appears within a landscape setting which shows very few signs of human life. If we look at their time of creation and the life dates of the artists, we can observe that all paintings were created in the late 19^th century and early 20^th century (Armand), thus suggesting a preference for the motif during that time.

6.2.2 The style of Max Liebermann

[57]Max Liebermann is one of the most important artists of the 19^th and 20^th centuries; he is known for his elaborate oil paintings as well as for his delicate drawings and sketches. We utilize the suggested method to gain more insight into Liebermann’s drawings by looking at similar images (see question two). We take his chalk drawing Einholung Bismarcks in Berlin (1890) as a query. Interestingly, all first results stem from an auction held at the Kunstsalon of Paul Cassirer in March 1925 (Fig. 13).‍[110] The title of the catalog already indicates that the auction only included drawings by Liebermann (namely 316 works).‍[111] While this result does not enable us to study similar drawings by other artists, it is interesting, because it suggests a strong individual artistic style. According to the results Liebermann is most similar to himself. Further analysis might study the drawings in more detail and look at the motifs, composition or textures he used.

[58]Both case studies demonstrate the potential of our method to search for identical and similar images in a large dataset, thus not only assisting provenance researchers but any discipline interested in visual patterns over time and space. The examples of Achenbach and Liebermann highlighted that by searching for similar images, we can address a variety of research questions which go beyond the provenance of objects.

Figure 13: Search results for Liebermann’s drawing Einholung Bismarcks in Berlin (1890). All results appear in an auction catalog published by the Kunstsalon Paul Cassirer in March 1925. [Graphic: Mathias Zinnen / Sabine Lang 2024]

7.1 General Remarks on Similarity

[60]Comparative processes are integral to determining similarity.‍[120] Humans, for example, compare paintings to reach a conclusion about their similarity. This process is guided by various criteria, which are manifold, subjective and often difficult to grasp. These criteria might include specific motifs, the image composition, color, forms, texture or artist, time period and location and thus encompass both internal and external criteria of the image. Our proposed method suggests a similarity based on image-inherent, visual criteria. This similarity might be determined by global image structures, numerous objects and their arrangement, or singular objects. Following this approach, we understand similarity as determined by criteria inherent and visible in an image. Thus, external criteria described for example by metadata, such as artist, title, year, or technique, are disregarded.

[61]While our method focuses on internal criteria, it still allows us to explore different similarity dimensions: By selecting a suitable pre-training method, the computer scientist can try to influence the type of similarity underlying the retrieval process. For example, we can assume that an extraction network trained to recognize body postures will put more weight on this aspect of similarity. Therefore, the process of attesting similarity is already influenced during pre-training. Fig. 14 illustrates this: A search for Ludwig von Zumbusch’s Einsames Land (1896) leads to different results depending on the embedding type selected by the user. While results for the models ImageNet and Arts show a clear preference for landscapes, the Poses model prefers images with persons and objects. In general, it can be observed that using the Poses model leads to worse results than utilizing the ImageNet and Arts models. We illustrate this by searching for Sperl’s Sommerlust using all three embedding types (see Fig. 15). Models trained on ImageNet and Arts retrieve all three instances of Sperl’s painting in the dataset, while the Poses model only finds one instance.

Figure 14: Search results using different embedding types; all models were trained on different data thus leading to different search results and allowing to focus on diverse similarity criteria (so our hypothesis). [Graphic: Mathias Zinnen / Sabine Lang 2024]

Figure 15: Search results using different embedding types; the models ImageNet and Arts perform best for the task of finding identical images to Johann Sperl’s Sommerlust. [Graphic: Mathias Zinnen / Sabine Lang 2024]

[62]In our method, we derive quantifiable, vector-shaped representations of image content and calculate the distance between two vectors using an arbitrary distance metric (i. e. Euclidean Distance). We assume that this distance approximates the degree of similarity between the images. However, the computation of the distance between two feature vectors does not provide any insights into interpretable similarity criteria; knowing the similarity distance function does not explain why two pictures might be perceived as similar, while others are seen as different.‍[121] To understand what makes artworks similar from a data-driven perspective, we must consider the properties of the feature space. This requires discussing the function that translates pixel-based image representations into feature vectors. We refer to this process as quantification.

7.2 Quantification

[63]According to the Duden (dictionary of the German language), quantification means the transformation of qualities into quantities, for example the properties of something (here: an image) in numbers and measurable values.‍[122] In this paper, quantification can relate to two different aspects, namely the process of translating an image into feature vectors and the fact that similarity is computed which requires a quantification process.

[64]In order to search for identical and similar images, we utilize a neural network to extract feature vectors which are essentially numerical representations of a group of features which describe an image (such as colors, edges, or objects). Thus, these features quantify and represent the content of an image. During the search process, the feature vector of the given query image is compared to all feature vectors stored in the feature database and their distances are calculated using the Euclidean Distance. The closer the distance, the more similar the images are according to the algorithm. Therefore, similarity is not based on a subjective impression, but on measurable, comparable, and objective numerical values. Since feature vectors are extracted from input data – in our case digital images – the question arises whether the similarity measure depends on the image quality and can therefore vary even for images which appear identical or similar to the human eye (see case study on Johann Sperl, section 6.1). For example, different reproduction and digitization techniques might result in different color representations or image contrasts which might influence the feature vectors and calculations respectively. Fig. 10 shows that even for the same image the quality and color of the reproduction differs significantly.

[65]Cognitive psychology picks up on the idea that similarity is founded in and expressed by numerical values. Psychological processes are modeled using computers; this means that only processes that can be expressed in variables and algorithms are represented. Accordingly, features are variables to which numerical values are assigned allowing computation. Essentially, for Cognitive Psychology, similarity is characterized as a ›feature overlap‹‍[123]: »People notice that a number of objects overlap substantially and proceed to form a category to include these items. […] Categorization is justified by the observation that objects tend to cluster in terms of their attributes, be these physical features, linguistic labels […].«‍[124] As Winkler concludes in his book on similarity, the computer provides the frame in which cognitivists think about similarity.‍[125] The same can be said about our project, since the method guides and essentially dictates how we think and write about similarity.

7.3 Process of Abstraction

[66]In his book, Winkler emphasizes that comparison and similarity separates things in aspects (features), which are similar and dissimilar.‍[126] Human observation, according to him, does not remain with the things themselves, but rather moves on to their properties and characteristics.‍[127] Accordingly, similarity abstracts something from things (the similar thing eventually results in a form; form then plays a crucial role for him). Thus, for Winkler, similarity is based on mechanisms of abstraction.‍[128] Accordingly, we understand the ›process of abstraction‹ as a process in which certain criteria of the input data are abstracted to determine if things are similar or dissimilar. As described previously in section 4.2, an image-based search requires the extraction of features from the data. Ideally, these features describe and represent the (content of the) data. This process aligns with Winkler’s conception of similarity, as it is based on specific criteria which are context dependent. This process of abstraction in the machine mirrors the human approach to recognizing similarities, most evidently illustrated by the conversion of input data into a lower-dimensional vector representation during feature extraction. This representation retains the essential information needed for a given task – in this case, retrieval – while disregarding unimportant information. This process is thus associated with a loss of (certain) information and potentially with a loss of possible similarity criteria.

[67]Winkler also raises the question wether similarity implies that our attention is focused on certain criteria: What exactly is guiding our attention and selection?‍[129] For humans, the cultural and societal context as well as personal experiences and preferences play a crucial role.‍[130] As mentioned before, the computer scientist can try to influence the type of similarity underlying the retrieval process by selecting a suitable pre-training method. We can assume, for example, that an extraction network trained to recognize body postures will put more weight on these similarity criteria (see results Fig. 14). If we interpret feature extraction as an abstraction in Winkler’s sense, selecting a specific pre-training method determines the understanding of similarity underlying the retrieval system. Selecting a specific method can then be correlated to the type of information we are ›omitting‹ during the translation of pixels into a feature vector. Whenever we select a specific method for this translation, we also decide which features are deemed irrelevant by the models. Similarity in this way is always influenced by the choice we make regarding the pre-training method. This means that the attention and selection process of the network is guided by human input, often motivated by the research question, task or personal interests. In comparison to humans, the attention and selection of similarity criteria in this context is much more conscious and goal-driven.

[68]We conclude the following: Attesting similarity requires comparative processes separating things into aspects (features), which are similar and dissimilar‍[131], which in the context of this paper refer to image-internal components. These criteria are abstracted from the image and quantified, thus becoming measurable. We also noted that the image quality might influence the similarity measure. The conversion of input data into a lower-dimensional vector representation during feature extraction is associated with a loss of information and therefore might result in a loss of similarity dimensions. We also elaborated on the fact that the selection of the pre-training method can influence the type of similarity underlying the retrieval process. This section thus highlighted that the computer guides as well as limits our understanding of similarity. Discussing similarity in the context of this paper, with a particular focus on quantification and the process of abstraction, provided interesting insights and leaves room for further discussion and research.