Object Detection is one of the fundamental tasks in computer vision and maintains its prominence as one of the most active research areas. Cf. Zou et al. 2023. It goes beyond image-level classification by requiring not only the classification but also the localization of objects within an image. Image-level classification, in contrast, involves assigning a label to an entire image, which generally poses less complexity. Classification benchmarks get their complexity from the amount of categories that need to be distinguished, as exemplified by the classic ImageNet Cf. Russakovsky 2015. benchmark, which required differentiation among 1,000, or in some versions, even 21,000 classes. Cf. Ridnik et al. 2021.

In detection, two-stage methods like the Region-based Convolutional Neural Network (R-CNN) family, Cf. Girshick et al. 2014; Girshick 2015; Ren et al. 2016. have dominated the field, offering strong accuracy at the cost of increased runtime. In contrast, one-stage methods like You only look once (YOLO) Cf. Redmon et al. 2016. or Single Shot MultiBox Detector (SSD) Cf. Liu et al. 2016. have traditionally prioritized speed over precision, although the performance gap has narrowed with more recent architectures. Recently, transformer-based detection architectures, particularly those based on the DEtection TRansformer (DETR), Cf. Carion 2020. have become the dominant paradigm. However, one-stage and two-stage methods maintain their relevance. Notably, recent versions of YOLO continue to show competitive performances on many detection benchmarks and are very popular due to their easy usage, efficient training and inference times. For our purposes, the complexity and performance of modern detection benchmarks far exceed our demands. The task of detecting images in printed layouts is straightforward and can nearly be considered as a resolved problem within the field of computer vision. Consequently, we favor the YOLO-v8 Cf. Jocher et al. 2023. architecture over more powerful but slower modern two-stage or DETR-based architectures.

While applying object detection to artworks is less common than in natural image analysis, it is recognized as a well-established practice within digital humanities. Pioneering work in object detection for artworks was conducted by Elliot J. Crowley and Andrew Zisserman. They trained detection algorithms on the photographic dataset Pascal VOC Cf. Everingham 2010. and employed transfer learning to apply the trained algorithms for identifying objects such as vehicles or animals in artworks. Cf. Crowley / Zisserman 2014; Crowley / Zisserman 2015; Crowley / Zisserman 2016. Another line of research focused on the detection of smell-related objects in artworks, Cf. Zinnen et al. 2022a. a topic addressed with the introduction of the ODOR challenge and dataset. Special cases of object detection, such as the detection of depicted faces Cf. Bengamra et al. 2021; Mermet et al. 2020. and persons Cf. Westlake et al. 2016. have a great relevance for art history, e.g. for studying portraiture, figure painting, or genre painting. Westlake et al. introduced a dataset of annotated persons depicted across a wide range of artistic styles. Cf. Westlake et al. 2016. Closely related to that is the application of pose estimation, which typically requires an initial person detection stage. Pose estimation was used to analyze and cluster artworks based on body postures, Cf. Impett / Moretti 2017; Impett / Süsstrunk 2016; Bell / Impett 2019; Impett 2020. hand gestures, Cf. Bernasconi et al. 2023. and overall image composition, Cf. Madhu et al. 2020; Madhu et al. 2023. or to recognize smell-related Cf. Zinnen et al. 2023. and sensory Cf. Zinnen et al. 2025. gestures.

A common challenge for object detection in artworks is the sparsity of annotated datasets. Gonthier et al. addressed this by employing a weakly supervised training approach, using image-level labels before assessing the method on instance-level labels. Cf. Gonthier et al. 2018; Gonthier et al 2022. Additionally, they shifted their focus from detecting modern categories to identifying objects with art historical relevance. This thread was picked up by Marinescu et al., who adapted COCO categories to be consistent with historical contexts, Cf. Marinescu et al. 2020. and by Reshetnikov et al., who compiled a large dataset categorized by art historical themes. Cf. Reshetnikov et al. 2023.

Another approach to address the challenge of data sparsity is the use of style transfer to mimic artistic object representations. Cf. Sanakoyeu et al. 2018; Gatys et al. 2015; Gatys et al. 2016. This technique leverages existing annotations from photographic data to obtain an adaptation to the artistic target domain using transfer learning. This strategy is commonly employed for various tasks, including pose estimation, Cf. Madhu et al. 2023; Springstein et al. 2022. emotion recognition, Cf. Patoliya et al. 2024. or painting captioning, Cf. Lu et al. 2021. and has also been successfully applied to object detection. Cf. Jeon et al. 2020; Kadish et al. 2021; Smirnov / Eguizabal 2018.

The issue of data sparsity, at least for person detection and pose estimation, has been significantly mitigated with the introduction of the Human-Art dataset. Cf. Ju et al. 2023. This dataset features 50,000 images of artistic creations like sculptures, paintings, or cartoons annotated with the position of persons and pose estimation keypoints.

The retrieval of similar depictions of identical objects from auction catalogs can be contextualized within several closely related research areas, with Content-Based Image Retrieval (CBIR) being the most prominent. CBIR, often used synonymously with ›Image Retrieval‹, can be defined as the problem of searching for relevant images in a database given a query image based on visual features. Cf. Chen et al. 2022.

The retrieval process can also be framed as a problem of Near Duplicate Detection (NDD) when considering different depictions of the same object as near duplicates. Primarily aimed at identifying digital copies or manipulated images, NDD differs from CBIR in its application focus. While CBIR is a relatively open task that accommodates a wide range of queries and image similarities, NDD specifically targets the duplicate nature of the query and target images. Cf. Thyagharajan / Kalaiarasi 2021. Irrespective of whether it is labeled as NDD or CBIR, a retrieval system typically operates in two phases: A preparatory phase, where a database of quantifiable image representations is created, and an online phase, where this database is queried. Cf. Zhou et al. 2017. These queries are not limited to image inputs alone; for example, a shared image-language embedding space, as provided by Contrastive Language-Image Pre-Training (CLIP), Cf. Radford et al. 2021. enables cross-modal querying. Cf. Garcia / Vogiatzis 2018. However, in the context of our application, we are focusing on image queries. In this scenario, the online phase entails a feature extraction similar to that during the database creation. This process maps the query input into the same embedding space as the already processed image corpus, enabling the computation of distances between the query features and the previously extracted features to assess their similarity. As computing the feature distance across all stored feature vectors can become intractable for large image corpora, the embedding space is often clustered or indexed to enable efficient querying. Chen et al. categorize methods based on their reliance on off-the-shelf models, originally trained for different tasks than retrieval, versus methods that incorporate an additional training stage dedicated to fine-tuning the models specifically for image retrieval. Cf. Chen et al. 2022. In our application, we employ the simpler approach by using pre-trained models to extract features. Moving forward, we plan to improve our system by integrating more elaborate training schemes specifically designed for retrieval tasks.

The process of fine-tuning models for retrieval is closely related to the field of metric learning. However, the term metric learning is often discussed more from a technical standpoint than from its practical application. Metric learning involves mapping data into an embedding space where similar data points are closer together and dissimilar data points are further apart, guided by a distance metric such as the Euclidean norm. Cf. Musgrave et al. 2020. Metric learning can be used for many purposes such as classifying samples according to their nearest neighbors in the embedding space or in self-supervised pre-training. However, retrieval can be considered its most natural application as it typically entails the search for the closest query result in the feature space. This way, retrieval can be framed as an instance-level open-set classification, where the query has to be matched to its closest neighbors in the feature space.

Independent of whether models are specifically trained for retrieval is the consideration of which features are used for the distance computation. Before deep learning, approaches typically relied on local features obtained with methods such as Scale-Invariant Feature Transform (SIFT). Cf. Lowe 2004. These local features can be aggregated into a global descriptor using methods like Bag-of-visual-Words (BoW). Cf. Csurka et al. 2004. Such combinations of classical local features and their global aggregation have been applied in various contexts, including the recognition of CD-covers, Cf. Nistér / Stewénius 2006. object retrieval Cf. Philbin et al. 2008. and product search. Cf. He 2012.

The question of feature aggregation remains relevant in modern, deep-learning-based approaches: Even when employing neural networks, various feature representations can be considered, involving different scales For example Sun et al. 2015 or Tolias et al. 2015. (intra-model) or different models Cf. Yokoo et al. 2020. (inter-model). Effective fusion of different feature levels can lead to a more meaningful feature representation. Cf. Chen et al. 2022. For our application, a comparatively simple strategy is sufficient. We simply use the flattened output of the last layer of a pre-trained Convolutional Neural Network (CNN) to represent the image contents and compute similarities. Future research could explore different scales and more elaborate feature fusion, but also revisit classical feature extraction methods such as SIFT, and analyze their impact on the type of similarity encoded in the query results.

In the fields of digital humanities and computational cultural heritage, research formerly centered around the visual retrieval of similar images or image parts to identify patterns across numerous artworks and uncover relations between them. Cf. Seguin et al. 2016; Ufer et al. 2021; Shen et al. 2019; Castellano et al. 2021. Eventually, projects and works also focused on the development of interfaces to search for images or image parts. Cf. Ufer et al. 2021; Springstein et al. 2021; Offert / Bell 2024; PortApp 2025. Besides feature vector distances, works also explored the application of different metrics to capture diverse aspects of image similarity, for example color concepts, Cf. Yelizaveta et al. 2005. image composition, Cf. Madhu et al. 2023. body posture, Cf. Impett / Moretti 2017; Bell / Impett 2019. or even symbolic meaning. Cf. Sartini et al. 2023; Sartini / Gangemi 2021.

The wealth of works which apply computational methods for art analysis and understanding is reflected in several review papers. While Bengamra et al. Cf. Bengamra et al. 2024. provide a summary of significant computer vision applications for art, they specifically focus on object detection. An overview of datasets and works on the task of recognizing and extracting patterns in visual arts using deep learning is given in a paper by Giovanna Castellano and Gennaro Vessio. Cf. Castellano / Vessio 2021. Another review provides an overview of how computational methods are used for classification, object detection, similarity retrieval or multimodal representations, among others. Cf. Cetinic / She 2022. Amalia Foka then presented past computer vision applications for art historical research and future possibilities. Cf. Foka 2021.

Computational methods have also been applied to art market research: Utilizing a subset of over 267,000 sale transactions from the Getty Provenance Index and complex network science, Schich et al. studied the history of the art market and collection dynamics to reveal social, temporal, and spatial networks. Cf. Schich et al. 2017. Fletcher et al. studied the art market in London between 1850 and 1914 on the basis of complementary datasets and visualizations. Cf. Fletcher et al. 2012. Scheithauer et al. suggested a two-step pipeline to analyze the layout and content of auction sales catalogs utilizing object detection and text sequence labeling models. Cf. Scheithauer et al. 2024. A similar approach and goal to the one presented in this paper was announced in 2021. Then, the Fraunhofer Institute published a report which informed about a feasibility study. The study developed AI-methods for image search in auction catalogs, enabling a successful comparison between current and historical images. Cf. Vicente-Garcia 2021. A second feasibility study, which again highlighted the success of the methods, was published in the magazine in 2024. Cf. Vicente-Garcia 2024. In the humanities, the art market has been a research topic for many years. Numerous publications on the art market between 1901 and 1945 in Germany and other German-speaking countries focus on individual art dealers Cf. Hoffmann / Kuhn 2016., auction houses Cf. Hopp 2012., or the valuation and price development. Cf. Jeuthe 2014. An overview of relevant literature is given in the bibliographies Cf. Bommert 2019. and . Cf. Bähr 2013.

The relatively new field of digital provenance research studies the impact of digitality on provenance research, focusing on chances and challenges, and uses digital methods for analyzing provenance data. Cf. Lang 2023a. Works report on the development of research databases Cf. Werner 2020. , the presentation and communication of provenance information and research results online Cf. Haffner 2020; Haffner 2019. , the future of provenance research and digital infrastructures in Germany, especially focusing on tendencies and desiderata. Cf. Hopp 2018. Special attention has been paid to aspects of incompleteness and vagueness in provenance research. Cf. Lang 2023b; Mariani 2022. Rother et al. study the transformation of unstructured provenance records into Linked Open Data and how computer-based methods can be utilized for a comprehensive analysis of provenance records. Cf. Rother et al. 2023; Rother et al. 2022.

Before we can store the feature representations of objects depicted in the auction catalogs, we must prepare the catalogs and crop the illustrations. First, we parse the PDF-files of the two bibliographies Cf. Bähr 2013; Bommert 2019. and convert the information into a structured, machine-readable format. Table 1 lists the extracted metadata fields describing the catalogs.

In a second step, we filter the catalogs using the metadata and process them further. Due to computational constraints, we initially limit our scope to a smaller number of catalogs, focusing on locations in Switzerland. Additionally, we narrow our selection to catalogs covering sales of ›Gemälde‹ as specified by the keywords in the bibliographies. This process results in a set of 86 catalogs, with an additional 25 catalogs which are included later for evaluation. We plan to extend the dataset in future work, eventually covering all catalogs in .

Using the links provided in the bibliographies, we download the catalogs from the online services of the Heidelberg Library and convert each page to an image file using the PyMuPDF Cf. PyMuPDF 2024. library. To minimize memory use, we store the pages as JPEG-files with 95 % export quality compression. Subsequently, we employ the YOLO-v8x Cf. Jocher et al. 2023. object detection algorithm to automatically detect images of objects on the pages, crop the detected images and save them separately. To train the detection algorithm, we manually label a set of 181 catalog pages with the position for 316 depicted objects. We split 33 for validation (containing 60 objects), and train the algorithm with the remaining 148 annotated pages (256 objects). Finally, the cropped object depictions are converted to grayscale and rotated to align with the longer edge, ensuring a consistent representation in terms of print color and orientation. Fig. 5 provides a complete overview of the preprocessing steps described above. 

Fig. 6 illustrates the process of feature extraction and query employed for the reverse image search in the auction catalogs. To prepare the system, the previously cropped images from the auction catalogs are used as an input to various feature extraction methods to compile a database of feature vectors as shown in the top row of Fig. 6. Specifically, we employ three different ResNet-50 Cf. He et al. 2016. feature extractors, each pre-trained for different tasks:

The motivation behind selecting these three pre-training schemes is to evaluate whether the choice of extraction method can influence which images are found to be similar. The underlying hypothesis is that the embeddings extracted from the images can be related back to the task the extraction model was originally trained for. Accordingly, pose recognition embeddings would be expected to emphasize the body posture of depicted persons. ImageNet embeddings, on the other hand, have been shown to generalize towards various tasks and put a specific emphasis on texture versus shape. Cf. Geirhos et al. 2018. The model trained for object detection in premodern artworks was selected to test whether the availability of artistic object representations in the training data supports the models in extracting more relevant embeddings for artwork similarity.

After deploying the application, users can upload arbitrary images to the system. These images are fed to the feature extractor, similarly to the process during the creation of the feature database. Afterwards, we compute vector distances $d\left(v_q\mathrm{, }{v}_i\right)$ between the query vector $v_q$ and the $n$ precomputed vectors $v_i\mathrm{, }i\mathrm{ \in }\left\{\mathrm{1, 2}\mathrm{, }\mathrm{...}\mathrm{, }n\right\}$ and return the $k$ images $I_1\mathrm{, }{I}_2\mathrm{, }\mathrm{...}\mathrm{, }{I}_k$ where the distance between the feature and query vectors are lowest (see bottom row of Fig. 6).

Our approach simplifies the process of image retrieval by using the flattened final feature vector obtained just before the final classification layer of the ResNet-50 architecture, which typically has 2048 dimensions. Incorporating more elaborate strategies and vector representations that account for multiple scales might further enhance performance. This improvement is a potential direction for future work, although it is beyond the scope of this current study.

For efficient querying, we implement FAISS (Facebook AI Similarity Search) Cf. Johnson et al. 2019., which performs initial clustering and hashing in the feature space to speed up the search process. The similarity between vectors is computed using the Euclidean Distance formula:

$d\left(v_q\mathrm{, }{v}_r\right)\text{ = }\sum _{i\text{ = }1}^{2048}\sqrt{{ (q_i\text{ - }{r}_i)}^2}$ ,

where qi and ri denote the i-th element of the feature vectors vq and vr extracted from the query images Iq and Ir, respectively.

In order to test our method and eventually enable its use by provenance researchers, we develop a user-interface with the open source package Gradio. Cf. Abid et al. 2019. Fig. 7 shows a screenshot of the interface and an exemplary search. Users first select a query image either by simply uploading it or by using the drag-and-drop functionality. They can then select one of the three models discussed above to determine the feature embedding used to compute image similarities. Before users initiate the search, they can adjust the number of results on the right hand side of the interface. After the search is started, the most similar images to the query are computed using Euclidean Distance (see paragraph 31–33 for more details). Eventually, the search results are displayed underneath the query with additional metadata, including the DOIs of the respective catalogs.

To measure the performance of the image detection step, we apply mean average precision as defined in the COCO challenge (COCO mAP). Cf. Lin et al. 2014. COCO mAP is the most widely used evaluation metric for object detection algorithms to date and realizes a trade-off between precision and recall of detected objects by averaging over multiple confidence and overlap thresholds. A detailed definition is beyond the scope of this work and can be found at Common Objects in Context 2024. Using the training and validation splits defined in Section 4.1, we train the detection algorithm for 50 epochs. We then evaluate the model on the 33 unseen validation pages and achieve a COCO mAP of 98.6 %. Exemplary predictions from the validation split are illustrated in Fig. 8.

The detected depictions of objects other than paintings could likely be corrected by integrating another training stage specifically targeted at recognizing objects which are not paintings. However, in our use case we can tolerate these false predictions as they will not produce visual features similar to any query image. Thus, we do not expect any negative impact apart from a slightly increased memory usage. Generally, the examples visually confirm the strong performance of the detection system, highlighting that the detection of images in auction catalogs is an easy-to-solve problem for modern detection algorithms.

In this section we want to demonstrate the potential and performance of our method to search for identical images in auction catalogs, thus assisting with the reconstruction of an object’s provenance. To perform the search, we use the demo-app which we have introduced in section 4.3. In June 2024, the Munich-based auction house Neumeister offered a work by Johann Sperl (1840–1914) entitled . Cf. Neumeister 2024. Can we find out more about the painting’s provenance using our method? To this end, we perform a search in our collected data set using the app. We upload the query image, set the number of results to ten and initiate the search. The results appear after a few seconds underneath the query image (Fig. 11). The first three results are identical to the query image, suggesting that the painting was offered in three auctions. The first result refers to an auction which took place on April 24, 1928, in the Kunsthaus Lempertz in Cologne. Cf. Kunsthaus Lempertz 1928a. The painting was listed as lot 32, titled Cf. Kunsthaus Lempertz 1928b. and displayed on panel 13. Cf. Kunsthaus Lempertz 1928c. Notably, the name differs from the current title, highlighting the issue of a text-based search. Two years later, on November 14, 1930, Sperl’s painting was included in an auction at Paul Cassirer’s in Berlin. Cf. Kunstsalon Paul Cassirer 1930a. The respective auction catalog shows the work on page 88. Back then, however, it was entitled  – a slight variation of the 1928-title. Cf. Kunstsalon Paul Cassirer 1930b. The last result points to an auction at Hugo Helbing’s gallery on March 26, 1927 – the earliest auction date in our list of results. Cf. Auktionshaus Hugo Helbing 1927a. The corresponding catalog lists the work as (lot 106) on page 22 Cf. Auktionshaus Hugo Helbing 1927b. and includes a reproduction of the painting on panel 16. Cf. Auktionshaus Hugo Helbing 1927c. The example of Johann Sperl demonstrates the efficiency of our method and is indeed confirmed by the provenance information given on the website of Neumeister which mentions all three provenances. Cf. Neumeister 2024.

If we look at the calculated feature distance, we notice that the first result (Lempertz) had a distance of 257, the second (Cassirer) a distance of 259 and the last (Helbing) a calculated distance of 448 to the query image. All images show the same content to the query; why do they have a different distance then? Here is a possible explanation: The coloration of the images vary slightly as well as the rotation (Helbing). These deviations from the query image might cause these observed distances, especially to the Helbing example.It is unclear whether the coloration of the pages is present in the originals or a result of the digitization process and only visible in the digital reproductions.

For many visual disciplines such as art history it is not only relevant to find identical but also similar images. Finding these similarities offers insights into reception processes over time and space, prevailing taste, artistic networks and focuses of auction houses and collectors. Can we use our method to address the following questions:

Comparative processes are integral to determining similarity.In his book, Winkler mentions that attesting similarity requires comparative processes; these processes, however, can happen unconsciously (cf. Winkler 2021, pp. 111–112). Humans, for example, compare paintings to reach a conclusion about their similarity. This process is guided by various criteria, which are manifold, subjective and often difficult to grasp. These criteria might include specific motifs, the image composition, color, forms, texture or artist, time period and location and thus encompass both internal and external criteria of the image. Our proposed method suggests a similarity based on image-inherent, visual criteria. This similarity might be determined by global image structures, numerous objects and their arrangement, or singular objects. Following this approach, we understand similarity as determined by criteria inherent and visible in an image. Thus, external criteria described for example by metadata, such as artist, title, year, or technique, are disregarded.

While our method focuses on internal criteria, it still allows us to explore different similarity dimensions: By selecting a suitable pre-training method, the computer scientist can try to influence the type of similarity underlying the retrieval process. For example, we can assume that an extraction network trained to recognize body postures will put more weight on this aspect of similarity. Therefore, the process of attesting similarity is already influenced during pre-training. Fig. 14 illustrates this: A search for Ludwig von Zumbusch’s (1896) leads to different results depending on the embedding type selected by the user. While results for the models ImageNet and Arts show a clear preference for landscapes, the Poses model prefers images with persons and objects. In general, it can be observed that using the Poses model leads to worse results than utilizing the ImageNet and Arts models. We illustrate this by searching for Sperl’s using all three embedding types (see Fig. 15). Models trained on ImageNet and Arts retrieve all three instances of Sperl’s painting in the dataset, while the Poses model only finds one instance.

In our method, we derive quantifiable, vector-shaped representations of image content and calculate the distance between two vectors using an arbitrary distance metric (i. e. Euclidean Distance). We assume that this distance approximates the degree of similarity between the images. However, the computation of the distance between two feature vectors does not provide any insights into interpretable similarity criteria; knowing the similarity distance function does not explain why two pictures might be perceived as similar, while others are seen as different. While we can influence the distance computation by selecting a different distance metric (i. e. Euclidean Distance) to some degree, the general tendency will remain similar. To understand what makes artworks similar from a data-driven perspective, we must consider the properties of the feature space. This requires discussing the function that translates pixel-based image representations into feature vectors. We refer to this process as quantification.

According to the (dictionary of the German language), quantification means the transformation of qualities into quantities, for example the properties of something (here: an image) in numbers and measurable values. Cf. Dudenredaktion (ed.) 2024. In this paper, quantification can relate to two different aspects, namely the process of translating an image into feature vectors and the fact that similarity is computed which requires a quantification process.

In order to search for identical and similar images, we utilize a neural network to extract feature vectors which are essentially numerical representations of a group of features which describe an image (such as colors, edges, or objects). Thus, these features quantify and represent the content of an image. During the search process, the feature vector of the given query image is compared to all feature vectors stored in the feature database and their distances are calculated using the Euclidean Distance. The closer the distance, the more similar the images are according to the algorithm. Therefore, similarity is not based on a subjective impression, but on measurable, comparable, and objective numerical values. Since feature vectors are extracted from input data – in our case digital images – the question arises whether the similarity measure depends on the image quality and can therefore vary even for images which appear identical or similar to the human eye (see case study on Johann Sperl, section 6.1). For example, different reproduction and digitization techniques might result in different color representations or image contrasts which might influence the feature vectors and calculations respectively. Fig. 10 shows that even for the same image the quality and color of the reproduction differs significantly.

Cognitive psychology picks up on the idea that similarity is founded in and expressed by numerical values. Psychological processes are modeled using computers; this means that only processes that can be expressed in variables and algorithms are represented. Accordingly, features are variables to which numerical values are assigned allowing computation. Essentially, for Cognitive Psychology, similarity is characterized as a ›feature overlap‹ Cf. Winkler 2021, pp. 96–97.: People notice that a number of objects overlap substantially and proceed to form a category to include these items. […] Categorization is justified by the observation that objects tend to cluster in terms of their attributes, be these physical features, linguistic labels […]. Anderson 1991, p. 411. As Winkler concludes in his book on similarity, the computer provides the frame in which cognitivists think about similarity. Cf. Winkler 2021, p. 98. The same can be said about our project, since the method guides and essentially dictates how we think and write about similarity.

In his book, Winkler emphasizes that comparison and similarity separates things in aspects (features), which are similar and dissimilar. Cf. Winkler 2021, p. 257. Human observation, according to him, does not remain with the things themselves, but rather moves on to their properties and characteristics. Cf. Winkler 2021, p. 93. Accordingly, similarity abstracts something from things (the similar thing eventually results in a form; form then plays a crucial role for him). Thus, for Winkler, similarity is based on mechanisms of abstraction. Cf. Winkler 2021, pp. 257–258. Accordingly, we understand the ›process of abstraction‹ as a process in which certain criteria of the input data are abstracted to determine if things are similar or dissimilar. As described previously in section 4.2, an image-based search requires the extraction of features from the data. Ideally, these features describe and represent the (content of the) data. This process aligns with Winkler’s conception of similarity, as it is based on specific criteria which are context dependent. This process of abstraction in the machine mirrors the human approach to recognizing similarities, most evidently illustrated by the conversion of input data into a lower-dimensional vector representation during feature extraction. This representation retains the essential information needed for a given task – in this case, retrieval – while disregarding unimportant information. This process is thus associated with a loss of (certain) information and potentially with a loss of possible similarity criteria.

Winkler also raises the question wether similarity implies that our attention is focused on certain criteria: What exactly is guiding our attention and selection? Cf. Winkler 2021, p. 295. For humans, the cultural and societal context as well as personal experiences and preferences play a crucial role.American philosopher Nelson Goodman famously stated that Circumstances alter similarities (quoted in Kimmich 2017, p. 24), thus emphasizing the context-dependency of similarity judgments (cf. Winkler 2021, p. 94). As mentioned before, the computer scientist can try to influence the type of similarity underlying the retrieval process by selecting a suitable pre-training method. We can assume, for example, that an extraction network trained to recognize body postures will put more weight on these similarity criteria (see results Fig. 14). If we interpret feature extraction as an abstraction in Winkler’s sense, selecting a specific pre-training method determines the understanding of similarity underlying the retrieval system. Selecting a specific method can then be correlated to the type of information we are ›omitting‹ during the translation of pixels into a feature vector. Whenever we select a specific method for this translation, we also decide which features are deemed irrelevant by the models. Similarity in this way is always influenced by the choice we make regarding the pre-training method. This means that the attention and selection process of the network is guided by human input, often motivated by the research question, task or personal interests. In comparison to humans, the attention and selection of similarity criteria in this context is much more conscious and goal-driven.

We conclude the following: Attesting similarity requires comparative processes separating things into aspects (features), which are similar and dissimilar Cf. Winkler 2021, pp. 111–112, 257., which in the context of this paper refer to image-internal components. These criteria are abstracted from the image and quantified, thus becoming measurable. We also noted that the image quality might influence the similarity measure. The conversion of input data into a lower-dimensional vector representation during feature extraction is associated with a loss of information and therefore might result in a loss of similarity dimensions. We also elaborated on the fact that the selection of the pre-training method can influence the type of similarity underlying the retrieval process. This section thus highlighted that the computer guides as well as limits our understanding of similarity. Discussing similarity in the context of this paper, with a particular focus on quantification and the process of abstraction, provided interesting insights and leaves room for further discussion and research.