The viewer/annotation tool was developed as a “Node.js” (http://nodejs.org, as of October 25, 2018, v8.x) application using the “Express” (http://expressjs.com, as of October 25, 2018, v4.x) web framework, with JavaScript as the programming language for both front and backend code. All data relative to annotations made by pathologists were stored using “CouchDB” (http://couchdb.apache.org, as of October 25, 2018, v2.x), a document-oriented database system designed for the web: It uses JavaScript Object Notation as a data format and the HyperText Transfer Protocol (HTTP) for communication.

The retrieval system is based on a client-side only frontend developed with HyperText Markup Language 5 (HTML5) and JavaScript, communicating with a Java-based retrieval backend using Representational State Transfer web services. No web framework (e.g., Angular, React) was used in the development of these tools, but rather a plain JavaScript approach with the use of libraries for speci昀椀c features.

Several existing libraries and software components were combined to develop the viewer/annotation tool, outlined below and shown in Figure 1: ● jQuery (http://jquery.com/, as of March 29, 2018, v2.1.3) was used as the base for interactions with HTML elements, event handling, and asynchronous HTTP calls on the client side ● OpenSeaDragon (http://openseadragon.github.io, as of October 25, 2018, v2.2.1) was used as the viewer for the high-resolution histopathology images. It allows loading, zooming and panning around high-resolution images while dynamically loading image tiles from a given tile source. Several plugins for OpenSeaDragon were also used, such as the “Scalable Vector Graphics (SVG) Overlay for OpenSeaDragon” (https://git.io/vA0tP, as of October 25, 2018) plugin for integrating the annotation tool with the viewer ● IIPImage (http://iipimage.sourceforge.net/, as of October 25, 2018, v1.0) was used as the image server generating the tiles loaded by the OpenSeaDragon viewer. It is a fast server that supports various high-resolution formats and protocols, including “DeepZoom” (https://en.wikipedia. org/wiki/Deep_Zoom, as of October 25, 2018), which was used for interacting with the viewer ● OpenSlide (http://openslide.org, as of October 25, 2018, ● ● ● ● v3.4.0) was used to read WSIs from various proprietary formats (e.g., Aperio, Hamamatsu) T he VASA R I I mage Processi ng System ( V I PS) (http://jcupitt.github.io/libvips, as of October 25, 2018, v8.2.2) was used in combination with OpenSlide to convert WSIs to a standard pyramidal Big Tagged Image File Format image that can be served by IIPImage, using the DeepZoom image protocol SVG‑Edit (http://github.com/SVG‑Edit/svgedit, as of October 25, 2018, v2.8.1) was used as the tool for drawing annotations on the images. It is a full-featured drawing application that was simplie昀椀d for the needs of the project The Shambala retrieval interface (http://shambala. khresmoi.eu, as of October 25, 2018, v1.0) was used as the basis for the development of the retrieval system’s frontend.[9] The main objective of the interface is to provide an easy-to-use and intuitive search interface for images The Parallel Distributed Image Search Engine (ParaDISE) (http://paradise.khresmoi.eu, as of October 25, 2018, v0.0.1) was used as the backend for the retrieval system.[29] The main objectives of ParaDISE are to enable indexing and retrieval of images using several types of visual features (both local features and global descriptors) in a simple and ef昀椀cient manner.

The platform was developed in an iterative way, with short iterations followed by user feedback and adjustments, especially in the beginning of the process. The various features (viewer, annotation tool, and retrieval) were built up step-by-step, taking into account the user comments, ensuring that the 昀椀nal product would 昀椀t their needs.

This section shows an overview of the workflow of a pathologist using the platform [Figure 2 and Table 2].

This section presents an overview of the user interface that was developed and it details each function. The main user interface, as shown in Figure 3, can be split into three principal areas: 1. The navigation bar provides access to the various pages of the interface, including the homepage, a page with more information about the platform and a page allowing authorized users to upload new images to the platform 2. The database/feature selection and annotation edition zone allows switching between images contained in different datasets, as well as selecting feature overlays to display on top of an image (see section “Visual Feature Selection” below). In the same space, the annotation edition zone appears, as detailed in Figure 4 3. The viewer is the main section where users can interact with images, zoom in and out quickly, create and edit annotations.

Currently, the process of including new images in the platform is split into two processes: ● Uploading and converting WSIs to make them available for viewing and annotating Extracting patches from WSIs, indexing their visual features, and making them available for retrieval. The first process is largely automated, as illustrated in Figure 12. The process consists of three steps: 1. A user imports a WSI 昀椀le using the web interface 2. Once the image is successfully uploaded, the VIPS tool is automatically started, converting the image to a pyramidal format that can be used by the IIPImage server (when necessary). This process typically takes around 4–5 min, even for very large images (e.g., ~20GB, ~80,000 × 80,000 pixels) 3. The 昀椀le can then be accessed by the OpenSeaDragon viewer using the DeepZoom image protocol.

The second process currently requires several manual steps: 1. The patch extraction process consists of two steps, depending on whether the WSI has manual segmentations or not. When there are manual annotations, a Python script (using the “scikit,” “Pillow,” and “openslide” libraries) is run to compute the overlap of the segmentation with the annotated areas and then extract nonoverlapping patches. When there are no annotations, a uniform extraction of patches is performed over the entire WSI. An upper bound of 2000 patches per WSI was set due to storage constraints. An important factor that affects the quality of the results is the quality of the annotations made by the pathologists, as some of them draw very precise areas and others mark only a rough region consisting of several classes and background 2. For each magni昀椀cation level, sets of visual features can be extracted. This is done through a call to the ParaDISE indexer that suppor ts several built‑in handcrafted unsupervised descriptors (e.g., Colour and Edge Directivity Descriptor, Fuzzy Colour and Texture Histogram, Bag of Visual Words). In our system, DL features generated by several deep architectures are extracted as well, using the Keras (http://keras.io, as of October 25, 2018) DL framework and several patch magnic昀椀ations to enforce that the network learns features at several levels of magnic昀椀ation 3. T h e u s e r i n t e r f a c e i s u p d a t e d w i t h t h e n e w dataset/magni昀椀cation levels.

The main dataset used is a proprietary database of ContextVision AB (CVDB), including 112 WSIs of the prostate that were uploaded to the viewer with ROIs of healthy tissue (manually annotated by two pathologists) and Gleason patterns graded from three to v昀椀e according to the Gleason grading system.[31,32] This dataset highlights the capability of the retrieval system to manage proprietary datasets (such as the ones owned by research groups or clinical pathology departments). Since some of the annotations were in disagreement at the moment of writing the manuscript, we did not evaluate our system at the Gleason level but just at the healthy tissue versus tissue containing an annotated Gleason pattern. One of the goals was to evaluate the ability of the system to retrieve similar patterns in the WSI database given a query. Such patterns can arise at different magni昀椀cation levels, for example, a mitotic cell is better observed at ×40 magnic昀椀ation, whereas a Gland pattern can be seen in any magni昀椀cation from ×5 to ×10 and cell characteristics are barely visible at such low magni昀椀cations. This poses an important computational limitation because if the user is interested in doing an exhaustive search of all the patterns at a high magni昀椀cation, millions of patches must be extracted. In our experiments, we limited the extraction to an upper bound of 2000 patches per WSI for all the magni昀椀cations, extracting patches from the annotated areas when available. Patches at distinct levels of magnic昀椀ation were extracted from the pyramidal content of the WSI to account for the scale‑speci昀椀c nature of some of the patterns. Besides this, state-of-the-art DL networks applied in histopathology have shown that learning the scale of a given histopathology image patch is feasible and that including patches from multiple scales improves the performance of classi昀椀cation tasks.[33,34] Using the openslide library, ×5, ×10, ×20, and × 40 magnic昀椀ation patches were extracted from manually annotated ROIs [Table 3 and Figure 13]. For each of the patches, both handcrafted Color and Edge Directivity Descriptor (CEDD)[35] and DL features were extracted.

The second dataset used for the validation is a subset of images from the PubMed Central dataset of biomedical open access literature. Images of the dataset were automatically classi昀椀ed as “light microscopy” images.[29] This second dataset has very different characteristics, as it is not composed of WSIs (with patches extracted at various magnic昀椀ation levels), but rather of g椀昀ures extracted from articles contained in a variety of medical journals. The data has the added bene昀椀t of including a caption for each 昀椀gure, enabling text‑based search. It includes around 240,000 images. This dataset highlights the capability of the retrieval system to manage publicly available collections of data, characterized by extremely high variability in scale and color.

For leveraging the discriminative power of DL models in our system, we trained a state-of-the-art architecture, the DenseNet model[36] that has a dense connectivity pattern among its layers. DenseNet introduces direct connections between any two subsequent layers with the same feature map size. The main advantage of DenseNet over other very deep architectures is that it reuses information at multiple levels without drastically increasing the number of parameters in the model, particularly with the inclusion of bottleneck and compression layers. The bottleneck layer applies a 1 × 1 convolution just before each 3 × 3 convolution to reduce the number of input feature maps. The compression layer uses a fraction of the feature maps in each transition layer. The details of the DenseNet architecture trained are: ● DenseNet-BC 121: We chose the 121-layer variation of DenseNet with seven million parameters and performed experiments 昀椀ne‑tuning all the layers from pre‑trained ImageNet weights ● The network was optimized to minimize the binary cross-entropy (healthy vs. cancer patches) using the

Adam method with initial learning rates explored logarithmically between 0.01 and 10−7 . The best learning rate was found to be 0.001 We implemented the architecture using the Keras DL framework with the TensorFlow backend. The training was set to v昀椀e epochs; nevertheless, an early convergence after two to three epochs was observed The training of the models took from 2–3 h using a Nvidia Titan Xp Graphics Processing Unit The dataset used for learning the model was the CVDB. We trained with 70% of the training patches and validated with the remaining 30%. The retrieval performance is then computed using the test data.All the patches from a single WSI are kept in the same partition (training/validation/test) to avoid bias.

Once the network is trained, an auxiliary layer shown in Figure 14 is added, to account for the fact that the retrieval system computes similarities based on feature vectors. It is thus necessary to extract feature vectors instead of having a probability of cancer/no cancer as output. The auxiliary layer is a dense layer which extracts 1000-dimensional feature vectors. After the feature vectors of all the patches are extracted, an index is built for each magni昀椀cation level and then included in the ParaDISE retrieval engine.

For assessing the retrieval performance of our system, the following standard evaluation metrics were selected: mean average precision (MAP), geometric MAP, precision after ten patches retrieved (P10), and precision after 30 patches retrieved (P30). Precision-recall graphs also show the performance in a graphical way.

We report these performance metrics that are common measures in the assessment of retrieval systems, also for image retrieval, and that assess different aspects of the system. This evaluation purely analyses the algorithm performance in a standard way. The user tests described below perform a qualitative evaluation of the user interface that also includes the retrieval quality.

As shown in Figure 15, the precision-recall graphs for both DL and CEDD features are compared. As shown in other articles,[7] the deep learning-based representations outperform handcrafted features such as CEDD in digital pathology tasks. Instead of computing the performance on a WSI basis, the reported measures are at a patch level, as there are several annotated areas in a WSI, which makes it hard to assign a single label to compare relevance with another WSI in the database. The number of patches evaluated was not exhaustive as this would result in a massive number of 35,539 × 82,925 = 2,947,071,575 entries (~40GB). We uniformly sampled 10% of the training patches and 50% of the testing patches to have an estimation of the quantitative performance of our system, i.e., 8292 training patches and 17,769 testing patches. Even with this small sample, the computation of the ranking matrix took ~18 h, since computing the histogram intersection for such a large number of samples incurs in a high computational cost. The performance measures are shown in Table 4.

http://www.jpathinformatics.org/content/10/1/19 An interesting measure is P@10 (0.2507) that can easily be interpreted: On average, 2.5 retrieved patches in the 昀椀rst ten retrieved patches are relevant or in this case of the same Gleason pattern. Sometimes also similar patterns and not only the same can be relevant in the differential diagnosis, so the chosen relevance criterion of considering only the exact same Gleason pattern is very strict. The other measures also show that the general performance of the system is good.

The interface of the system was evaluated qualitatively by two senior pathologists (P1 and P2) who interacted with the system through a series of 昀椀ve tasks: 1. Open the WSI number 1, mark a gland region and search for similar results in the WSI database.

P1: Pathologist 1 remarked the ease of use of the interface but that the results can be improved, suggesting that training with more powerful features in larger datasets should be performed. P2: Pathologist 2 pointed out that the results are not all relevant – some results are benign for malignant queries. Despite this, he remarked the large collection of relevant patches and that results take you to directly to the WSI image to which it belongs. He said that the results quality needs to be improved, possibly through a larger data set with annotated areas.

Open a WSI, mark an area of in昀氀ammation, search in the PubMed Central database.

P1: Pathologist 1 pointed out the ease of use of the interface and that the PubMed Central images included results from monkeys and rats, which are not of interest for clinical use when compared to humans. This aspect was improved subsequently using the MeSH terms attached to the articles for 椀昀ltering out animal tissue.

P2: Pathologist 2 said that the articles found were not all relevant to the prostate cases he searched with. He liked the easiness for starting the query and opening the articles. He said that the retrieved articles could be better focused.

within the ContextVision database (i.e., otherwise very small), which allows good retrieval results.

P2: Pathologist 2 liked the fact that the more speci昀椀c you are with your annotation, the more relevant images were found and that the interface is easy to use. He remarked again the need for including a larger number of relevant images in the database.

Open a WSI, mark a Gleason 4 cribriform area and search for similar results in the PubMed Central database. P1: Pathologist 1 pointed out that the quality of the results is not optimal for use in research or clinical practice, which suggests that more work needs to be done to extract and l昀椀ter valuable information in the PubMed Central database (which was done after the end of the user tests leading to an improvement in retrieved results).

P2: Pathologist 2 mentioned that showing the caption of the article with the image would be useful. He liked the speed of the search and that the interface was easy to use. Finally, he commented on the relevance of the results and insisted to include a larger number of relevant articles.

Open a WSI, mark a Gleason 4 cribriform area and search for similar results in the WSI database.

P1: Pathologist 1 remarked that the results were suf昀椀ciently good and that it would be nice to have a split screen with “side-by-side” view of the images. The high incidence of good results may be related to the high incidence of Gleason 4 cribriform areas 5. Free Interaction with the system.

P1: Several points were mentioned: a. The system should include a larger database, as it only included few WSIs at the time of the tests b. The tool can be helpful in the daily job of general pathologists and residents c. The speed of the system was considered good for research use but possibly slightly slow for clinical use d. The system was considered useful for 昀椀nding

patterns in radical prostatectomie e. The magnification at which the search was performed had an impact on the quality and the type of the results retrieved.

P2: The following was given as feedback: a. The system is useful but needs more images to make it even more powerful b. The most useful functionality is the ability to re昀椀ne the search with relevance feedback and 椀昀nd images in the PubMed Central database c. The principal improvement was considered to be a larger data set to have more relevant cases. dIscussIon This article describes the concept, the 昀椀rst implementation, and test of a web-based digital pathology retrieval system that provides an interactive visualization and the possibility to perform visual similarity retrieval in proprietary WSIs and publicly available datasets of the biomedical literature. The retrieval system is based on automatically extracted hand-crafted and DL-based visual features. The retrieval is possible with text, as content-based visual retrieval and a combination of the two. This approach allows to include hospital archives where no annotations exist, as well as image datasets with additional data such as text. Unlike most other pathology viewers, features generated with a convolutional neural network trained and n昀椀e‑tuned on pathology images are used for the integrated retrieval. The method uses manually annotated regions for training. However, once trained, the model can also be used on large nonannotated data sets, for which manual annotations can be too time-consuming and expensive (e.g., digital pathology repositories in hospital). The system was tested on two strongly differing databases (in this paper, using a proprietary dataset of WSIs and the PubMed Central dataset). The platform was tested using prostate cancer biopsies, although it can easily be extended to other cellular and tissue samples. The use of proprietary datasets allows clinical departments and scientic昀椀 research groups to develop their own resources to guide difc昀椀ult diagnoses or train students. The use of publicly available datasets, such as the PubMed Central data allows pathologists to benet昀椀 from the increasing number of images in the biomedical literature that are available. The proposed retrieval system could be particularly helpful when dealing with rare patterns and it allows to save time in comparison to searching in books.

A user test with two pathologists was performed to have feedback already in an early development stage by potential users of the platform. The overall impression of the interface was very positive, particularly the speed and the user-friendly interaction of the different functions in the platform were mentioned. Comments were made regarding the retrieval http://www.jpathinformatics.org/content/10/1/19 quality and the limited number of available images: the version of the system described here was already improved in terms of retrieval results and amount of indexed data, integrating the comments of both pathologists. Further 昀椀ne‑tuning of the retrieval results was performed and should ease the integration into clinical work昀氀ows.

An online video demonstration of the platform is available at the following link: https://youtu.be/uwIezxabiaw (as of November 20, 2018). conclusIons Interactive systems equipped with AI‑based models may change the current work in histopathology. Thanks to the recent approval that digital histopathology can also be billed in the US, new systems for slide storage, analysis and retrieval can be used for diagnosis and not only for research. Image retrieval on histopathology data is a solution that can enhance the usage of AI resources according to the pathologists’ experience and requirements.

The developed retrieval prototype enables the viewing of WSIs and the search for images similar to manually de昀椀ned ROIs in diverse data sources, including proprietary and public datasets. The presented results and user tests show that visual retrieval is possible, even with extremely large WSIs. The proposed solution aims to reduce the pathologists’ workload while increasing the reproducibility of their assessment on digitally scanned tissue slides. The technical implementation is based on the combination of open source solutions, thus helping the scientic昀椀 community to test, develop improved similar systems for research and clinical practice. Further improvements to the system should be studied to introduce this tool into the clinical work昀氀ow and facilitate the diagnostic process.