The issue of annotation software availability and the ability of researchers to use available software efficiently has been recently raised.1 While large-scale funded projects can design and modify their own platform and use available codes,2 smaller and unfunded research projects might need to depend on existing freely accessible software. The aim of this study was to assess existing annotation software that has been used in ophthalmology projects and compare their practicality and utility in fundus image annotation tasks. By assessing these tools according to specific criteria, the best choices for fundus image annotation were identified. This initiative was designed to support researchers in ophthalmological practice by offering them selected and appraised tools, thus eliminating the need to search for and judge these tools individually.
We first conducted a systematic search for artificial intelligence (AI) ophthalmology articles that used annotation software as part of their methodology. We performed a PubMed search on the 8th of July 2022 using the advanced search strategy with the following query: (“artificial intelligence” or “machine learning” or “deep learning” or “supervised machine learning” or “unsupervised machine learning” or “neural networks (computer)”) AND (oct or “Optical Coheren*” or fundus or retin* or macula) AND (Annot* OR Segment*). We restricted the search to original articles, excluding reviews, comments, opinions, or abstracts and in English language. We screened searched articles and further excluded non-ophthalmic articles. We read all ophthalmic articles included to determine annotation software used. After extracting software previously used in ophthalmic AI articles, we performed a separate google search looking for other software not previously found in the initial search. We used the above keyword search in Google search engine, supplemented by search within relevant websites.
We compared annotation software in terms of development year, accessibility status, and number of citations received by the original development paper. We downloaded all the software found in the prior section to compare their usage. We assessed software or cloud-based platforms, size for downloadable software, operating systems, output formats, cloud-based or local, and Functioning independently without dependency on external libraries. We defined several practicality criteria through discussions among our research team that allow researchers with limited computer skills to use a commonly used computer system, which comprises both software experts and ophthalmologists: Functioning independently without dependency on external libraries; size less than 100 MB; free of charge; accepting more than one image format input; output of more than two possible formats.
We found 131 image annotation software and platforms in our search. Ten software satisfied our practicality criteria. A detailed comparison between software and platforms is provided in supplementary material. In our PubMed search for image annotation software used by ophthalmology-related articles, we found that the majority of studies either did not report the software used or used an automated method for annotation, with IOWA protocol being the most reported.3 While seven different software were utilized for image annotation in ophthalmology articles, the only freely accessible software was CVAT and ImageJ.
An overview of the selected 10 annotation software that fulfilled our criteria is provided in supplementary material.
We compared the annotation task between the included software and provided a detailed analysis in supplementary material. Here’s an overview of the top software tool for image annotation:
ImageJ is an open-source image processing and analysis software developed by the National Institutes of Health (NIH). Although it is primarily designed for scientific and medical image analysis, it can also be used for image annotation tasks. ImageJ provides various plugins and extensions to enable customizable image annotation workflows. While it may not have advanced annotation-specific features, its flexibility and extensibility make it a versatile option for researchers and developers.
VIA is a lightweight data annotation tool; it is web-based and runs in the browser which eliminates the need for software installation or setup. It was developed by VGG (Visual Geometry Group), a research group from the University of Oxford that has contributed significantly to computer vision research. While not a specific software tool, VGG is renowned for its deep convolutional neural network architectures, such as VGG16 and VGG19, which have been widely used for image classification tasks. Though VGG models are not designed explicitly for annotation, they can be integrated into larger annotation pipelines for automated object detection and classification.
Seg3D is an open-source software developed by the Scientific Computing and Imaging (SCI) Institute at the University of Utah. While it primarily focuses on volumetric image segmentation, it also offers image annotation capabilities. Seg3D provides 3D annotation tools, enabling users to annotate image volumes with regions of interest, making it useful for projects involving medical imaging and other 3D data.
MakeSense.AI is a web-based image annotation platform that offers a user-friendly interface for annotating images with bounding boxes, polygons, key points, and more. It supports collaborative annotation, making it suitable for team projects. Additionally, it provides options for both manual and semi-automated annotation, combining human expertise with AI assistance.
CVAT is an open-source annotation platform developed by Intel. It offers a comprehensive set of annotation tools, including object detection, image segmentation, and video annotation. CVAT supports multiple annotation formats and provides APIs for easy integration with other computer vision applications and workflows. CVAT is a web-based tool and can be downloaded and installed on custom local or cloud servers this makes it customizable and extensible, It also includes tools for medical image conversion and annotation.
Colabeler is a user-friendly image annotation platform that caters to both beginners and experts. It provides a range of annotation tools, including bounding boxes, polygons, and semantic segmentation. Colabeler also supports video annotation and offers features like collaborative annotation, version control, and quality assurance for accurate and efficient annotations.
Labelme is an open-source image annotation tool developed by MIT. It allows users to annotate images with polygons and key point annotations. Labelme provides a simple graphical user interface and supports exporting annotations in various formats for use in machine learning projects.
Ratsnake is a web-based annotation platform designed to simplify the annotation process. It offers various annotation tools like bounding boxes, polygons, and semantic segmentation. Ratsnake supports real-time collaboration, making it useful for distributed teams working on large-scale projects.
Pixie is an AI-powered image annotation tool that automates the annotation process using machine learning algorithms. It provides a user-friendly interface for manual annotation and uses AI to assist with semi-automated annotation tasks. Pixie is suitable for speeding up annotation workflows and improving efficiency.
Classifai is an AI-powered image annotation platform that supports both image and video annotation tasks. It offers a variety of annotation tools and provides automated annotation capabilities using pre-trained machine learning models. Classifai aims to streamline the annotation process and improve accuracy through AI assistance.
To our knowledge, this is the first study that systematically gathers, assesses, and compares various annotation software used in existing literature and available on the web for the task of fundus image annotation. In total, 10 tools were included. Of these, 3 were cloud-based, 5 were downloadable and operated as desktop applications, and 2 offered both cloud and desktop options. We evaluated these tools based on several criteria such as file size, platform (desktop or web-based), pricing model (either free or paid), output formats, and any external dependencies they might have required. For a thorough examination, we either downloaded each software or accessed it directly online (in the case of web-based tools) to compare their functionalities and user experience.
AI, in simple words, is the ability of machines to reason, think, and learn.4,5 Machine learning (ML), a subset of AI, applies computational algorithms to analyze complex and large datasets.6,7 ML is categorized into supervised and unsupervised learning with supervised learning being more prevalent.6,8 In this type, algorithms are trained on previously labeled data, pairing known inputs (data source) along with their corresponding output labels.8,9 Additionally, ML is classified into shallow and deep learning (DL).10 In DL algorithms, multi-layered computational models learn multi-level data representations, contributing to significant advancements in medical areas like ophthalmology, dermatology, and radiology.8,11,12 During DL training, an ideal standard is required to measure an algorithm’s performance. This is referred to as Ground Truth (GT).13,14 Essentially, GT represents a set of data that has been accurately annotated or labeled, serving as a reference point taking on various forms according to the purpose of the annotation.13,15 Labeling an image with keywords that correspond to its contents, either manually or automatically, is referred to as image annotation.16 In ophthalmology, experts such as ophthalmologists typically annotate fundus photography and optical coherence tomography (OCT), serving as primary data sources in deep learning applications for identifying retinal diseases, including diabetic retinopathy, glaucoma, and retinopathy of prematurity.11,14,17–19
The main limitation of our study is that we focused on software related to artificial intelligence applications. Future studies should consider other applications in their reviews. Moreover, the field of image annotation software is rapidly changing, warranting updating this review in the future.
None
OpenAI’s ChatGPT has been used for paraphrasing and content formulation. https://chat.openai.com/?model=gpt-4
Image annotation software to be use for image processing in aartificial intelligence projects with detailed comparison