Image Auto-Tagging Technology - How Object Detection, Scene Recognition, and Caption Generation Work

Overview of Image Auto-Tagging - Why AI-Based Image Understanding Is Needed

Image auto-tagging is a technology in which AI analyzes the content of an image and automatically labels the objects, scenes, and attributes it contains. It is used in a wide range of applications, such as digital asset management (DAM), classifying product images on e-commerce sites, search features in photo apps, and improving accessibility (automatic alt text generation).

The limits of manual tagging:

• Scalability: On platforms where tens of thousands of images are uploaded per day, tagging by hand is physically impossible. On Instagram, about 100 million images are posted per day
• Consistency: Human tagging varies from person to person. The problem of different tags, such as "dog", "doggy", "pup", and "Shiba Inu", being assigned to the same subject
• Cost: Large-scale tagging work involves enormous labor costs. At 30 seconds per image, 100,000 images amount to about 833 hours of work
• Coverage: People tend to focus only on the main subject in an image and overlook background elements and fine attributes (color, texture, time of day)

The technology stack of AI tagging:

• Image classification (Classification): Assigns one or more category labels to the whole image. For example, "landscape", "food", "people"
• Object detection (Object Detection): Identifies the position (bounding box) and category of each individual object in the image. "Cat (x:100, y:200, w:150, h:120)"
• Semantic segmentation: Classifies the category of each region pixel by pixel. Precisely separates the regions of "sky", "road", "building", and "person"
• Image caption generation: Describes the content of the image in natural language. "A woman and a dog sitting on a park bench"

How Object Detection Works - YOLO and Transformer-Based Models

Object detection is the task of simultaneously predicting the position (bounding box) and category (class label) of each of multiple objects in an image. The YOLO (You Only Look Once) series is a representative family of models for real-time object detection.

Basic principles of YOLO:

• One-stage detection: Processes the whole image in a single inference and predicts the positions and classes of all objects at the same time. 10-100 times faster than two-stage methods (Faster R-CNN)
• Grid division: Divides the input image into an S x S grid, and each grid cell predicts B bounding boxes and the probabilities of C classes
• Anchor boxes: Uses boxes with predefined aspect ratios as a reference and predicts adjustments to offset and size. Handles objects of diverse shapes
• NMS (Non-Maximum Suppression): A post-processing step that removes overlapping detection results and keeps only the one box with the highest confidence for each object

The evolution of YOLO (as of 2024):

• YOLOv8 (Ultralytics): Released in 2023. Adopts an anchor-free design, the C2f module, and a decoupled head. mAP of 53.9% on the COCO dataset
• YOLOv9: Introduces PGI (Programmable Gradient Information) and the GELAN architecture. Minimizes information loss and achieves high accuracy even with lightweight models
• YOLO-World: Open-vocabulary object detection. Not limited to predefined classes, it can detect any object from a text prompt

Transformer-based object detection:

• DETR (DEtection TRansformer): Developed by Meta. Captures relationships between objects with the Transformer attention mechanism and achieves end-to-end detection without NMS
• Grounding DINO: Through multimodal understanding of text and images, detects objects specified in natural language. Complex specifications such as "a person wearing a red hat" are possible

Image Classification and Scene Recognition - From CNN to Vision Transformer

Image classification is the task of assigning one or more category labels to an image as a whole. Whereas object detection answers "what is where", image classification answers "what is this image". Scene recognition is a type of image classification that recognizes places / situations such as "beach", "office", and "forest".

Classification based on CNNs (convolutional neural networks):

• ResNet: A groundbreaking architecture that made it possible to train deep networks (50-152 layers) through residual connections (Skip Connection). ImageNet Top-5 accuracy of 96.4%
• EfficientNet: Scales the width, depth, and resolution of the network in an optimal balance. Achieves accuracy equivalent to ResNet with 1/8 the number of parameters
• ConvNeXt: A 2022 model that brings the design philosophy of Transformers into CNNs. Achieves performance equivalent to Vision Transformer with a CNN architecture

Classification based on Vision Transformer (ViT):

• ViT (Vision Transformer): Divides an image into 16x16px patches and feeds each patch into a Transformer as a token. Outperforms CNNs on large-scale datasets
• DINOv2: A self-supervised learning model developed by Meta. Pre-trained on a large number of unlabeled images, it achieves high-accuracy classification with a small amount of labeled data
• CLIP (Contrastive Language-Image Pre-training): Developed by OpenAI. Trained on pairs of text and images, it achieves zero-shot classification (classification is possible even for unknown categories). It can answer questions such as "Is this image a ___?"

Characteristics of scene recognition:

• Models trained on the Places365 dataset are the standard. They recognize scenes in 365 categories (airport, bedroom, forest, stadium, and so on)
• Combined with object recognition, compound tagging such as "a dog on a beach" is possible
• Recognizing the time of day (morning, midday, evening, night), weather (sunny, cloudy, rainy), and season (spring, summer, autumn, winter) is also possible

Image Caption Generation - Natural Language Description with Multimodal AI

Image caption generation (Image Captioning) is the task of describing the content of an image in natural language sentences. Going beyond simple tagging ("dog", "park", "ball"), it generates descriptions that include context, such as "a brown dog chasing a red ball in a park".

Technical approaches to caption generation:

• Encoder-decoder approach: Extracts image features with a CNN (image encoder) and generates natural language with an RNN/Transformer (text decoder). Classic but still effective
• Vision-Language models: Multimodal models that handle images and text in a unified way. BLIP-2, LLaVA, and GPT-4V are representative examples
• Prefix tuning: Feeds image features into a large language model (LLM) as a prefix and generates text based on the image

Major caption generation models (2024):

• BLIP-2: Developed by Salesforce. Uses the Q-Former to convert image features into a format the LLM can understand. Generates high-quality captions while being lightweight
• LLaVA (Large Language and Vision Assistant): Trained on instruction data generated with GPT-4. Can also answer questions about images
• GPT-4V / GPT-4o: OpenAI's multimodal models. They can generate the highest-quality captions, but the API cost is high
• Florence-2: Developed by Microsoft. Performs caption generation, object detection, and segmentation with a unified model

Applications to accessibility:

• Automatically generates alt text for images and conveys image content to screen reader users
• Put to practical use in apps for people with visual impairments, such as Microsoft's Seeing AI and Google's Lookout
• It is possible to build a CI/CD pipeline that automatically generates alt text at build time for images on a website that have no alt text set

Implementation in Web Applications - Cloud APIs and Edge Inference

Methods for building image auto-tagging into web applications fall broadly into two types: using a cloud API and running inference at the edge (browser/device). Understand the characteristics of each and choose the appropriate method for your requirements.

Implementation with cloud APIs:

• Amazon Rekognition: Object detection, scene recognition, face analysis, text detection, and inappropriate content detection. $1.00 per 1,000 images (for the first 1 million images)
• Google Cloud Vision AI: Label detection, object detection, OCR, SafeSearch, and landmark recognition. $1.50 per 1,000 images
• Azure Computer Vision: Tagging, caption generation, object detection, and OCR. $1.00 per 1,000 images
• OpenAI GPT-4V API: The highest-quality image understanding. Question answering in natural language is possible. However, it is costly at $0.01-0.03 per image

Implementation patterns for cloud APIs:

• When an image is uploaded, call the tagging API with Lambda/Cloud Functions and save the results in a DB
• Run the work in the background with asynchronous processing (SQS/Pub/Sub) to eliminate user waiting time
• Caching results: Store them in DynamoDB with the hash value of the same image as the key to avoid duplicate processing

Implementation with edge inference (in the browser):

• TensorFlow.js: Runs MobileNet (image classification) and COCO-SSD (object detection) in the browser. GPU acceleration with the WebGL backend
• ONNX Runtime Web: Runs models in the ONNX format with Wasm or WebGL. Supports ONNX exports of YOLOv8
• MediaPipe: Made by Google. Runs object detection, image classification, and image segmentation in real time in the browser

Advantages of edge inference: privacy protection (images are not sent to a server), offline operation, and zero API cost. Constraints: model size (a 5-50MB download), inference speed (100-500ms/image on mobile), and accuracy (may be lower than cloud APIs).

Using Tagging Results - Search, Filtering, and Recommendations

By making effective use of AI tagging results, you can greatly improve the user experience, for example by improving the accuracy of image search, filtering content, and providing personalized recommendations.

Using tags for image search:

• Integration with text search: Include tags in the index so that images can be searched with natural language queries such as "blue car" or "beach at sunset"
• Faceted search: Group tags by category (object, scene, color, number of people) so that results can be narrowed down with a filter UI
• Similar image search: Use vector representations of tags (TF-IDF or Word2Vec) to achieve image search based on tag similarity
• CLIP-based search: Search by the cosine similarity between the text query and the image embedding vectors. Direct text-image matching is possible without tagging

Content moderation:

• Automatic detection and flagging of inappropriate content (violence, adult content, hate)
• Can be implemented with Moderation Labels in Amazon Rekognition or with Google SafeSearch
• Used not for automatic blocking but as filtering for the final judgment by human moderators

Recommendations:

• Analyze the tag distribution of images a user has viewed / liked and recommend images with similar tags
• A hybrid of collaborative filtering (preferences of similar users) and content-based filtering (tag similarity)
• Tag weighting: style tags (minimal, vintage) reflect user preferences more readily than object tags (dog, car)

Database design:

• Because tags have a many-to-many relationship (multiple tags per image, multiple images per tag), manage them with an intermediate table or an array-type column
• In DynamoDB, achieve search keyed on the tag name with a GSI (Global Secondary Index)
• Run full-text search and faceted aggregation of tags quickly with Elasticsearch / OpenSearch
• Store a confidence score (confidence) together with each tag and filter by a threshold (for example, 0.7 or higher)