How AI Improves Japanese Manga Text Recognition? Application of a New OCR Model

For many manga readers and developers, accurately recognizing text in manga has always been a challenge. Recently, an AI Optical Character Recognition (OCR) model fine-tuned specifically for Japanese manga has increased recognition accuracy from 27% to 70%, offering new possibilities for manga translation and related applications.

For readers who enjoy reading manga in its original language, the language barrier is often the first challenge. For those who want to use tools to assist with reading or translation, how to make computers accurately “read” the text in manga is an important technical problem.

The core technology behind this is called Optical Character Recognition (OCR). While current OCR technology is quite mature for processing standard documents, it faces many difficulties when applied to manga.

Why is recognizing manga text so difficult?

The way text is presented in manga is very different from general documents, which brings several major challenges for OCR technology:

• Varied font styles: Manga artists often use various artistic fonts to convey characters’ emotions or the intensity of sounds, and these non-standardized fonts are difficult for computers to recognize.
• Irregular layouts: Text within speech bubbles can be written vertically, horizontally, or even diagonally, increasing the complexity of localization and recognition.
• Complex background interference: Text is often superimposed on rich visuals or effect lines, unlike clear black text on white paper.
• Special manga symbols: A large number of onomatopoeia and effect words are unique expressions in manga, and general OCR models are usually not trained for such content.

Due to these factors, most general OCR tools do not achieve ideal recognition accuracy when processing manga.

PaddleOCR-VL-For-Manga Model Designed Specifically for Manga

To solve this problem, a developer has launched a specialized AI model called “PaddleOCR-VL-For-Manga” tailored to the characteristics of Japanese manga.

The project is based on the visual language model PaddleOCR-VL developed by Baidu’s PaddlePaddle team. To make it better adapted to manga scenarios, the developer performed “fine-tuning,” which means training the model with additional data from a specific domain.

The training data mainly comes from the Manga109-s dataset, supplemented by 1.5 million additional synthetic samples. Through this specialized manga data, the model learned how to recognize various special text styles and layouts in manga.

About Manga109-s Dataset

Manga109 is a research dataset compiled by academic institutions, containing 109 Japanese manga titles. The Manga109-s subset is specifically licensed for commercial development, providing valuable resources for research in related applications.

Recognition Results: Accuracy Increased from 27% to 70%

After this specialized fine-tuning, the model’s performance significantly improved.

According to information released by the developer, the original model’s full sentence recognition accuracy on manga was approximately 27%, while the fine-tuned “PaddleOCR-VL-For-Manga” model increased the accuracy to 70%. This advancement means that the model can recognize complete sentences in speech bubbles more accurately, not just fragmented words.

The new model performs well in handling manga speech bubbles and stylized fonts. However, the developer also pointed out that there is still room for improvement in distinguishing between “full-width” and “half-width” characters. Nevertheless, this remains a noteworthy development in the field of manga OCR technology.

How to use this model?

This model is open source, and developers interested in this technology can find it on the Hugging Face platform.

Users can invoke this model through Transformers, PaddleOCR, or other libraries that support PaddleOCR-VL. The developer suggests that for documents with fixed layouts, it can be combined with the PP-DocLayoutV2 layout analysis tool, but also reminds that manga layouts differ from standard documents.

Potential Applications of this Technology

The advancement of this type of technology brings practical value to many fields:

• Assisting manga translation: Translation teams can use this tool for initial text extraction, followed by professional human translation and refinement, which helps improve work efficiency.
• Developing language learning tools: In the future, we may see more applications combining OCR technology, such as instant translation of manga by taking photos with a mobile phone, assisting Japanese language learners.
• Promoting academic text analysis: Researchers can more conveniently extract text data from a large number of manga for linguistic or cultural research analysis.

Overall, this OCR model fine-tuned specifically for manga demonstrates the potential of AI technology in specific application scenarios. It provides an effective approach to solving a long-standing technical challenge and brings more possibilities for manga-related digital applications.