About this project
A student project for Smart India Hackathon problem SIH26096: group unindexed heritage scans by topic without labels, and let a new page be checked against those groups.
Where the books come from
The National Digital Library of India gathers records from many libraries; the scans behind many of them come from the Digital Library of India, whose copies are served by the Internet Archive. The sample was drawn from that collection with a fixed random seed, an equal share per language and per era, published up to 1960 or undated.
Each book is read from the archive’s own whole-book OCR text where it exists. Five pages from inside each book are also read with Tesseract 5 in the book’s own script, to measure OCR quality and to test how uploads are placed.
- Collection
- digitallibraryindia, Internet Archive
- Languages
- English, Hindi, Sanskrit, Bengali, Gujarati, Telugu, Kannada, Tamil
- Pages per book
- 5, at 15, 30, 50, 70 and 85% of the way through
- OCR
- Tesseract 5 with orientation and script detection
- Methods
- Hierarchical clustering (Ward, average, complete), DBSCAN, PCA, silhouette
Rights
The catalogue marks these scans' rights as not available. The project keeps only the text it read, the catalogue details and links back to the archive. Page images on this site are shown by link from the Internet Archive and are not copied or redistributed.
What it cannot do
- Read handwriting: the archive’s “manuscripts” here are printed books.
- Give a book a subject heading: groups are patterns of shared words, named after their most distinctive ones.
- Promise a group is right: the study page reports how stable each grouping is and where it is unsure.
- Read Urdu: Tesseract reads its script poorly, so it was left out of the sample.
- Work offline: the topic model is Google’s Gemini Embedding 001, so checking a page needs an internet connection, and the page’s text is sent to Google.
Team
- Data and OCR
- Rajith Sundar
- Clustering study
- Etikala Sai Pranav
- App and delivery
- Vemuri Rohan