Archivist Lab: the Historical Documentary Atlas
- Sector
- Research & Knowledge
- Work
- Custom Software
- Duration
- 22 weeks
5,657 pages
33 volumes of Gulf history, 1622 to 1810, in two languages
16,377 events
dated, placed, and put on a map and a timeline
3,985 ↔ 3,985
narrative records matched across English and Arabic, enforced at load
150 queries
retrieval benchmark across six question types
Fig. 1
From a research question to a cited analysis
A question is worked across the whole collection, and the analysis that comes back links every claim to its source passage.
- 1. You begin with a research question in your own words, not a keyword.
- 2. Search, identities, events, relationships and the map work across the whole collection at once.
- 3. What comes back has tables, figures and sources, and each claim links to the passage it came from.
01The starting point
A major archive, with no way to work across it.
The corpus was a set of thirty-three InDesign-exported PDF volumes covering Persian Gulf, Arabian and Persian history from 1622 to 1810: 1,939,670 words across 5,657 pages. As documents they were readable. As research material, they could not be searched, structured or cited reliably: no records, no entities, no dates a machine could sort, and no way for a historian to cite a specific passage and have anyone else find it again.
A parallel Arabic edition of the same thirty-three volumes existed, but it presented an additional problem. The PDF text layer was glyph-scrambled and unrecoverable, so the Arabic scholarship was effectively invisible to every tool that could read the English.
The requirement was not a search box over PDFs. It was a structured, bilingual, citable knowledge graph (people, places, ships, commodities, events and the relations between them) with every claim traceable to the page it came from.
Fig. 2
One person, many spellings
The same person is named differently across thirty-three volumes. The names are resolved to one identity, and every spelling is kept.
- 1. The same person is named differently from one volume to the next, and again in the Arabic edition.
- 2. Those names are resolved to one identity for that person.
- 3. Every spelling is kept with that identity.
02What we built
A bilingual evidence system built for real research.
Extraction is deterministic and font-aware rather than model-guessed: year headers are identified by weight and point size, body text by its own typographic signature, so segmentation is reproducible across all thirty-three English volumes. The Arabic edition carried no recoverable text at all, and was recovered separately by page-level vision transcription at 200 dpi: 8.31 million characters the original text layer could not surrender.
Entities are resolved through an eight-tier cascade (exact, normalised, transliteration-family, token-overlap, prefix, phonetic and edit-distance among them) backed by thirty-eight curated transliteration families and a hand-built gazetteer. Candidate merges are scored across name overlap, shared correspondents, shared places and year overlap, with explicit vetoes for conflicting residences and titles.
Events, relations and context facts are layered on top and written into Postgres 16 with PostGIS and pgvector. A FastAPI service exposes the graph; a bilingual Next.js 14 application renders it as an interactive Leaflet map, a correspondence network, a facsimile viewer and a cited research agent designed to quote only from verified source passages: citations are sliced from character offsets, or refused.
Fig. 3
Putting the chronicle on a map
The chronicle has no coordinates. Places named in its text are resolved to points on a map, each carrying the events that happened there.
- 1. The chronicle names places in its prose but gives no coordinates for any of them.
- 2. Each named place is resolved to a point on the map.
- 3. The events that happened there are attached to the place, so the map shows what happened where.
03How the system works
Questions become maps, records and citations.
Fig. 4
Recovering the Arabic edition
The Arabic files give a computer no readable text. Their numbers still come through and mark each episode, and the words are read from an image of the page.
- 1. Asked for its text, the page returns scrambled letters, but its numbers come through intact.
- 2. The words are read from an image of the page instead.
- 3. The numbers set where each episode begins and ends, so what is read from the image cannot move a boundary.
Fig. 5
Both editions must match before loading
After every load the English and Arabic records are compared both ways. If either side has a record the other lacks, the load stops.
- 1. Each English narrative record is checked against its Arabic twin, in both directions.
- 2. A record with no twin on the other side fails the check.
- 3. The load stops there, and the check cannot be switched off without changing the code.
04What became possible
Research across the whole archive.
Pages of 1622–1810 Gulf history, opened as a map
5,657 pages
Printed only → Searchable, two languages
Arabic episodes a computer could read
3,985
Matched one-to-one with the English, in both directions
Dated events placed on the map and the timeline
16,377
Ask which year, instead of reading for it
Retrieval benchmark, six question types
150 queries
Benchmark result: the system did not yet meet its cross-language retrieval threshold, and says so.
Built with
Next.js 14React 18TypeScriptTailwind CSSLeafletD3.jsnext-intlFastAPIPython 3.11SQLAlchemy 2.0PostgreSQL 16PostGISpgvectorAlembicPyMuPDFspaCyStanzagemini-2.5-flashLiteLLMBAAI/bge-m3DockerCaddy
What would become possible for your work?
Bring us the material, system or idea you are starting with. We will help work out what it could make possible.
Keep reading.
Staffing & Recruitment
From Incoming CVs to Client-Safe Candidate Profiles
Manlytic Group’s recruiters work from their phones. CVs reach them on WhatsApp as scans, photos and two-column tables, passport details have to be exact, and anything sent to an employer has to leave the agency as the only route to the candidate.
- Every file made for an employer is built without the candidate’s contact details, then searched again before it can be shared.
- CVs are read without guessing: gaps stay visible, and anything unsure waits for a person to confirm.
- A likely duplicate is reported and held for a recruiter, never merged on its own.
Regulated Services
Building an AI Assistant You Can Control
How a private AI product company turned difficult regulatory material into a controlled assistant for professional firms.
- Your own material, not a public model's: administrators decide what is in the knowledge base and what is not
- Add a new regulation yourself: upload the PDF and the system cleans, splits and indexes it without a developer
- Provision a company, invite named people into it, and cut their access server-side when it ends
Commerce
Making Interior Products Easier to Find
How a MENA AI studio turned an interior photograph into searchable product language, UAE results and a shortlist customers could export.
- Photograph something you saw and get back things you can buy nearby, without typing a word
- Keep what you like, set quantities against it, and export the shortlist as a PDF you can hand to someone else
- Hand it a photograph of something that is not an interior item and it stops, rather than confidently returning the wrong furniture