Personal·2026·Web App · Data Visualization · Open Data
Catadex — A Directory of Specialist Catalogs
1,248 specialist data catalogs in 34 fields, explored through cards, a relationship graph and guided trails, and searchable in Thai and English

- Role
- Solo developer — data model, data curation and checking, search, graph, API and testing
- Team
- Solo project
- Duration
- 1–4 Oct 2026
- Year
- 2026
Key figures
- 1,248
- catalogs
- 34
- fields
- 679
- topics
- 5,321
- connections in the graph
- 329
- Thai–English synonym groups
- 9
- guided trails
- 62
- automated tests passing
Catadex gathers specialist “catalogs” — astronomy databases, species records, minerals, old maps, sound archives — into one directory. Each entry has a cover, a Thai description, tags and a link to the original site. You can explore it as cards or as a relationship graph (Cytoscape.js) that connects catalogs to fields, topics, shared domains and data that demonstrably overlaps. Search understands Thai and English, links are checked automatically, and there are personal collections, guided trails and a read-only public API. The starting data is 1,248 entries, 679 topics and 5,321 connections that I curated and checked myself, and it doesn’t claim to cover every website.
The problem
- 01Excellent specialist data sources exist all over the web but are hard to find, and collections of them are usually flat lists that don’t say which site relates to which.
- 02Thai users search in Thai while most sites are named and tagged in English, so literal search finds nothing.
- 03Links die all the time and some sites block bots, so declaring a link dead too early hides good entries for no reason.
- 04Letting users add an entry from a URL means the server fetches other people’s pages, which risks SSRF and excessive resource use.
The approach
- 01Six tables (Category, Catalog, Tag, CatalogTag, GraphNode, GraphEdge) on SQLite. Each graph node stands for exactly one Catalog, Category or Tag and is guarded by CHECK constraints. Four edge types — RELATED_FIELD, SHARED_TOPIC, SHARED_DOMAIN and DATA_OVERLAP, which must carry an evidence URL — and data overlap is never inferred from two entries sharing a category.
- 02Search is ranked by match quality (title > tag > domain > description) and understands Thai and English across each other through a 329-group synonym dictionary. It segments run-together Thai, highlights matches, suggests a spelling when nothing is found, and shows partial matches when no entry matches every word.
- 03Every view lives in the URL — query, field, topic, sort, Grid/Graph, graph field and the open entry — so it survives a reload, a share and the Back button. Unknown values in an old link are skipped rather than opening an empty page, and shared links preview with a title, description and cover.
- 04Graph: search a node and zoom to it, and “Find a path” between two catalogs through topics, fields, domains or evidenced data links (20 documented relations such as PDB, INSDC and OpenStreetMap). The home page doesn’t embed the graph; it loads from /api/graph when you open that view.
- 05Link status: a checker runs 100 links per batch through GitHub Actions every three hours. A link is flagged “Link may be down” after two failures in a row, sites that refuse bots or have certificate problems are marked restricted rather than broken, and the details page shows the check date and a Wayback Machine link.
- 06Several personal collections with a note per entry, JSON export/import, and share links that embed the list (notes aren’t shared). Nine guided trails are ordered with a note for every step, and bookmarks persist in the browser’s localStorage.
- 07The metadata fetcher accepts only public HTTP/HTTPS on standard ports and rejects credentials, localhost and private or reserved IPs (including encoded IPv4 and IPv4-mapped IPv6). It validates DNS on every redirect, pins the approved IP to the connection against DNS rebinding, and caps HTML at 2 MiB, 4 redirects and 10 seconds. POST endpoints accept JSON only, with a 16 KiB body limit and 20 requests a minute per IP.
- 08A read-only public API (/api/v1/*) with search, filters and pagination, CSV export and an Atom feed of new entries. Covers use a site’s own OG preview when there is a usable one, otherwise a cover drawn for that entry by field (a Python script), with 480px WebP thumbnails generated by sharp.
The result
- 01A directory you can explore three ways — cards, a graph and guided trails — where every view can be shared as one link.
- 0262 automated tests pass, covering rejection of internal URLs, the metadata parser, search, view state, graph paths, link status, collections, the public API and the seed data, alongside Playwright end-to-end tests in the repo.
- 03The data is honest about itself: it avoids numbers that keep changing (such as species counts) and documents the limits of link checking. The latest audit is stored in docs/catalog-link-audit.json.
Deliverables
- A Next.js 16 site with card, graph (Cytoscape.js), trail and collection views
- A Prisma + SQLite model with migrations and an idempotent seed (1,248 catalogs)
- Thai–English search, access filters, and a public /api/v1 with CSV export and an Atom feed
- A link-status checker (script + GitHub Actions) and a safe metadata fetcher
- Cover and thumbnail scripts (Python, sharp) with image provenance documented in ASSETS.md
- 62 node:test tests and Playwright end-to-end tests
Gallery
Search, filter and browse
16:9 / LG
The home page: 1,248 specialist catalogs in 34 fields with 5,321 connections, and an editor’s pick

Filtered to Astronomy & space (74 catalogs): cards with a cover, domain, summary and tags, plus topic, access level and “Has an API” filters

Searching in Thai (“planets”) returns 78 results through a 329-group Thai–English synonym dictionary, with matched words highlighted

Access filters: Open together with “Has an API” and “Reuse terms listed”. Access comes from tags or records I checked by hand, and unknown entries are never matched
Retrospective
Challenges
- Thai–English search with no external service: I built the synonym dictionary myself, segmented Thai lightly, and ranked an exact title match above a description match.
- Telling a dead site apart from one that refuses bots needed a restricted state and a two-consecutive-failures rule, otherwise links would be mislabelled.
- A graph of more than a thousand catalogs is hard to read, so fields sit on an ellipse, shared topics in the middle, label size follows the zoom level, and you can filter to a single field.
- Keeping entry IDs stable when re-seeding so users’ bookmarks survive, and making the seed re-runnable without creating duplicates.
Learnings
- Keeping view state in the URL makes sharing, going back and testing much easier.
- Curated data has to state its own limits plainly — no claim of completeness, no relationship without evidence — to be believable.
- A fetcher of external content has to be defensive from the start (IP checks, pinning, size, time and rate limits), not patched later.
Interested in work like this?
Tell me about your brief — I reply within 24 hours.