the lexicon web

How this was made

The repo comes down as one CAR file from the account's PDS (com.atproto.sync.getRepo) — 91 MB, 301,721 blocks, parsed by a ninety-line CAR + DAG-CBOR reader with no dependencies. Every block whose $type is app.bsky.feed.post is a post; the Merkle tree never has to be walked.

Tokenising uses /rite/lexicon's stopword list, read from its source so the two surfaces cannot drift, plus three fixes that each changed the answer here. Pasted bsky.app links had put bsky, app and profile in the top ten most-used words. [a-z'] does not match a curly apostrophe, so every phone-typed don't was splitting into don + t — which is why don ranked seventh. And contractions needed folding onto their stems before the stopword list could see the it in it's. Before those three, the top of this vocabulary was punctuation and a URL scheme.

The topics are not from an embedding service. They are PPMI over co-occurrence within a session — a run of posts with no gap longer than forty-five minutes, plus anything sharing a thread root — reduced by power iteration and clustered with spherical k-means. Two things had to be got right for any structure to appear at all. The context unit had to be the session and not the post: at six content words a post, post-level co-occurrence is almost all zeros and ones, and the first run produced a flat eigenvalue spectrum with six of eight clusters empty. And the leading eigenvector had to be thrown away, because the first component of any PPMI matrix is the frequency axis — keeping it made every "topic" a frequency band.

Nine hundred words carry the embedding. The other thirty-eight thousand get a wedge by the company they keep: across the sessions a word appears in, sum the wedge membership of the embedded words beside it and take the argmax — but of the lift, not the raw count. On a raw vote the hub cluster wins everything, because being everywhere is what makes it the hub; the first build put 37,829 of 39,554 types into it.

No post text leaves the build. The data file carries word types, counts, document frequencies, first and mean dates, and a wedge number — a bag of words, which is what the chart is about. Shipping the prose would be a different and much more exposing artefact for no gain.

Your own web is built on your own machine. Type a handle and the whole pipeline above — the CAR reader, the tokeniser, PPMI, the eigendecomposition, k-means — runs in a Web Worker in your tab. The repo is fetched from that account's data server directly; it never touches this server, and neither does the result. It is the same engine.mjs that produced the committed example, not a second implementation, and the selftest asserts the two agree.

One outbound call is not covered by that, and it should be said plainly rather than buried: the name box is a typeahead, so the prefix you type is sent to Bluesky's public directory (app.bsky.actor.searchActorsTypeahead) to find matching accounts, and the avatars in the list are loaded from Bluesky's CDN. That is a lookup of public accounts by name and nothing more — but it is a request leaving your machine, which the rest of this page is careful not to make. It is also entirely optional: the box is an ordinary text field, so if the directory is unreachable, or you would rather it were not asked, type the handle in full and press Enter. Picking a suggestion instead of typing skips a round trip, because the list already carries the account's DID.

There are two ways in, and the slow one exists because of a hard floor. Asking for the whole repository at once is fast — eight seconds for fifty thousand posts — but a single 91 MB response costs the browser about 80 MB to receive, measured with every byte thrown away unread. No amount of care in the reader touches that number; only not asking for it in one piece does. Gentle mode walks com.atproto.repo.listRecords a hundred posts at a time instead, so nothing larger than about 70 KB is ever in flight. It takes several minutes rather than several seconds and produces exactly the same web — posts are sorted by their timestamps, so the order they arrive in does not survive into the answer. If a build ever dies without an explanation, this page turns gentle mode on by itself the next time you open it.

The archive is never assembled. It would be easier to download the whole repo, join it up, decode every record and then start — and that is what this did at first, which cost about 390 MB of browser memory for a 91 MB account and killed the tab on bigger ones. The response is read as it arrives instead, and each post is reduced to a list of word numbers and a date the moment it is seen: the text, the facets, the embeds, the link cards, the language tags and the reply refs are all walked past or dropped on sight. What is alive at the end of the download is roughly the size of the answer rather than the size of the repo — about 120 MB now, and quicker, because most of the old cost was building things in order to throw them away.

Two things follow from that which are worth knowing. It is a real download: a 50,000-post account is about 90 MB. And a repo only has what its server will hand out — com.atproto.sync.getRepo is public on the main Bluesky hosts, but a self-hosted PDS can decline, and then there is nothing to read.

Build script: build.mjs · CAR reader: car.mjs · data: data.json · the weaver next door: silk