Content discovery for CIDs, without the DHT

If you've published data with IPFS, you know CIDs are the easy part. The hard part is telling the world where to fetch them.

ATFISL takes a different approach to that problem. Publishers write signed location records to their own AT Protocol repositories. Indexers collect those records, and clients fetch the bytes from any URL and check them against the CID. You don't need a DHT or libp2p, and a plain HTTP server is enough.

What CIDs give you

IPFS popularised the idea of universal content addressing with Content IDs (CIDs). The premise of a CID is simple but highly ambitious: a single addressing scheme for immutable data. This was not a new idea, of course, but the web evolved around location addressing with URLs instead.

IPFS combined content addressing with peer-to-peer networking, threw in a DHT, and attempted the impossible task of getting it to work in browsers by implementing the stack on top of web standards (the latter a thankless job that never fully came to fruition despite the efforts of some great people and many resources, largely because browsers are hard to change and the companies funding their development have different interests).

IPFS is like BitTorrent with universal content-addressed identifiers as a first-class citizen. Instead of passing around a torrent file (or magnet URI), you simply pass a CID. As it turns out, that's a huge UX improvement for anyone distributing content online. A CID is derived from the contents of whatever it addresses, so it's “self-certifying” and timeless. It's the only thing you need to save, pass around, and maybe associate with some metadata.

Because CIDs have both a string and a binary format, with implementations for every language, they can be introduced as a modular building block in existing data management and web publishing software.

Then came production

The promise was huge, but the problems often started when you went to production. As the volume of data you published increased, so did the resource consumption, often asymmetrically.

For a long time, IPFS nodes were resource-hungry monsters, frequently crashing home routers due to excess network activity, and requiring a degree in networking black magic to run correctly. Even in 2025, the hourly network crawls of Kubo's Accelerated DHT Client could crash consumer-grade routers or trigger throttling from residential ISPs. Many early IPFS adopters painfully discovered the tax of content addressing, concluding that maybe it wasn't worth it for their use case.

One of the shifts that followed was to redefine IPFS as a set of standards, with interoperability conceptualised as HTTP APIs. Instead of relying on a whole vertical networking (libp2p) and data (IPLD) stack packaged as a node, you could pick only the data part (CIDs, CBOR, UnixFS, IPLD) and do publishing, routing, and transfer over HTTP. It was a noble cause, and it largely succeeded: almost all of the old public IPFS gateway traffic has shifted to the Service Worker Gateway, which lets you fetch and verify CIDs in the browser without being tied to any single provider.

Content discovery is rigid

Despite the welcome shift to HTTP, content discovery (also known as providing in IPFS nomenclature) is stuck between a rock and a hard place. The DHT is prone to Sybil attacks, and for a long time it didn't scale. Until late 2025, every CID needed its own 10–20 second lookup, repeated every 22 hours, so a Kubo node could keep only about 5,000 CIDs discoverable. Provide Sweep fixed much of this by batching CIDs that share the same closest peers, which cuts lookups by about 97%. IPNI, the alternative, requires running additional software and isn't suited to the typical Kubo/IPFS Cluster user.

Scale aside, two challenges remain in real world deployments of IPFS:

• Private and permissioned networks. Much of the IPFS tooling focused on the public network (“Mainnet”), so private or permissioned setups are relatively uncharted territory.
• “dumb” HTTP servers. The DHT only supports libp2p providers. HTTP providers, which only IPNI supports, need to be a trustless gateway. You can't simply serve a CID-addressed file from an S3 bucket or a self-hosted Caddy server.

A real-world problem: data consortia

This can seem abstract, so let's make it concrete with a real-world problem. How do you build a data consortium where a number of trusted institutions can reliably publish public CID-addressed data (like ORCESTRA)? One key requirement of a content-addressed system is to let consortium members selectively replicate a subset of each other's data to increase redundancy. There are many ways to approach this, but for it to be verifiable, it has to involve an identity layer, so that you can verify each consortium member's claims about the CIDs they're providing. Ideally, that identity is tied to the real world via ICANN domains.

ATFISL — flexible content discovery on the AT Protocol

As it turns out, we don't need to build that identity layer from scratch. The AT Protocol (and the PLC directory) already serve this role well.

AT Protocol repositories have public reads and permissioned writes: anyone can read your data, but only you can write to it. Every record is part of a signed commit, and the repository itself is a tree of CIDs, so you get integrity and attribution without inventing yet another signature scheme. Each account has a DID tied to a handle, which is just a domain name. The handle points to the DID via DNS or a .well-known file, and the DID document points back to the handle. That bi-directional attestation means only whoever controls example.org can claim data.example.org. And because records live in the publisher's own PDS (Personal Data Server), publishers keep credible exit: they can move hosts and their data moves with them.

ATFISL uses all of this as its publication layer. A publisher announces “this CID is retrievable at these URLs, until this time” by writing a record to their own repository. An indexer watches the network and builds a lookup table from CIDs to locations. A client asks the indexer, fetches the bytes, and verifies them against the CID. In other words, CIDs signed by institutions.

Publishing a location

A location record lives in the dev.fisl.at collection, and the record key is the CID itself:

{
  "$type": "dev.fisl.at",
  "addrs": [
    { "url": "https://berjon.com/.well-known/rasl/bafkreifn5yxi7nkftsn46b6x26grda57ict7md2xuvfbsgkiahe2e7vnq4" },
    { "url": "https://mirror.example.com/kitten.jpg", "expires": "2026-09-01T00:00:00Z" }
  ],
  "size": 94201,
  "expires": "2026-09-18T00:00:00Z"
}

Using the CID as the key is probably the most important design choice. Each repository holds at most one record per CID, so the regular repository operations carry all the meaning: a put creates or replaces the announcement, and a delete retracts it. There's no separate update or revocation mechanism to learn (or get wrong).

The expires field exists because retrieval locations rot. Domains lapse, certificates expire, servers move. A record can mix a stable mirror with a short-lived one, each with its own expiry, and if you want content to stay discoverable you simply re-publish before it expires.

A nice side effect of keying by CID is that every PDS becomes a tiny discovery endpoint on its own. A single getRecord call with the CID as the record key answers “does this DID serve this CID?” without any indexer in the middle.

What if content discovery worked well

Compare this with providing to the DHT. There, your node has to re-announce every CID to about 20 peers every 22 hours, because provider records expire after 48 hours. The cost grows with the number of CIDs and never stops. The provider records are unsigned claims from a PeerID, and a lookup means walking across many peers (hopefully honest ones).

With ATFISL, you write one record per CID to your repository, batching many into one commit with applyWrites if you have a lot of them. Nothing needs to stay online except your PDS, which is online anyway, and the record lasts until the expires time you choose, not a lifetime the network sets. An indexer follows the firehose (or backfills repositories), checks that each record was part of a signed commit, and stores one row per (did, cid) pair. If it loses its state, it replays the repositories and rebuilds it from scratch. A client then asks the indexer for a CID, gets back a list of URLs and who announced them, fetches from whichever it supports, and hashes the bytes.

Indexers can be global, or scoped to the DIDs you care about, so you can run one for your data consortium, mirror network, or application. And because the data lives in the publishers' repositories rather than the indexer, an indexer is replaceable: anyone can build another one from the same records.

FISL — the lookup API

The query side is a separate spec called FISL. If you've used the IPFS Delegated Routing V1 HTTP API, it will feel familiar, because that's what inspired it:

GET /routing/locations/{cid}/json

{
  "locations": [
    {
      "publisher": "did:plc:ewvi7nxzyoun6zhxrhs64oiz",
      "addrs": [
        { "url": "https://berjon.com/.well-known/rasl/bafkreifn5yxi7nkftsn46b6x26grda57ict7md2xuvfbsgkiahe2e7vnq4" },
        { "url": "https://mirror.example.com/kitten.jpg", "expires": "2026-09-01T00:00:00Z" }
      ],
      "size": 94201,
      "expires": "2026-09-18T00:00:00Z"
    }
  ]
}

Roughly speaking, provider records become location records, PeerIDs become publisher DIDs, and multiaddrs become plain URLs. There are a few deliberate differences though. The response format (json or jsonl) is part of the path rather than negotiated with the Accept header, so every answer has exactly one URL and caches well on any CDN. JSONL lets an indexer stream results as it finds them. A CID with no locations returns a 200 with an empty list rather than a 404. And since CORS is required, browsers can query any indexer directly.

FISL doesn't depend on ATFISL either. It only defines the query interface, so any index with location records can sit behind it.

A dumb HTTP server is enough

This brings us back to one of the earlier problems. The URL in a location record can be any https URL: a file in an S3 bucket, a static site, or a RASL server at /.well-known/rasl/{cid}. The server doesn't need to speak libp2p or be a trustless gateway. It just needs to return the bytes.

This is safe because the client does the verification. It hashes what it receives and compares it with the CID, and if the bytes are wrong, it drops them and tries the next URL. A bad location costs one failed request, not a corrupted file. That's the whole point of content addressing, and it's what lets you accept location claims from anyone.

The URL scheme names the transport, so FISL isn't tied to HTTP. A record can list an https URL alongside a radiroh:// URL, which names an iroh endpoint serving content over iroh-blobs. Clients pick the schemes they support and skip the rest. Adding a new transport means defining a scheme, not a new version of the API.

One caveat for IPFS users: ATFISL keys records by DASL CIDs (and BDASL CIDs for large content). These are CIDv1 in base32 with the raw or dag-cbor codec and SHA-256 (or BLAKE3 for BDASL). So a UnixFS root CID (dag-pb) isn't a valid key, and how best to handle UnixFS DAGs with trustless gateways is still an open question in the spec.

What about PDS blobs?

The AT Protocol already stores files as blobs. You upload them to your PDS, reference them from records, and anyone can fetch them by DID and CID. Blob CIDs are even valid DASL CIDs.

But blobs are designed for avatars and images, not datasets. The PDS operator pays for storage and sets an upload limit, usually tens of megabytes. In practice, apps serve blobs from their own CDNs, such as https://cdn.bsky.app/img/feed_fullsize/plain/{did}/{cid}. Those URLs are app-specific and scoped to one account, and they may serve resized or transcoded bytes that don't match the CID.

ATFISL keeps what the AT Protocol does well (identity, signatures, small records) and moves the bytes out of the PDS. The record is a few hundred bytes, however large the content. The content can live anywhere, any number of publishers can announce it, and every copy can be verified.

Back to the consortium

So what does the data consortium look like with ATFISL? Each institution has an AT Protocol account whose handle is on its own domain, say data.example.org. When it publishes a dataset, it writes a dev.fisl.at record for each CID pointing at its own HTTP server. When another member replicates a subset of that data, it writes its own record for the same CIDs pointing at its mirror. Since those records sit in separate repositories, they never conflict.

The consortium runs an indexer scoped to its members' DIDs (though anyone else can run one too). A client asks the indexer for a CID and gets back every member serving it. It can decide which publishers it trusts or prefers, and it verifies the bytes either way.

Every claim is signed by an identity tied to a real world domain, every replica is visible, and no member has to run a peer-to-peer node.

Finding the data is a separate problem

There's a catch in all of this: ATFISL assumes you already have the CID. It answers “where can I get the bytes for this CID?”, but not “which dataset do I want, and what's its CID?”. That's metadata and data discovery, and while it's closely related, it's a separate problem (and arguably the harder one).

The consortium has this problem too. Before anyone fetches a file, a researcher needs to find the dataset: what it contains, who published it, when, and under which licence. Today that knowledge is scattered across data portals and catalogues, each with its own search and its own API.

This is what Matadisco sets out to solve, and it takes the same approach as ATFISL. Publishers write records to their AT Protocol repositories, this time in the cx.vmx.matadisco collection. Each record points to a resource containing the metadata (for example a STAC item or a catalogue entry), along with when it was published, optional tags, and an optional preview. Anyone can then follow the firehose and build their own portal or workflow on top. It's already being used to publish records for Sentinel-2 satellite imagery from Element 84's Earth Search catalogue, and for the German geodata catalogue. The IPFS Foundation's announcement goes into the ideas behind it.

The two fit together nicely. Matadisco helps you find the dataset, and once you have a CID, ATFISL tells you where to fetch it. Both are signed by the same publisher identity, both flow through the same firehose, and a single indexer could follow both collections. But neither depends on the other, and keeping them apart lets each one stay simple.

Trade-offs

None of this comes for free. Location records are claims, not proofs, and a publisher can announce URLs that don't work. Verification limits the damage to a wasted request, and clients should de-prioritise publishers whose locations keep failing.

Because there's one record per CID per repository, adding or removing a single URL means reading the record, editing it, and writing it back (use swapRecord to avoid clobbering concurrent updates). And indexers are services someone has to run. They're simple and replaceable, but they're not a DHT. For many private, semi-private, and permissioned use cases, that's arguably a feature.

Try it

The specs live at dasl.ing/fisl.html and dasl.ing/atfisl.html. The console at fisl.dev lets you sign in with your AT Protocol account, publish a location record, and look up a CID straight from a PDS.

Feedback is very welcome, especially from anyone who has battled with providing on IPFS.