mycel · a fast, decentralized web search engine in one Rust binary
This is what grew back.
† april 2026
Open-source web search in Rust. Essentially one developer. Archived by its owner; no successor.
what grew back: survivability as architecture. ~8,500 lines, 27 boring dependencies, and data formats that stay valuable even if development pauses for a year.
† 2023
The decentralized part was never built. Its design contained no ranking at all. Died of funding exhaustion.
what grew back: no moonshot. A node is complete alone; federation is additive. Ranking was designed first: BM25 × harmonic centrality.
asleep since 2023
"Elasticsearch in Rust." Said "far from production ready" for four straight years. Zero releases, ever.
what grew back: no custom index server. tantivy + SQLite + WARC: someone else's well-tested problem, with multi-decade format guarantees.
est. 2003 · still standing
A global index sharded across a DHT. Six-second timeouts; NAT'd peers that cannot answer; ranking nobody fixed.
the lesson: keep no global anything. Whole engines, explicit peers, and scores that are never compared across nodes.
est. 2021 · alive
Still standing. 969 million documents. One person. About $200 a month.
the seed: mycel is planted from this lesson: a million high-quality documents beat a billion mixed ones.
The mortality record above is the founding research finding. The failures were never the algorithms. They were weight: infrastructure bills, custom formats, global indexes, one exhausted maintainer. mycel is designed against the graveyard.
01 · seed
Nothing is crawled that you didn't plant. mycel seed blog.example.org activates a host and enqueues its root. Links discovered off-host become candidates, recorded in the webgraph but never fetched, until you promote them.
$ mycel seed blog.example.org
02 · crawl
robots.txt is obeyed per RFC 9309. One request per host, ever, enforced by the claim query rather than by good intentions. A 429 doubles that host's delay, sticky, never lowered again. And it refuses to crawl at all until contact_url puts your name in the user agent.
03 · store
Every fetch is appended to WARC, the same format Common Crawl publishes, and fsynced; only then does the same transaction advance the durable watermark. Kill the process mid-write: nothing corrupts, nothing orphans.
WARC is the source of truth. Everything else regrows.
04 · index
Text is extracted; near-duplicates are composted, exact by sha256, near by simhash. What survives enters a tantivy BM25 index. Corrupt it, upgrade it, delete it: mycel reindex grows it back from WARC.
10 Best Hikes Near Portland (2026 Edition, Updated)
10 Best Hikes Near Portland, 2026 Edition [Updated]
The 10 Best Hikes Near Portland, Updated for 2026
10 Best Hikes Near Portland (Updated 2026 Edition)
Best Hikes Near Portland: Top 10, 2026 Update
10 Best Portland Hikes: Updated 2026 Edition
10 Best Hikes Near Portland (2026 Edition, Updated)
skip_reason: dup-near · simhash Δ ≤ 305 · rank
mycel computes harmonic centrality over its own host webgraph, offline, and folds it into every score: bm25 × (1 + 0.3 × centrality). Text relevance dwarfs link signals; centrality is a boost, never the ranking.
06 · search
Phrases, site: filters, highlighted snippets. From your corpus, on your machine, in milliseconds. The daemon serves a UI and a JSON API on localhost.
Mycorrhizal networks and how forests share
https://fungi.example.org/nets
…fungal mycelium networks connect trees underground and move nutrients between them…
$ mycel search '"mycelium networks" site:fungi.example.org'
07 · federate
Peers dial each other by public key over QUIC. An allowlist is the only gate; trust is social, not cryptographic. Queries fan out and results interleave round-robin, so a peer's score can never buy rank. Corpora travel as sealed, blake3-verified WARC shards: no DHT, no tokens, no transitive flooding.
When both sides list each other, the network exists. Otherwise it simply doesn't.
the instruments
These panels run the real rules: the same invariants the binary enforces, animated. Poke them.
One request per host is not a promise in the fetch code; it is a property of the SQL that hands out work. Watch four hosts get crawled: each lane holds exactly one token, and the politeness gate must drain before the next claim. Throttle a host and its delay doubles, sticky, never lowered.
SELECT h.id, h.host, f.id, f.url, f.kind, f.attempts, f.depth,
h.robots_body, h.robots_fetched_at, h.crawl_delay_ms, d.sha256
FROM hosts h
JOIN frontier f ON f.id = (
SELECT f2.id FROM frontier f2
WHERE f2.host_id = h.id AND f2.state = 0 AND f2.next_attempt_at <= ?1
ORDER BY f2.next_attempt_at, f2.id LIMIT 1)
LEFT JOIN docs d ON d.url = f.url
WHERE h.state = 1 AND h.in_flight = 0 AND h.next_fetch_at <= ?1
ORDER BY h.next_fetch_at
LIMIT ?2A WARC member is appended and fsynced, and the same transaction that catalogs it advances shards.bytes, the durable watermark. So pull the plug. Go on. On reboot the shard is truncated back to the watermark: torn tails are unobservable, orphans are impossible.
Federated results interleave round-robin (local first, then each peer), deduplicated by URL. They are never re-sorted by score: BM25 scores from different indexes aren't comparable, so a malicious 99.9 buys nothing. Kill a peer mid-query; it contributes zero hits and zero delay.
The architecture is the product of adversarially verified research: 282 agents, three rounds, every load-bearing claim attacked by independent verifiers. 59 claims survived; 8 were refuted, and the refuted ones are logged so they can never silently re-enter the design.
The graveyard upstairs isn't decoration. It's section one of the research: the dominant failure mode of this project category is maintainer collapse and funding exhaustion, not algorithms. Every design decision cites its evidence.
plant one
Prebuilt binaries for Linux and macOS are on the releases page, or build from source. It refuses to crawl until you sign your name.
$ cargo build --release
$ mycel init # config, data dir, node identity
$ $EDITOR mycel.toml # set crawl.contact_url: sign your name
$ mycel seed blog.example.org
$ mycel crawl --limit 500
$ mycel search "some phrase"
$ mycel run # daemon: crawler + indexer + UI on :8080No planet-scale crawl required: bootstrap 1 to 100M documents from Common Crawl with two CSVs. Ranged fetches, throttled, resumable. Recipes in the README.
AGPL-3.0: run it, self-host it, fork it. Serve a modified mycel over a network? Publish your changes. The reciprocity shard sync asks of your corpus, applied to code.