HyperSearch

A distributed P2P search engine. Every Hyperspace node is an indexer, a ranker, and a crawler. The cascade resolves locally first, then peer-fanouts, then the DHT, then a live crawl — in that order. Federated DPO tunes the scoring weights every minute against real network clicks. BSP PageRank with seven layers of anti-sybil produces verifiable authority. SWC compression gives agents query-driven access to 3× more page content per token of context. The network IS the search engine.

Web 1.0 had one company crawling. Hyperspace has every node crawling. Different physics — quality compounds with N, not with $.

Four-tier cascade with progressive latency budgets. Most queries terminate at tier 1 or 2 — tiers 3 and 4 only fire when they must.

Why HyperSearch

Today's agent search products — Exa, Parallel.ai — are sophisticated DAGs over a static index. The DAG is well-engineered. But the index is whatever they crawled, the ranker is whatever their team last tuned, and the answer is whatever their LLM provider released last. Quality is bounded by their crawl budget, their staffing, and their pricing.

HyperSearch runs on a different physics. Every Hyperspace node is a search-engine participant: it crawls (passively, via the browser extension; actively, via recipes), it stores index shards (BM25 + HNSW), it answers queries from its peers, and it contributes to the BSP PageRank computation. The same network that answers questions is the network that generates the knowledge being answered — via autoresearch agents publishing experiments back to the index.

The result: quality compounds with N nodes, not with $ spent. A static system loses ground over time to a self-improving one. After 6–12 months at scale, the gap is structural.

What is it

HyperSearch is the search subsystem of the Hyperspace P2P network. It runs as part of the standard hyperspace CLI — no separate install. When you start a node with the crawl,index,search capabilities, your machine joins the global search mesh.

Architecturally, every node has six things:

1. A crawler. Active recipes (GitHub, arXiv, HF, npm, news) and passive observation (every page-view in the browser extension). Coordinated by a CRDT frontier with consistent-hash leasing.
2. A document builder. Ten-stage pipeline: field extraction → readability → entity extraction → classification → schema.org → code → summary → quality → multi-field embeddings → SWC compression.
3. An index. Local HNSW + BM25 shards with 3× replication, sharded by consistent hash, served over the libp2p search protocol.
4. A ranker. A hot-swappable BlendedScorer combining seven signals (title, body, BM25, authority, freshness, click-rate, diversity), tuned by federated DPO.
5. A query cascade. Local → peer fan-out → DHT → live crawl, with progressive timeout budgets.
6. An intelligence loop. Click signals → DPO pairs → SGD → gradient gossip → FedAvg → LoRA adapter → updated scorer.

The same node also runs distributed-rank, a separate package that owns BSP PageRank, anti-sybil scoring, and Merkle-committed rank state. PageRank scores feed back into the BlendedScorer's authority signal continuously.

Install

If you've already installed Hyperspace, you have HyperSearch. Just enable the search capabilities on your node:

By default, your node now contributes crawl coverage from your browsing (if the extension is installed), serves search queries to peers, stores its share of the global index, and earns work-points for each operation.

The cascade

A query travels through four tiers in increasing latency. Most queries terminate at tier 1 or 2; the cascade only descends when no tier above produced a high-quality result.

The 4-level cache hierarchy under tier 1 is itself a mini cascade: L1 in-memory LRU (1k entries, 30s TTL, ~0ms), L2 node-local extended (10k entries, 5min TTL, ~1ms), L3 peer cache ring (ask 3 nearest peers, ~50ms), L4 compressed-DOM cache (5k entries, 10min TTL, ~100ms). Hot items promote upward; cold items expire and demote. At Zipfian query distribution, the top 10k queries hit L1 across the network within seconds.

SWC pipeline. Raw page → semantic tree → budget-constrained view → query-driven unfold. 3× compression with hierarchical recovery.

SWC compression

Exa's Contents API gives you "text + Readability extract + optional summary." That's a static truncation. SWC — Semantic Web Cache — gives agents structure-aware, query-driven access to a 3×-compressed hierarchical representation of the page.

The pipeline:

1. DOMChunker classifies regions by semantic role (nav, form, list, content, section). Marks stable chunks (nav, footer, sidebar — 30-day TTL, deduped across the domain) vs dynamic (price, date).
2. ChunkCompressor runs an LLM bottom-up: leaves → 1/10–1/20 summary, parents → structural description, root → page overview. Result: a SemanticTree where every node has { summary, fullContent, embedding, isStable, interactiveElements }.
3. TreeSerializer outputs a budget-constrained view (default 500 tokens for a 15k-token page). Root + section summaries first, on-demand expansion thereafter.
4. SemanticUnfolder takes the agent's query and selectively unfolds the most relevant subtrees within budget. The agent receives query-conditional content, not arbitrary truncation.
5. SymbolicCompressor represents repeated patterns (e-commerce listings, log lines, tables) as template + delta values, dropping token-count further.
6. P2P distribution: trees stored in DHT keyed by URL hash; stable chunks shared across the domain.

The categorical advantage: an agent asking "what's the price?" on a product page gets the dynamic price subtree. With Readability+highlights, it gets pre-decided highlights that may or may not include the price. For deep research over 50 pages, 3× compression with hierarchical recovery means the agent fits 50 pages in 100k context vs needing 300k. Interactive elements (buttons, form fields) are preserved at every tree level — so SWC output is also actionable for browser-agent flows.

Distributed PageRank

Authority is the layer where SEO spam attacks search engines. Google had 25 years of internal anti-spam ML. Exa is younger and its anti-spam is opaque. HyperSearch's authority signal is a BSP PageRank computation with structural anti-sybil, hourly Merkle commits, and EigenTrust propagation — all in 8.9k LOC of @hyperspace/distributed-rank.

An auditor can re-run BSP against the committed Merkle roots and verify the score for any URL at any historical hour. SEO spam attempts produce signatures the system structurally identifies. There is no editorial committee that can de-rank you for the wrong reason.

BSP supersteps. Compute → message → barrier → converge. Anti-sybil + EigenTrust + collusion detection run on every step; Merkle root committed hourly.

The intelligence loop

The single thing centralized SaaS providers cannot replicate: the network's ranker, embedding model, query reformulator, and crawl planner all improve continuously on real network traffic. @hyperspace/search/intelligence — 18 components, ~25k LOC — runs the loop:

Additional intelligence components: QueryPatternAnalyzer (trending queries, seasonal patterns), ReformulationLearner (better query rewrites from click chains), IndexGapDetector (failed queries trigger targeted crawls), DomainExpertiseTracker (each node specializes in its user's domain — SearchPeerSelector routes accordingly), CanaryDeployer (A/B test new weights on live traffic, auto-promote winners), DataShapleyValuator (poisoning defense — valuates each peer's gradient contribution, excludes low-Shapley contributors), SearchExperimentLoop (the autosearch loop running MS MARCO LtR research continuously).

Click signal feeds the loop. Eight stages. The ranker hot-swaps without restart and improves continuously, every minute, against real network traffic.

The compound loop

The structurally-impossible-to-copy advantage. Hyperspace runs autoresearch and autosearch on the same P2P substrate. Autoresearch agents produce knowledge. Autosearch nodes index and rank that knowledge. The output of one is the input of the other.

At 100k nodes, every research finding gets crawled and indexed within ~5 minutes; every search query teaches the ranker via DPO within ~1 minute; every novel ranking-weight discovery propagates network-wide within ~10 minutes. The intelligence loop cycle time drops below a minute. See COMPOUND_LOOP_DEMO.md for the full demo with concrete curl commands.

The loop has no external input. Output of one is the input of the other. The same network that answers questions is the network that generates the answers.

Migration

Both Exa and Parallel.ai have drop-in replacements. The migration is literally the import path and (optionally) deleting your API key.

Two-line diff. Same API surface. Local-first by default. Cost-receipt in every response.

From exa-js

Every Exa method now returns the same shape your existing code expects, plus a costDollars receipt showing what the call would have cost on Exa, Parallel.ai, and what it cost in Hyperspace points (always $0 paid).

From parallel-web

Method support: client.search() · client.taskRun.create() · client.taskRun.retrieve() · client.taskRun.pollUntilFinished(). Processor tiers (lite/base/core/pro/pro-fast/ultra/ultra-fast/ultra8x) map to research step budgets.

Or use the typed graph framework

Starting fresh? @hyperspace/search-graph is our equivalent of Exa Canon — typed nodes, automatic parallelization, cost receipts.

Combinators: pipe, parallel, map, tap, conditional, retry, timeout, leaf. 18 smoke tests pass; framework is ~700 LOC. See @hyperspace/search-graph.

Cost comparison

Every search response now carries a costDollars field showing what the operation would have cost on Exa, what it would have cost on Parallel.ai, and what it cost on Hyperspace (always $0 paid — Hyperspace points are network-internal credits earned by serving). Aggregating these over a day's worth of agent activity gives you the savings number.

Money never leaves the network. Compute is paid in points the node earned by serving. The flywheel is closed.

Sample response shape

REST API surface

Every Hyperspace node serves the same REST API at localhost:8080/v1/…. The compat shims and the SDK use this surface internally; you can also hit it directly with curl.

Every search-family endpoint returns a costDollars receipt. The /v1/version endpoint exposes service discovery for compat shims so they can confirm they're talking to a Hyperspace node.

SDKs and compat shims

Python SDKs are available at @hyperspace/search-sdk-python (the hypersearch package).

14 subsystems

The @hyperspace/search package is ~106k LOC across 14 subsystems. Each subsystem is composed of small focused modules.

Every node runs all 14. The engine is the network — not a separate service it talks to.

vs. Exa · vs. Parallel

Architecture guarantees

No central server

Every layer is P2P. Crawl coordination via CRDT frontier + consistent-hash leasing. SWC trees distributed via DHT + GossipSub. Index sharded with 3× replication. PageRank via BSP supersteps. Query cascade fans out across peers. Intelligence loop via gradient gossip + FedAvg. If the gateway disappears, the network keeps working.

No external APIs in the search path

Embeddings are computed locally (all-MiniLM-L6-v2 on CPU). LLM compression and answer synthesis route through the local model or a peer's pod via @hyperspace/inference. Web crawling is direct HTTP. There is no Google, no DuckDuckGo, no Exa, no OpenAI in the cascade.

Failure modes

Changelog

FAQ

Why is the search query free? What's the catch?

Every node earns Hyperspace points by serving operations to peers (crawl, index, search-serve, SWC compression, BSP PageRank, DPO gradient contributions). Your queries spend points; your serving earns them. The marginal dollar cost is zero because the underlying compute is paid in network credits the network mints to its participants. There is no centralized billing line because there is no centralized provider.

How is privacy preserved if queries fan out to peers?

The cascade resolves locally first — queries that hit your L1 / L2 cache (most queries, by Zipf) never leave your machine. When fan-out is needed, the SearchPeerSelector picks peers by capability, not by content, and the query travels with libp2p Noise encryption. There is no central log; the only persistent record is the click signal you opt into via the DPO loop, which is differentially-privatized before gossip.

What if my node has no GPU?

Most operations are CPU-bound. Embedding (all-MiniLM-L6-v2) runs on CPU, ~5ms per embedding. BM25 indexing is CPU. SWC compression and answer synthesis route through @hyperspace/inference — if you have no local GPU, calls go to a peer with one (you pay points; they earn points). Your node still serves crawl, index, and search.

Is the rank robust against link-farm attacks?

Yes — that's what the 6 anti-sybil files in @hyperspace/distributed-rank are for. SybilDetector runs 7 layers (graph density, temporal, stake, behavioral, IP, contribution, PoI). CollusionDetector spots mutual-boost rings, ring topologies, star topologies, temporal sync. EigenTrust propagates trust from stake-verified seeds. PenaltyEnforcer applies graduated punishments. All this runs on every BSP superstep, with hourly Merkle commits an auditor can verify against.

How do I run BrowseComp / SimpleQA against a Hyperspace node?

The benchmark harnesses are framework-agnostic — point them at http://localhost:8080 via either compat shim or the search-sdk directly. Numbers will be lower than Parallel's published 48% on BrowseComp at small network sizes; the federated-DPO loop closes that gap as your network grows. Published benchmark runs are on the roadmap.

What's the relationship to the avm-site / pods-site?

Hyperspace ships several P2P primitives. Pods (pods.hyper.space) is distributed inference clusters. AVM (avm.hyper.space) is the agent verification machine that gates risky actions. HyperSearch is the search engine. They share the same bootstrap network, points economy, libp2p protocol stack, and CLI — one binary serves them all.

Get started

Install Hyperspace, start with search capabilities, point your existing Exa / Parallel agent code at it. Two-line migration; instant cost-savings audit per request.