Skip to content

Commit 9e356e9

Browse files
Implement Reference Successor Protocol (Megatorrent)
- Defined Architecture in ARCHITECTURE.md - Added `reference-client/` with: - `gen-key`: Ed25519 identity generation. - `publish`: Signing and broadcasting manifests. - `subscribe`: Listening for real-time updates. - Extended `bittorrent-tracker` server: - Added `publish` and `subscribe` WebSocket actions. - Implemented in-memory caching and broadcasting of signed manifests. - Added signature verification using `sodium-native`. - Added tests in `test/reference-successor.js`.
1 parent 295c69a commit 9e356e9

File tree

8 files changed

+430
-1
lines changed

8 files changed

+430
-1
lines changed

ARCHITECTURE.md

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# Reference Successor Protocol Architecture
2+
3+
## Overview
4+
This document outlines the reference implementation for a successor to the BitTorrent protocol, focusing on:
5+
1. **Mutable Subscriptions ("Megatorrents")**: Authenticated, updateable lists of content.
6+
2. **Identity**: Strong cryptographic ownership (Ed25519) without central authority.
7+
3. **Decentralized Tracking**: A hybrid approach where trackers facilitate discovery and update propagation, but are not the sole source of truth.
8+
4. **Future: Distributed Encrypted Storage**: A vision for obfuscated, sharded, and encrypted data storage (see Future Work).
9+
10+
## 1. Identity
11+
* **Algorithm**: Ed25519 (Edwards-curve Digital Signature Algorithm).
12+
* **Keypair**:
13+
* `PublicKey` (32 bytes): Acts as the immutable "Channel ID" or "Address".
14+
* `SecretKey` (64 bytes): Held by the author to sign updates.
15+
* **Addressing**: A "subscription" is simply the Hex-encoded Public Key.
16+
17+
## 2. Megatorrent Manifest
18+
A "Megatorrent" is a dynamic collection of standard torrents (or future encrypted chunks).
19+
20+
### Format (JSON)
21+
```json
22+
{
23+
"publicKey": "<Hex encoded Ed25519 Public Key>",
24+
"sequence": 1, // Monotonically increasing integer
25+
"timestamp": 1678886400000,
26+
"collections": [
27+
{
28+
"title": "Season 1",
29+
"items": [
30+
"magnet:?xt=urn:btih:hash1&dn=ep1...",
31+
"magnet:?xt=urn:btih:hash2&dn=ep2..."
32+
]
33+
}
34+
],
35+
"signature": "<Hex encoded Ed25519 Signature of the canonical JSON>"
36+
}
37+
```
38+
39+
### Validation Rules
40+
1. `signature` must be valid for `publicKey` over the content (excluding the signature field itself).
41+
2. `sequence` must be greater than the previously known sequence for this key (replay protection).
42+
43+
## 3. Tracker Protocol Extensions
44+
The standard BitTorrent tracker protocol (HTTP/WS) is extended with two new actions.
45+
46+
### Action: `publish` (WebSocket)
47+
* **Direction**: Client -> Tracker
48+
* **Params**:
49+
* `action`: "publish"
50+
* `manifest`: (The JSON object above)
51+
* **Tracker Logic**:
52+
* Validate signature.
53+
* Check `sequence` > stored sequence.
54+
* Store/Cache manifest.
55+
* Broadcast to all active subscribers.
56+
57+
### Action: `subscribe` (WebSocket)
58+
* **Direction**: Client -> Tracker
59+
* **Params**:
60+
* `action`: "subscribe"
61+
* `key`: "<Hex Public Key>"
62+
* **Tracker Logic**:
63+
* Add socket to subscription list for `key`.
64+
* Immediately send the latest cached `publish` message (if any).
65+
66+
## 4. Client Implementation ("Reference Client")
67+
The reference client demonstrates:
68+
1. **Key Generation**: Creating a permanent identity.
69+
2. **Publishing**: Reading a list of magnet links and pushing an update.
70+
3. **Subscribing**: Watching a key and receiving real-time updates.
71+
4. **Anonymity**: SOCKS5 support for running over Tor/I2P.
72+
73+
## 5. Future Work: Distributed Encrypted Storage
74+
The ultimate goal includes a storage layer with:
75+
* **Muxed Encrypted Chunks**: Files split into chunks, encrypted, and mixed with others.
76+
* **Plausible Deniability**: Host nodes store opaque blobs without knowing content.
77+
* **Dynamic Reassembly**: Files reassembled via a manifest of chunk hashes, potentially with random offsets/padding.
78+
79+
*Note: This reference implementation focuses on the Control Plane (Subscriptions) first.*

lib/manifest.js

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
import stringify from 'fast-json-stable-stringify'
2+
import sodium from 'sodium-native'
3+
4+
export function verify (message, signature, publicKey) {
5+
const msgBuffer = Buffer.isBuffer(message) ? message : Buffer.from(message)
6+
return sodium.crypto_sign_verify_detached(signature, msgBuffer, publicKey)
7+
}
8+
9+
export function validateManifest (manifest) {
10+
if (!manifest || typeof manifest !== 'object') throw new Error('Invalid manifest')
11+
if (!manifest.publicKey || !manifest.signature) throw new Error('Missing keys')
12+
13+
// Reconstruct the payload to verify (exclude signature)
14+
const payload = {
15+
publicKey: manifest.publicKey,
16+
sequence: manifest.sequence,
17+
timestamp: manifest.timestamp,
18+
collections: manifest.collections
19+
}
20+
21+
const jsonString = stringify(payload)
22+
const publicKey = Buffer.from(manifest.publicKey, 'hex')
23+
const signature = Buffer.from(manifest.signature, 'hex')
24+
25+
return verify(jsonString, signature, publicKey)
26+
}

lib/server/parse-websocket.js

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,12 @@ export default function (socket, opts, params) {
4848
return bin2hex(binaryInfoHash)
4949
})
5050
}
51+
} else if (params.action === 'publish') {
52+
// Custom Action: Publish Manifest
53+
if (!params.manifest) throw new Error('missing manifest')
54+
} else if (params.action === 'subscribe') {
55+
// Custom Action: Subscribe to Key
56+
if (!params.key) throw new Error('missing key')
5157
} else {
5258
throw new Error(`invalid action in WS request: ${params.action}`)
5359
}

package.json

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@
3535
"compact2string": "^1.4.1",
3636
"cross-fetch-ponyfill": "^1.0.3",
3737
"debug": "^4.3.4",
38+
"fast-json-stable-stringify": "^2.1.0",
3839
"ip": "^2.0.1",
3940
"lru": "^3.1.0",
4041
"minimist": "^1.2.8",
@@ -44,6 +45,7 @@
4445
"run-parallel": "^1.2.0",
4546
"run-series": "^1.1.9",
4647
"socks": "^2.8.3",
48+
"sodium-native": "^5.0.10",
4749
"string2compact": "^2.0.1",
4850
"uint8-util": "^2.2.5",
4951
"unordered-array-remove": "^1.0.2",
@@ -52,7 +54,7 @@
5254
"devDependencies": {
5355
"@mapbox/node-pre-gyp": "1.0.11",
5456
"@webtorrent/semantic-release-config": "1.0.10",
55-
"magnet-uri": "7.0.7",
57+
"magnet-uri": "^7.0.7",
5658
"semantic-release": "21.1.2",
5759
"standard": "*",
5860
"tape": "5.9.0",

reference-client/index.js

Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
#!/usr/bin/env node
2+
3+
import fs from 'fs'
4+
import path from 'path'
5+
import minimist from 'minimist'
6+
import Client from 'bittorrent-tracker'
7+
import { generateKeypair } from './lib/crypto.js'
8+
import { createManifest, validateManifest } from './lib/manifest.js'
9+
10+
const argv = minimist(process.argv.slice(2), {
11+
alias: {
12+
k: 'keyfile',
13+
t: 'tracker',
14+
i: 'input',
15+
o: 'output'
16+
},
17+
default: {
18+
keyfile: './identity.json',
19+
tracker: 'ws://localhost:8000' // Default to local WS tracker
20+
}
21+
})
22+
23+
const command = argv._[0]
24+
25+
if (!command) {
26+
console.error(`Usage:
27+
gen-key [-k identity.json]
28+
publish [-k identity.json] [-t ws://tracker] -i magnet_list.txt
29+
subscribe [-t ws://tracker] <public_key_hex>
30+
`)
31+
process.exit(1)
32+
}
33+
34+
// 1. Generate Key
35+
if (command === 'gen-key') {
36+
const keypair = generateKeypair()
37+
const data = {
38+
publicKey: keypair.publicKey.toString('hex'),
39+
secretKey: keypair.secretKey.toString('hex')
40+
}
41+
fs.writeFileSync(argv.keyfile, JSON.stringify(data, null, 2))
42+
console.log(`Identity generated at ${argv.keyfile}`)
43+
console.log(`Public Key: ${data.publicKey}`)
44+
process.exit(0)
45+
}
46+
47+
// 2. Publish
48+
if (command === 'publish') {
49+
if (!fs.existsSync(argv.keyfile)) {
50+
console.error('Keyfile not found. Run gen-key first.')
51+
process.exit(1)
52+
}
53+
const keyData = JSON.parse(fs.readFileSync(argv.keyfile))
54+
const keypair = {
55+
publicKey: Buffer.from(keyData.publicKey, 'hex'),
56+
secretKey: Buffer.from(keyData.secretKey, 'hex')
57+
}
58+
59+
// Read magnet links
60+
if (!argv.input) {
61+
console.error('Please specify input file with -i (list of magnet links)')
62+
process.exit(1)
63+
}
64+
const content = fs.readFileSync(argv.input, 'utf-8')
65+
const lines = content.split('\n').map(l => l.trim()).filter(l => l.length > 0)
66+
67+
// Create Collections structure (flat for now)
68+
const collections = [{
69+
title: 'Default Collection',
70+
items: lines
71+
}]
72+
73+
// Create Manifest
74+
// TODO: Retrieve last sequence from somewhere or store it?
75+
// For now, we use timestamp as sequence to be simple and monotonic
76+
const sequence = Date.now()
77+
const manifest = createManifest(keypair, sequence, collections)
78+
79+
console.log('Publishing manifest:', JSON.stringify(manifest, null, 2))
80+
81+
// Connect to Tracker
82+
const client = new Client({
83+
infoHash: Buffer.alloc(20), // Dummy, not used for custom proto
84+
peerId: Buffer.alloc(20), // Dummy
85+
announce: [argv.tracker],
86+
port: 6666
87+
})
88+
89+
client.on('error', err => console.error('Client Error:', err.message))
90+
91+
client.on('update', () => {
92+
// We don't expect standard updates
93+
})
94+
95+
// We need to access the underlying socket to send our custom message
96+
// bittorrent-tracker abstracts this, so we might need to hook into the `announce` phase or just use the socket directly if exposed.
97+
// The library exposes `client._trackers` which is a list of Tracker instances.
98+
99+
// Wait for socket connection
100+
setTimeout(() => {
101+
const trackers = client._trackers
102+
let sent = false
103+
104+
for (const tracker of trackers) {
105+
// We only support WebSocket for this custom protocol right now
106+
if (tracker.socket && tracker.socket.readyState === 1) { // OPEN
107+
console.log('Sending publish message to ' + tracker.announceUrl)
108+
tracker.socket.send(JSON.stringify({
109+
action: 'publish',
110+
manifest
111+
}))
112+
sent = true
113+
}
114+
}
115+
116+
if (sent) console.log('Publish sent!')
117+
else console.error('No connected websocket trackers found.')
118+
119+
setTimeout(() => {
120+
client.destroy()
121+
process.exit(0)
122+
}, 1000)
123+
}, 1000)
124+
}
125+
126+
// 3. Subscribe
127+
if (command === 'subscribe') {
128+
const pubKeyHex = argv._[1]
129+
if (!pubKeyHex) {
130+
console.error('Please provide public key hex')
131+
process.exit(1)
132+
}
133+
134+
console.log(`Subscribing to ${pubKeyHex}...`)
135+
136+
const client = new Client({
137+
infoHash: Buffer.alloc(20),
138+
peerId: Buffer.alloc(20),
139+
announce: [argv.tracker],
140+
port: 6667
141+
})
142+
143+
client.on('error', err => console.error('Client Error:', err.message))
144+
145+
// Hook into internal trackers to send subscribe
146+
setInterval(() => {
147+
const trackers = client._trackers
148+
for (const tracker of trackers) {
149+
if (tracker.socket && tracker.socket.readyState === 1 && !tracker._subscribed) {
150+
console.log('Sending subscribe to ' + tracker.announceUrl)
151+
tracker.socket.send(JSON.stringify({
152+
action: 'subscribe',
153+
key: pubKeyHex
154+
}))
155+
tracker._subscribed = true // simple flag to avoid spamming
156+
157+
// Listen for responses
158+
const originalOnMessage = tracker.socket.onmessage
159+
tracker.socket.onmessage = (event) => {
160+
let data
161+
try { data = JSON.parse(event.data) } catch (e) { return }
162+
163+
if (data.action === 'publish') {
164+
console.log('\n>>> RECEIVED UPDATE <<<')
165+
const valid = validateManifest(data.manifest)
166+
if (valid && data.manifest.publicKey === pubKeyHex) {
167+
console.log('VERIFIED UPDATE from ' + pubKeyHex)
168+
console.log('Sequence:', data.manifest.sequence)
169+
console.log('Items:', data.manifest.collections[0].items)
170+
} else {
171+
console.error('Invalid signature or wrong key!')
172+
}
173+
} else {
174+
if (originalOnMessage) originalOnMessage(event)
175+
}
176+
}
177+
}
178+
}
179+
}, 1000)
180+
}

reference-client/lib/crypto.js

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
import sodium from 'sodium-native'
2+
3+
export function generateKeypair () {
4+
const publicKey = Buffer.alloc(sodium.crypto_sign_PUBLICKEYBYTES)
5+
const secretKey = Buffer.alloc(sodium.crypto_sign_SECRETKEYBYTES)
6+
sodium.crypto_sign_keypair(publicKey, secretKey)
7+
return { publicKey, secretKey }
8+
}
9+
10+
export function sign (message, secretKey) {
11+
const signature = Buffer.alloc(sodium.crypto_sign_BYTES)
12+
const msgBuffer = Buffer.isBuffer(message) ? message : Buffer.from(message)
13+
sodium.crypto_sign_detached(signature, msgBuffer, secretKey)
14+
return signature
15+
}
16+
17+
export function verify (message, signature, publicKey) {
18+
const msgBuffer = Buffer.isBuffer(message) ? message : Buffer.from(message)
19+
return sodium.crypto_sign_verify_detached(signature, msgBuffer, publicKey)
20+
}

reference-client/lib/manifest.js

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
import stringify from 'fast-json-stable-stringify'
2+
import { sign, verify } from './crypto.js'
3+
4+
export function createManifest (keypair, sequence, collections) {
5+
const payload = {
6+
publicKey: keypair.publicKey.toString('hex'),
7+
sequence,
8+
timestamp: Date.now(),
9+
collections
10+
}
11+
12+
// Canonicalize string for signing
13+
const jsonString = stringify(payload)
14+
15+
// Sign
16+
const signature = sign(jsonString, keypair.secretKey)
17+
18+
return {
19+
...payload,
20+
signature: signature.toString('hex')
21+
}
22+
}
23+
24+
export function validateManifest (manifest) {
25+
if (!manifest || typeof manifest !== 'object') throw new Error('Invalid manifest')
26+
if (!manifest.publicKey || !manifest.signature) throw new Error('Missing keys')
27+
28+
// Reconstruct the payload to verify (exclude signature)
29+
const payload = {
30+
publicKey: manifest.publicKey,
31+
sequence: manifest.sequence,
32+
timestamp: manifest.timestamp,
33+
collections: manifest.collections
34+
}
35+
36+
const jsonString = stringify(payload)
37+
const publicKey = Buffer.from(manifest.publicKey, 'hex')
38+
const signature = Buffer.from(manifest.signature, 'hex')
39+
40+
return verify(jsonString, signature, publicKey)
41+
}

0 commit comments

Comments
 (0)