1 |
The Overnet Protocol |
2 |
-------------------- |
3 |
|
4 |
The Overnet protocol is divided in two parts: localisation and download. |
5 |
Localisation using UDP messages, while download is done using TCP/IP. The |
6 |
download protocol is mainly the same as in edonkey, except that the |
7 |
Connect/ConnectReply messages miss the server:port fields (ie they are 6 |
8 |
bytes shorter). |
9 |
|
10 |
In this draft, I focus on the localisation part. |
11 |
|
12 |
I Messages Format |
13 |
================= |
14 |
|
15 |
1) General Format: |
16 |
------------------ |
17 |
|
18 |
UDP messages start with two bytes: |
19 |
byte: 227 eDonkey general magic number for messages |
20 |
byte: the opcode of the message |
21 |
char[]: message args |
22 |
|
23 |
2) Opcodes: |
24 |
----------- |
25 |
|
26 |
10 OvernetConnect |
27 |
11 OvernetConnectReply |
28 |
12 OvernetPublicize |
29 |
13 OvernetPublicized |
30 |
14 OvernetSearch |
31 |
15 OvernetSearchReply |
32 |
16 OvernetGetSearchResults |
33 |
17 OvernetSearchResult |
34 |
18 OvernetNoResult |
35 |
19 OvernetPublish |
36 |
20 OvernetPublished |
37 |
|
38 |
24 OvernetFirewallConnection |
39 |
25 OvernetFirewallConnectionACK |
40 |
26 OvernetFirewallConnectionNACK |
41 |
27 OvernetGetMyIP |
42 |
28 OvernetGetMyIPResult |
43 |
29 OvernetGetMyIPDone |
44 |
|
45 |
3) Conventions: |
46 |
--------------- |
47 |
|
48 |
int8: one byte |
49 |
int16: two bytes in little endian format |
50 |
int32: four bytes in little endian format |
51 |
ip: an int32, for eg 127.0.0.1 = 127 + 0 << 8 + 0 << 16 + 1 << 24 |
52 |
md4: the 16 bytes md4 of the file/client |
53 |
|
54 |
peer: an overnet peer in the following format: |
55 |
md4: the peer md4 |
56 |
ip: the peer ip |
57 |
int16: the peer port |
58 |
int8: the peer kind/uptime/last seen ? |
59 |
|
60 |
tags: |
61 |
int32: <n> number of tags |
62 |
tag[n]: the array of tag (as in edonkey proto) |
63 |
|
64 |
4) Messages args: |
65 |
----------------- |
66 |
|
67 |
OvernetConnect: |
68 |
--------------- |
69 |
md4: client md4 |
70 |
ip: my client ip |
71 |
int16: my client port |
72 |
int8: my client kind/uptime ? 0 is ok |
73 |
|
74 |
OvernetConnectReply: |
75 |
-------------------- |
76 |
int16: <n> number of peers propagated in this message |
77 |
peer[n]: the peers propagated |
78 |
|
79 |
OvernetPublicize: |
80 |
----------------- |
81 |
md4: |
82 |
ip: ip |
83 |
int16: port |
84 |
int8: kind |
85 |
|
86 |
OvernetPublicized: no args |
87 |
------------------ |
88 |
|
89 |
OvernetSearch: |
90 |
-------------- |
91 |
int8: kind of search ? |
92 |
md4: the hash you are looking for |
93 |
|
94 |
OvernetSearchReply: |
95 |
------------------- |
96 |
md4: the hash you have asked for |
97 |
int8: <n> number of peers that could be interesting for you |
98 |
peer[n]: the better peers you should ask |
99 |
|
100 |
OvernetGetSearchResults: |
101 |
------------------------ |
102 |
md4: the hash you are looking for |
103 |
kind: ? |
104 |
int16: min (the range of results you want) |
105 |
int16: max |
106 |
|
107 |
OvernetSearchResult: |
108 |
-------------------- |
109 |
md4: the hash you have asked for |
110 |
md4: the hash of the client/file of the result (depends if you look for |
111 |
a file description or for a localisation) |
112 |
tags: the tags describing the result |
113 |
|
114 |
OvernetNoResult: |
115 |
---------------- |
116 |
md4: the hash you have asked for |
117 |
|
118 |
OvernetPublish: |
119 |
--------------- |
120 |
md4: |
121 |
md4: |
122 |
tags: |
123 |
|
124 |
OvernetPublished: |
125 |
----------------- |
126 |
md4: the hash of the published file |
127 |
|
128 |
OvernetGetMyIP: |
129 |
--------------- |
130 |
int8: 54 |
131 |
int8: 18 |
132 |
|
133 |
OvernetGetMyIPResult: |
134 |
--------------------- |
135 |
ip: your ip |
136 |
|
137 |
OvernetGetMyIPDone: no args |
138 |
------------------- |
139 |
|
140 |
OvernetFirewallConnection: |
141 |
-------------------------- |
142 |
md4: the md4 of the client you want to access |
143 |
int16: ? |
144 |
|
145 |
OvernetFirewallConnectionACK: |
146 |
----------------------------- |
147 |
md4: |
148 |
|
149 |
OvernetFirewallConnectionNACK: |
150 |
md4: |
151 |
|
152 |
II Client behavior: |
153 |
=================== |
154 |
|
155 |
1) Basics: |
156 |
---------- |
157 |
|
158 |
In Overnet, the description and localisation of files are stored on peers |
159 |
such that the peer hash is as close as possible to the file hash. |
160 |
|
161 |
When a peer wants a file, it tries to ask the peers which have the |
162 |
closest hashs to the file it wants. These peers reply by sending the |
163 |
closest peers they know to the requested hash. |
164 |
|
165 |
The distance between two hashs is computed using what Kademlia calls a |
166 |
Xor metric: a xor is computed between the two hashs, and the result |
167 |
hash represents the distance. Two distances are then compared by using |
168 |
a lexicographical order on their bytes (ie first bytes are compared, and |
169 |
if equal, the second bytes, and so on...) |
170 |
|
171 |
For this to work, the client needs to maintain a small list of peers it |
172 |
knows, which are as spreaded as possible in the space of possible hashs. |
173 |
|
174 |
2) Sequences: |
175 |
------------- |
176 |
|
177 |
a) Establishing a "connection" with other peers (booting) |
178 |
--------------------------------------------------------- |
179 |
|
180 |
Client Peer |
181 |
------------------------------------------------ |
182 |
|
183 |
1: ------- OvernetConnect ------> |
184 |
|
185 |
2: <----- OvernetConnectReply ---- |
186 |
|
187 |
OK, the peer is connected, it can be added to the list of available peers |
188 |
for searches. The peers propagated in the reply can be used for new |
189 |
connections. |
190 |
|
191 |
b) Searches |
192 |
----------- |
193 |
From the keywords search, remove all keywords of 1 or 2 chars. |
194 |
Then, hash each keyword separately, and start one search per hash. When you |
195 |
will get results, you will have to filter them so that all keywords are |
196 |
matched, the protocol won't do that for you. |
197 |
|
198 |
The search works as follows. You need a set of peers to be searched for the |
199 |
hash. Typically, you take the peers with the hashes with the same first char |
200 |
as the one you search. |
201 |
|
202 |
Then, periodically (every 5 seconds ?), you take some peers (2 ? with the |
203 |
closest hashes), and ask them for the hash: |
204 |
|
205 |
Client Peer |
206 |
------------------------------------------------ |
207 |
|
208 |
1: --------- OvernetSearch -------> |
209 |
1': ------- OvernetGetSearchResults --> |
210 |
|
211 |
2: <----- OvernetSearchReply ------- |
212 |
2': <----- OvernetSearchResult ------ |
213 |
2'': <----- OvernetNoResult ------- |
214 |
|
215 |
1: query for better peers to ask for the hash |
216 |
2: return new peers which should be queried for the hash |
217 |
|
218 |
1': query the peer directly for the hash: do you know it ? |
219 |
2': yes, I know it, here are the files containing this keyword |
220 |
(or the clients sharing this file if you are looking for a file-hash) |
221 |
2'': no, I have no info, try someone else... |
222 |
|
223 |
Now, when you have the results, you need to remember what you are |
224 |
searching for: |
225 |
1) keywords: results are file descriptions, closed to edonkey ones |
226 |
(here, maybe some tags are different :))) |
227 |
2) files: you should look in the tags for one called "loc". It contains |
228 |
a string with the format "bcp://ip:port" or "bcp://????:ip:port" |
229 |
containing the source ip and port in ascii. |
230 |
|
231 |
Now, you just need to connect and download ... |
232 |
|
233 |
|
234 |
|
235 |
|