The UDP situation
By erdgeist | January 2, 2008
Most of the problems we have to cope with in our everyday opentracker struggle arises from operating systems having to handle several thousand tcp connection setups, reassembly and tear downs. As a mean to help ease the impact of these requests, a udp extension to the tracker protocol has been agreed upon.
However that specification page does a rather poor job – besides of the obvious html layout deficiencies – when it comes to explaining what design decissions were made and why. This lead to many client writers out there wondering what the connect packet is about.
While there are sites around with a layout that at least does not try to prevent implementation, rationale for the extension completely is absent.
1. The connect packet
Now, what is the extra connect packet all about? Wasn’t the udp extension all about saving packets hammering the server? The answer is, it was designed to prevent address spoofing. While tcp inherently prevents address spoofing with its initial handshaking – you can only finish it if the recipient of SYN+ACK is the same address that sent the SYN – udp does not provide any means of ensuring that the packet really originated from the address it claims to be.
So Olaf van der Spek – designer of that protocol extension – demands that a client first uses a connect packet to acquire a uniq-ish connection_id that it later presents to the tracker to prove listening on that very address.
What way of chosing, storing and comparing connection_ids a tracker implements is up to him, one possible way would be chosing one very long global tracker secret, and then calculate a cryptographically secure checksum of this secret appended by the clients alleged ip address. This approach would not require the tracker to store anything per client for it can be calculated everytime that client comes back.
However, when writing the specification, Olaf made the mistake of not mentioning this feature and – even worse – using the term connection_id for two completely unrelated values: A connect packet contains a static protocol identifier at the same offset where on announce and scrape packets that unique connection_id is expected. Olaf even also calls that value connection_id. Hence half of the non-connect udp packets arriving at opentracker still contain 0×41727101980 as connection_id.
To be honest, opentracker is not completely innocent. Since we could freely chose which values to put there, we just left the old connection_id. This might have given the impression that the whole connection_id thing just is a sanity feature to distinguish stray udp packets arriving on that port.
Now we’re facing the problem that enforcing connection_id checking would really hurt udp announces. This is not in our interest. As a first step we changed our code to return a different value as connection_id. Now we want to encourage all client authors supporting udp to check their implementation against the way it was meant to be. In some months from now opentracker will enforce connection_id checking.
2. Auto UDP
As you may have guessed by now we do love udp announces a lot more than the tcp stuff. So introducing ways to increase the number of those more lightweight client interactions is in our own very interest.
We do know that it is hard to encourage users to add udp:// addresses to their torrents. Still most tracker responses on earth are being handled by trackers that are capable of speaking udp (and soon also udp-v6). This means that sending udp connect packets to the corresponding udp port of a http tracker url will most likely succeed.
We hereby invite all client authors to probe the corresponding udp announce address, store the result on a per-tracker base, use udp whenever feasible (i.e. for announces and [non-full-]scrapes) and revert to tcp only if udp fails.
3. IPv6
And while you are at it: you might also want to take a look at our v6 protocol extension proposal.
As always, feel free to comment on that topic at our coders mailing list.
Topics: coding, tech | 3 Comments »
The IPv6 situation
By erdgeist | December 28, 2007
If you haven’t spent the last decade behind a C64 you may have noticed a HugeNewThing™ called IPv6 is coming. There still are some issues that prevent its wide spread use. No consumer DSL has it. Most people wouldn’t even care what *IPv4* is. However, bittorrent has been designed to work with IPv6 out of the box. Many convenience extensions in the Bittorrent-Tracker protocol – compact tracker responses, for example – do not. Worse: others like the UDP-Tracker protocol do not even look like having taken IPv6 into account at all.
Still, we consider IPv6 very important and convenient. Its use could solve all NAT-dependent connectivity issues and heavily reduce complexity in clients. On the other hand we noticed that introducing IPv6 in a transitional way creates many headaches in inter-protocol communication. To remove some barriers we propose the following request for comments which will be incorporated into opentracker and thus available at tpb soon.
1. The clouds
To be frank: there will be two clouds for each torrent, one in IPv4 and one in IPv6. Clients capable of handling IPv6 connections and still interested in IPv4 peers will need to announce their torrents to trackers twice. This is a harsh decission, but in the end most IPv4 peers are not interested in IPv6 peers and there is no easy way to tell whom to return which peers.
The proposed way to identify a peer’s internet protocol version defaults to the incoming socket address and can be overridden by the ip parameter from the announce http request.
This is very much how Bram Cohen layed it out in his Tracker-Protocol. Of course, overriding only works for trackers that allow overriding ip addresses with the one passed in the query. All others have no means of obtaining the ip address for the other protocol version.
Scrapes will be answered for each cloud independently according to the requestors ip address protocol version.
2. Tracker TCP Responses
Most trackers (e.g. opentracker) have completely switched to only returning compact responses in order to save, or better: not to waste bandwidth. That compact format is just a string of multiple of six bytes that contain ip address and port for as many peers as do fit there. Filling it with or mixing in IPv6 addresses is impossible without breaking backward compatibility.
So we propose adding another “peers6″ entry in the announce response dictionary that contains a string of zero or more entries, each 16 bytes network byte order IPv6 addresses followed by 2 byte network byte order port number. The “peers6″ entry must not be present in non-compact mode.
3. Tracker UDP Responses
The UDP-protocol has not left any space for IPv6-addresses in its current form. So we need to define a new action together with its input and output block formats.
Proposed extension to the UDP Tracker format to work over IPv6, we claim a value for the action identifier of “4″.:
Request:
| Offset | Size | Name | Value |
| 0 | 64-bit integer | connection_id | |
| 8 | 32-bit integer | action | 4 |
| 12 | 32-bit integer | transaction_id | |
| 16 | 20-byte string | info_hash | |
| 36 | 20-byte string | peer_id | |
| 56 | 64-bit integer | downloaded | |
| 64 | 64-bit integer | left | |
| 72 | 64-bit integer | uploaded | |
| 80 | 32-bit integer | event | |
| 84 | 16-byte string | IP address | |
| 100 | 32-bit integer | key | |
| 104 | 32-bit integer | num_want | -1 |
| 108 | 16-bit integer | port |
The corresponding reply looks as follows:
Response:
| Offset | Size | Name | Value |
| 0 | 32-bit integer | action | 4 |
| 4 | 32-bit integer | transaction_id | |
| 8 | 32-bit integer | interval | |
| 12 | 32-bit integer | leechers | |
| 16 | 32-bit integer | seeders | |
| 20+n*18 | 16-byte string | IPv6 address | |
| 36+n*18 | 16-bit integer | TCP port |
We consider those the most sane design decissions that do not deviate from existing protocol too much and hope to receive feedback on our proposal, from closed source implementations as well as the usual open source suspects.
Topics: coding, tech | 8 Comments »
Going social
By erdgeist | December 28, 2007
This topic is only for our German readers, sorry ’bout that.
Wir sind nun auch audio-visuell in Erscheinung getreten und haben unser opentracker-Projekt und Bittorrent im Allgemeinen in einem Chaos Radio Express mit Tim und einem Talk über opentracker auf dem 24C3 vorgestellt.
Wenn es das Video gibt, reichen wir den Link nach.
Update: So jetzt gibts auch das Video, natürlich als torrent: 24c3-2355-de-trecker_fahrn.mkv oder 24c3-2355-de-trecker_fahrn.mp4
Topics: free speech | 8 Comments »
Review-A-Tracker-Team
By erdgeist | December 11, 2007
Six days ago – on December 5th – opentracker celebrated its first birthday, just two days before officially becoming the tracker handling most tracker requests on earth. During that year openracker already handled several billions http connections in what can only be described as the hardest fuzzing attack ever done to a software (:
However, becoming so huge and being so exposed brings its risks. opentracker is written in plain C and handles strings. Strings that have been sent through the internet. So we kindly ask the community to help us make the code more secure by reviewing critical parts of it. Especially the part that parses URIs is a natural point to start looking into.
So if you are experienced in C, serious about helping to review or just need some explanation on opentrackers rougher edges, feel free to contact us at code@denis.stalker.h3q.com. We do also appreciate patches that fix bugs and warnings on operating systems we have not tested opentracker on and source packages for certain package distribution systems.
Topics: coding, tech | 3 Comments »
On fake-torrents
By taklamakan | December 9, 2007
From a recent mail we received:
im sick of seeing your tracker host malicious files. im sick of seeing your
tracker on the top pages of indexing sites. you are a disgrace. seeing your
tracker on any torrent means to the non-nub BT community that it contains
malicious files. you allow your uploaders to prey on the ill-informed.
control your site, or take it down. you are the only ones who can stop
malicious files from finding their way into the community. indexing sites
dont yet have the ability to filter out torrents by trackers alone, so it is
YOUR responsibility to take control of YOUR tracker. you should be ashamed
of yourselves.
Again *sic*, for the people who don’t understand what a bittorrent tracker does (basically all the people blaming us of being fake):
A bittorrent-tracker collects hashes (a 160-bit digest, SHA1) of bittorrent-files, it starts collecting ipaddr. of peers who are interested who else is interested in the files identified by those hashes. Thats it.
Its like an agency for arranged lifts! You (peer) ask them (tracker) who (peer-list) is going from A to B (torrent) and would like to share the car and costs (files) of the drive (download). Or you tell them that you (peer) are going from A to B (torrent) and that you are interested in people (peer-list) who would like to join you on your drive (download) to share the costs (files).
The agency for arranged lifts does not know if the drive from A to B (torrent) is safe, if the roads are free or if the car (files) is in any condition for driving this distance or keeping the people inside dry in case of rain.
A bittorrent-site is something completely different. On such sites you can upload and download bittorrent-files, on most sites you can comment the bittorrent-file in a forum like system. Thats where the so called “bittorrent community” lives.
So if you download a bittorrent-file from a bittorrent-site and it is fake or malicious, tell the community, comment the torrent on the bittorrent-site, let the other people know! But please don’t blame us or other operators of bittorrent-trackers for tracking this fake-torrent! You know, it wastes our tracker resources too if morons keep on downloading fake-torrents! So maybe people should follow some simple advises on downloading torrents.
We are a bittorrent-tracker, not a bittorrent-site. The only ones who can stop fake-torrents are the users on bittorrent-sites, its in their responsibility! Don’t ask what bittorrent can do for you, ask what you can do for bittorrent!
Topics: free speech, tech | 14 Comments »
Opentracker running at ThePirateBay
By taklamakan | December 7, 2007
Today ThePirateBay switched the last tracker (tracker.prq.to) to opentracker. With the migration of open.tracker.thepiratebay.org to opentracker three weeks ago, now all their Bittorrent Tracker are running with our software (as you can see on their about page), Arrr!
Of course you can take a look at the new tracker (tracker.prq.to: trackerstats-tpb145 to tpb149, open.tracker.thepiratebay.org: trackerstats-tpb138 to tpb141) via the MRTG Graphs.
Topics: history, tech | 8 Comments »
« Previous Entries Next Entries »