This is going to be a fairly technical post about the technologies behind Machsend and using P2P over TCP rather than UDP. I’v written a lot of context to the problem, so if you want to skip to the juicy protocol details look for the set of bullet points.
We recently launched Machsend, which enables people to send unlimited amounts of data from inside the browser. You literally drag a file onto the site, send the recipient a link, and they can download it straight away. Machsend uses TCP P2P connections to transfer data between clients.
Most of the packets on the internet are controlled by TCP which provides a reliable, ordered delivery of a stream of bytes between two endpoints. The alternative is UDP which is much lower level than TCP – information isn’t guaranteed to reach its destination. Most P2P implementations use UDP to communicate (for reasons we’ll come to later), but I’m going to show you how to get the flow control and reliability of TCP in P2P connections.
Routers (combined with firewalls) are both critical to the stability of the internet, and an enemy to P2P connections. Networks can have lots of computers behind a single IP using a router – and the router will make sure the right packets get to the right computers, AKA NAT. However, I guess they just weren’t designed with P2P in mind – so most require a bit of coaxing/trickery to set up P2P connections.
With UDP it’s fairly straight forward – that’s why it’s so commonly used in P2P software.
Client A sends a packet to Client B, opening a hole in Client A’s firewall. That packet will be discarded by Client B’s firewall, but that doesn’t matter. Client B then does the reverse, sending a packet back Client A – now there’s a hole in both firewalls and a P2P connection. I’m glossing over some of the more technical details, such as STUN and port prediction, but that’s the general gist. Here’s a good article on how Skype uses UDP P2P techniques to traverse firewalls.
UDP is perfect for Skype, for example, since you don’t care that much about reliability when delivering audio & video – but rather responsiveness. The data is very time dependent and needs to get to its destination quickly. However, with most other data transfers we care that all the data reaches the destination properly. If you’re transferring a picture to someone, you want to be sure all of it reaches the recipient.
I have actually created a UDP protocol for data transfer that’s as reliable as TCP, yet is also faster. That, however, is the subject of another post.
While routers don’t meddle with UDP connections too much, that’s not the case for TCP ones. Routers will enforce TCP standards and prevent you from accepting random incoming TCP connections. On top of that, most operating systems require root access to create raw sockets, which you need to manipulate TCP streams at the packet level.
So, it’s a bit of a pickle – on hand you’ve got proper P2P connections with UDP, but you have to implement your own flow control and reliability algorithms. On the other hand, TCP P2P connections seem very tricky to setup and aren’t supported on many routers.
Most people presented with this problem, will either use UPnP, a protocol for programs to configure routers, or tell people to configure their routers manually.
Obviously, this is far from ideal, both approaches have major caveats, but unfortunately it’s going to be the status quo for a while.
I was in the aforementioned situation, until I found a paper titled Characterization and Measurement of TCP Traversal through NATs and Firewalls. The students tested about 7 methods of TCP P2P traversals, and advocated a fairly simple one that works with about 80% of the firewalls they tested.
To save you reading the paper, I’ll elaborate. The TCP protocol starts with a 3 way handshake of SYN and ACK packets. However, although it’s an unlikely combination, most routers will let through incoming SYN packets, if an outgoing SYN packet has been sent to exactly the same ip and port combination. So, to sum up:
- Client A sends a SYN packet to Client B
- Client A starts listening on exactly the same port the previous SYN packet was sent on
- Client B then creates a normal TCP connection to Client A, to exactly the same local port Client A used to send the SYN, binding locally to exactly the same port that Client A sent the SYN packet too.
And that’s all there is to TCP P2P – those three steps.
To send a SYN packet without using raw sockets you just open a socket to an address, and then immediately close it.
To open a socket on a particular local port, since it’s usually chosen for you, you just need to set some options. If you enable SO_REUSEADDR and SO_REUSEPORT the operating system shouldn’t complain.
The document for Ruby’s Socket class is fairly sparse, so here’s an abstraction over it:
To accept connections from Client B, Client A sends a SYN packet, and then listens:
Client B’s connection to Client A looks like this:
If the connection fails, try reversing Client A and B, so Client B makes the connection to Client A – this usually works.
So, you might be asking yourself – how does Client A know Client B’s IP & port, and vica versa? Who’s controlling it? Well, for that you need a third party server known to both endpoints – usually called a STUN server, or with TCP traversal, STUNT. For Machsend, we built a JSON RPC server using Event Machine .
At the moment we haven’t implemented any port prediction since I’ve noticed most of the routers keep the same ports that the computers choose. However, to improve reliability, that could certainly be implemented.
As well as the previously linked paper, there’s a good article on the subject here.
Enjoy this article?
Consider subscribing to our RSS feed!