Intro

As a network programmer, you probably aren’t surprised to learn that I love networking. I think it’s one of the most interesting areas you can work on in the games industry. 

Recently I have started a project which requires a multiplayer component and the network library I wrote a few years ago is becoming quite messy. As such I am going to write a new library for this project.

I have decided that it would be quite fun and possibly interesting for others if I write this library and as I do so, I make posts on this blog. I hope that this will allow people who are trying to learn about network programming to follow development of a network library at their own pace and that anyone with a passing interest can pick up a few things on the way. I also hope that as the library is built up, people can take the code in a form closest to what they need instead of having to take a full network library which is very complicated, or a very simple wrapper which will require rolling a lot of their own functionality.

I don’t have any definitive plan as to exactly what will be included or the order in which I will do things, but these are things I definitely will be including in the initial stages:

  • Using Sockets
  • Virtual connections
  • Structuring and Packing/Serializing Packets
  • Packet Handling
  • Connecting through NATs
  • Session management
  • Threading

The library will be written for Windows using VC10 and Winsock, but it shouldn’t be very complicated to get it to run on other platforms.

If you are still reading and still interested, I hope you find it useful.
I’m very interested in hearing feedback from other people, so please don’t be shy.

I won’t be getting into the gritty code in this post, but I’ll be establishing some background and also explaining some decisions I have made and why. If you have an understanding of network programming, this first post is likely to cover things you already know, this is aimed at people with very little experience. Let’s dive right in shall we?

TCP/IP

What is TCP/IP?

Despite its name, the TCP/IP model actually describes a framework for computer network protocols and does not necessarily refer to either TCP or IP although they are included.

Most standard documents referring to TCP/IP do not have strict layering in the way that the Open Systems Interconnection (OSI) Model does. However, TCP/IP is typically described as having four layers in the form of a protocol stack (hence why people refer to it as the TCP/IP stack). From top to bottom these are: the Application Layer, the Transport Layer, the Internet Layer and the Link Layer. Layers closer to the top are closer to the user application and those at the bottom to the physical transmission of the data. Conceptually each layer builds on top of the layer below it meaning that a protocol on the Application Layer uses the layers below it to deliver a message.

What is a Protocol?

In its simplest form a Protocol is basically a well-defined format which describes a message so that it can be formed, sent somewhere else and then understood once it reaches its destination. It describes the language in which computers and applications talk. There is a bit more to it, but we’ll cover that later.

The Application Layer

The application layer refers to the protocol used in process-to-process communications. For us this means the protocol that we create for our game to communicate. This can be absolutely anything we like. Some examples of application layer protocols are the Hypertext Transfer Protocol (HTTP) which is the protocol of the World Wide Web and the Simple Mail Transfer Protocol (SMTP) which is the protocol typically used to send emails.

The Transport Layer

The transport layer refers to the protocol which offers end-to-end communication for a given application. There are numerous transport layers available, some are quite well known and you may have have heard of them, such as the Transmission Control Protocol (TCP) and the User Datagram Protocol (UDP). There are also some lesser known protocols such as the Stream Control Transmission Protocol (SCTP).

We’ll be coming back to TCP and UDP shortly.

The Internet Layer

The Internet Layer, as the name suggests, concerns the protocol which is used to transfer messages across network boundaries (inter-network). There are several internet layer protocols but they are outside the scope of what we need to look at. The only one which concerns us is Internet Protocol (IP) and for the purposes of this guide we’ll be using Internet Protocol version 4 (IPv4).

The Link Layer

The Link Layer is the protocol used on the logical (often physical) link between nodes. An example of a link layer protocol is the Point-to-point protocol (PPP). Any more information about the Link Layer is well outside the scope of this post. I encourage you to read up about it if you’re interested in how things such as Ethernet work, but from a games perspective, you will most likely never need to worry about it.

IP Addresses and Port Numbers

An IP address is a unique ID assigned to a device in a network. In our case, an IP address will almost always represent another device on the Internet or a Local Area Network (LAN). 

A port number is an ID representing a port an application is using.

If you imagine that the internet is like a phone network, an IP Address represents the phone number for a business (the device) and a port represents the extension of the person you are trying to reach (the application).
This is an extremely simple explanation and is perhaps not a technically accurate description, but I don’t want to confuse the issue at this stage. 

What is Latency?

Latency is the time it takes a message to get from one node to another. Round-trip latency is the time it takes for a message to get from one node to another and then back again.

Sockets

What is a Socket?

A Socket typically represents the endpoint of a Transport Layer protocol, this is not always the case but in the interests of understanding, it’s a good way to think about it. A local socket and a remote socket set up to communicate with one another are typically called a Socket Pair. 

For our purposes, a socket is represented by an API which we can use in our library. Multiple APIs exist for interacting with sockets, as we’re targeting Windows we’ll be using Winsock, which is based on Berkeley Sockets.

There are multiple types of sockets, but at the moment we’re only interested in TCP and UDP sockets.

So which type of socket should we be using?

TCP

A TCP socket is a connection based socket, you open a connection between two sockets. Once you have established a connection it acts like a stream, TCP will handle splitting everything up for you and provides flow control so you don’t send too much data at once. TCP also offers guaranteed and ordered delivery, this means that your data is always guaranteed to get to the remote socket and it’s always guaranteed to get there in the right order.

One device is running an application which has a socket listening on a specific port, an application on a remote device uses a socket to connect with the IP address and port number. Once the request is received, the listening application accepts the connection which is then typically moved to another free port (if this were not to happen when another device tried to connect it would fail, like a phone line being engaged).

These properties make TCP extremely easy to use as it handles quite a lot for you. 

UDP

When compared to TCP, UDP is an extremely simple protocol. Instead of adding the complexity you see in TCP, it is just a very simple wrapper for IP. There is no concept of a connection, we have to do this ourselves. It’s also not a stream, so we have to form messages, called datagrams or packets, ourselves.  

Packets are not guaranteed to arrive at all, and they are not guaranteed to arrive in order. They can even arrive more than once! Typically most packets will arrive in order, but it’s not safe to rely on this and it’s not unusual to see persistent packet loss of a single digit percent. UDP does offer a guarantee that a packet will either arrive in full or not at all (you can never get partial packets).

As there is no concept of a connection, an application simply creates a socket listening for packets on a specific port. An application on another device then uses a socket to send a packet to that port. The listening machine gets the message no matter who sent it, so we have to handle identifying people ourselves.

TCP or UDP?

Based on everything I have just said, we should use TCP right? It’s does everything we could possibly want and it’s easier to use than UDP.

Well…. No, and the reason is to do with the way TCP is implemented. When you stream data into a TCP socket, underneath it will be split into packets (it still has to travel over IP). Basically, when a packet is sent TCP detects whether that packet is lost and resends it.

Because TCP offers guaranteed order and delivery, you can end up in a situation where the remote machine is waiting for a packet, even though it has later data. If we send packets 1, 2 and 3 at the same time. It’s perfectly possible that packet 2 will be dropped along the way and because of this, packet 3 will arrive first. You can’t access packet 3 until packet 2 has been resent and received due to the ordered guarantee. If you consider a connection with 100ms round-trip latency, you are going to be waiting at least 100ms before the resend of packet 2 gets through, that’s a lot of time.

Even though typically UDP is more difficult to work with, it avoids the problems of TCP by allowing us to implement the behaviour we want.We can potentially decide different behaviour per type of packet or implement virtual connections and have it act exactly like TCP in the situations we want, we can even have different reliability queues for different types of packets.

For these reasons, I have decided to use UDP.

Wait a minute, I heard that World of Warcraft uses TCP?

Yes, it does. In fact, most MMOs use TCP as do some other games. Why? Well, the question of whether you should use UDP or TCP depends on the behaviour you want, MMOs typically do not have the same real-time constraints that a game like Halo does (which uses UDP). If it is not essential that you get the latest data straight away, TCP is perfectly suitable and as I discussed earlier, it’s much easier to work with.

There is absolutely no reason that World of Warcraft couldn’t use UDP, it’s possible to mimic TCP using UDP. However, it would not be suitable to implement Halo with TCP. As such, a library using UDP is likely to work with a much wider range of games than a library using TCP.

Ok then, but why can’t you just use both?

Ah, well, that’s a good question. We could use both TCP and UDP, we could use TCP for data we want to be guaranteed and use UDP where we’re only interested in the latest data. If you’re smart (which I am sure you are) you might even now be thinking “Couldn’t we use multiple TCP sockets so only packets of a similar type are queued?”. Well, you could but what you have to understand is that as they are both using IP, the packets can actually affect one another. As it happens, TCP tends to cause packet loss for UDP, and that’s bad. The reasons for this are quite complicated and unless you want to do some further research on the subject (papers do exist on it, so a bit of Googling will help) you’ll have to take my word for it.

Conclusions

Ok, so for my library I’m going to be using UDP and only UDP. My general advice is to use UDP. If you are absolutely sure that TCP meets your requirements, use it. Be sure to choose one, don’t mix and match.

I realize that for anyone who already has an understanding of networking, that was probably quite dull. I also hope that it was understandable for those without any experience. In my next post I’ll be writing a wrapper around Winsock and showing you how to send messages.

Further Reading