How a page gets to you

Problem

Between Enter and the page

You type shop.example.bg in the browser, press Enter, and half a second later you see the coffee and tea catalog. The page lives on a computer in another city, maybe in another country. How does the browser find it? How does it know you want the catalog and not the cart? What exactly do the two programs say to each other?

In the next lesson you will write a server that answers the browser. To do that, you need to know the language they speak.

The idea

Packets, routers and IP addresses

The internet is a network of networks: your home router, your provider's network, the networks of countries and data centers, all connected to each other.

Data does not travel in one piece. It is split into packets, small parts of about 1,500 bytes each. Every packet carries the address it is meant for.
A router is a device that receives a packet and sends it on to the next network along the way. A packet passes through 10 to 20 routers, and each one brings it closer to its target.
An IP address is the number of a device on the network, for example 93.184.215.14. Your own computer can always be called localhost (127.0.0.1).
A port is a number from 0 to 65,535 for a program on that computer. The IP address is the building, and the port is the apartment. Web servers usually listen on 80 (HTTP) and 443 (HTTPS).

The rules for how to talk are called protocols. IP delivers single packets but does not promise they will arrive. TCP builds on IP: it makes a connection between two programs, numbers the packets, puts them in order and resends the lost ones. That is why the two together are called TCP/IP.

More precisely, if you are curious

Addresses like 93.184.215.14 are IPv4: four numbers from 0 to 255, about 4 billion addresses. They are not enough, so there is also IPv6: 2606:2800:21f:cb07:6820:80da:af6b:8b2c.
Besides TCP there is also UDP: no connection and no resending of what was lost, but faster. It is used for live voice and video and for games. The web runs on TCP.
The two ends of a TCP connection are called sockets: an address and a port on each side. The browser's port is picked automatically, from the free ones.

The idea

A name instead of a number: DNS

People remember names, computers look things up by IP address. A domain is the human name of a site: shop.example.bg. DNS (Domain Name System) turns the name into an address:

The browser asks the provider's DNS server: "what is the address of shop.example.bg?"
If it does not know, it asks along the chain: the server for .bg, then the server of example.bg.
The answer is an IP address. The browser remembers it for a while and does not ask next time.

Only then does the browser connect over TCP to that address, on port 443.

The whole trip of one page, with the part of the address each step uses:

Uri address = new Uri("https://shop.example.bg/products?category=coffee");
Console.WriteLine("1. DNS: what is the address of " + address.Host + "?");
Console.WriteLine("2. TCP: connection to port " + address.Port);
Console.WriteLine("3. HTTP: GET " + address.PathAndQuery);

Step by step, in words

The address says everything needed: which protocol, which computer and which resource.
The name comes first: the browser asks DNS for the IP address of shop.example.bg.
DNS answers with an IP address, for example 93.184.215.14. The browser remembers it.
A TCP connection to that address. The port is not in the address, so it is 443, the one for https.
An HTTP request goes over the connection: the GET method and the path /products?category=coffee.
The server answers: status 200 OK and HTML. The browser draws the page.
Three steps: a name in an address (DNS), a connection (TCP), a request and a response (HTTP).

Hosting is the computer the site lives on: a server in a data center, always on and connected. The domain points to the hosting: DNS says "shop.example.bg is at this IP address".

More precisely, if you are curious

You register a domain for a yearly fee through a registrar: a company accredited for that ending (.bg, .com, .eu). The registration is for a fixed term, usually a year, and you renew it. If you do not renew it, the name is released and anyone can take it. A good domain is short, easy to remember and says what the site is for.
There are several kinds of hosting. Shared: many sites on one server, cheap, but your neighbors affect you. VPS (virtual private server): your own part of a server with guaranteed resources. Cloud: you pay for what you use and grow when you need to. Dedicated server: a whole machine just for you. The choice depends on the traffic, the resources you need and the budget.

See it

The address and its parts

The address of a page is called a URL (Uniform Resource Locator). It says everything needed to get to the page. In documentation you will also meet the more general name URI (Uniform Resource Identifier), which is where the class name comes from. In C#, the Uri class gives you the parts:

Uri address = new Uri("https://shop.example.bg:8443/products?category=coffee&page=2");
Console.WriteLine("Scheme: " + address.Scheme);
Console.WriteLine("Host: " + address.Host);
Console.WriteLine("Port: " + address.Port);
Console.WriteLine("Path: " + address.AbsolutePath);
Console.WriteLine("Query: " + address.Query);

Uri home = new Uri("https://shop.example.bg/");
Console.WriteLine("Default port: " + home.Port);

Scheme: https
Host: shop.example.bg
Port: 8443
Path: /products
Query: ?category=coffee&page=2
Default port: 443

Scheme (https): which protocol.
Host (shop.example.bg): which computer. The name goes through DNS.
Port (8443): which program on it. If it is not in the address, it is the default for the scheme: 443 for https, 80 for http.
Path (/products): which resource on the server.
Query (?category=coffee&page=2): extra parameters as name=value pairs, separated by &.

What will the program print?

Uri cart = new Uri("http://shop.example.bg/cart?step=2");
Console.WriteLine(cart.Port + " " + cart.AbsolutePath);

Answer

80 /cart

443 /cart

80 /cart?step=2

More precisely, if you are curious

An address can contain only Latin letters, digits and a few symbols. Everything else, such as accented letters and spaces, is written as UTF-8 bytes with a % in front. The browser shows you "café crème", but what goes over the network is the encoded form:

Uri search = new Uri("https://shop.example.bg/search?q=café crème");
Console.WriteLine(search.AbsoluteUri);
Console.WriteLine(Uri.UnescapeDataString(search.Query));

https://shop.example.bg/search?q=caf%C3%A9%20cr%C3%A8me
?q=café crème

After the path there can also be a fragment: #reviews. It never reaches the server: the browser just scrolls the page to that part.

The idea

An HTTP request

The browser has connected. Now it sends an HTTP request (HyperText Transfer Protocol). The request is text:

GET /products?category=coffee HTTP/1.1
Host: shop.example.bg
Accept: text/html
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) Firefox/131.0

The first line is the method, the path and the version. The method says what you want: GET is "give me", POST is "here is some data" (an order, a form). There are also PUT (replace), DELETE (delete) and others.
Then come the headers, one per line: name: value. Host says which site you want, because one server can host many sites. Accept says what the browser understands, and User-Agent says who it is.
A blank line ends the headers. After it there can be a body: the data of the request. GET has no body, POST usually has one:

POST /cart HTTP/1.1
Host: shop.example.bg
Content-Type: application/x-www-form-urlencoded
Content-Length: 19

product=C-101&qty=2

Content-Length says how many bytes the body is, so the server knows when it has received all of it.

The idea

An HTTP response

The server answers with the same kind of text. The first line carries the status: a number and a short explanation. The first digit tells you the kind:

Status	Kind	Example
2xx	success	`200 OK`: here is what you asked for
3xx	redirect	`301 Moved Permanently`: it is at another address now
4xx	error in the request	`404 Not Found`: there is no such thing
5xx	error on the server	`500 Internal Server Error`: the program on the server failed

After that come the response headers and the body: the HTML of the page, JSON with data, an image.

HTTP/1.1 404 Not Found
Content-Type: text/html; charset=utf-8
Content-Length: 24

<h1>No such product</h1>

Content-Type says what the body is and in which encoding. On a redirect, the Location header gives the new address, and the browser sends a new request there by itself:

HTTP/1.1 301 Moved Permanently
Location: https://shop.example.bg/products

More precisely, if you are curious

HTTP has no memory (it is stateless): every request stands alone, and the server does not remember the earlier ones. So to "remember" you in the cart, the site gives you a cookie: a header the browser sends back with every following request.
A DDoS attack over HTTP uses the same lack of memory: thousands of infected computers send a huge number of requests to one server. Each one looks normal, and the server has to handle it, until it can no longer keep up with real customers either. The defense sits in front of the server: services that filter suspicious traffic and limit the number of requests from one address.

See it

The response, taken apart with code

A response is just text, so you can take it apart with string methods. Every line in HTTP ends in \r\n, and the blank line between the headers and the body is \r\n\r\n:

string response = "HTTP/1.1 200 OK\r\n" +
    "Content-Type: text/html; charset=utf-8\r\n" +
    "Content-Length: 21\r\n" +
    "\r\n" +
    "<h1>Café crème</h1>";

string[] parts = response.Split("\r\n\r\n");
string[] lines = parts[0].Split("\r\n");
string[] status = lines[0].Split(' ', 3);
Console.WriteLine("Version: " + status[0]);
Console.WriteLine("Status: " + status[1] + " (" + status[2] + ")");

Dictionary<string, string> headers = new Dictionary<string, string>();
for (int i = 1; i < lines.Length; i++)
{
    string[] header = lines[i].Split(": ", 2);
    headers[header[0]] = header[1];
}
Console.WriteLine("Type: " + headers["Content-Type"]);
Console.WriteLine("Length: " + headers["Content-Length"] + " bytes");
Console.WriteLine("Body: " + parts[1]);

Version: HTTP/1.1
Status: 200 (OK)
Type: text/html; charset=utf-8
Length: 21 bytes
Body: <h1>Café crème</h1>

Split("\r\n\r\n") splits it in two: the header part and the body.
The header part is split into lines. The first one is the status. Split(' ', 3) gives at most 3 parts: that way, with 404 Not Found, the explanation stays whole.
The headers go into a dictionary: name → value. Split(": ", 2) splits only at the first : .

Every program that speaks HTTP works like this. In the next lesson, HttpClient will do it for you.

More precisely, if you are curious

Content-Length is 21 bytes, but the text is 19 characters. What travels over the network is UTF-8 bytes, and there every accented letter, like é or è, takes 2 bytes:

string body = "<h1>Café crème</h1>";
Console.WriteLine(body.Length + " characters");
Console.WriteLine(System.Text.Encoding.UTF8.GetByteCount(body) + " bytes");

19 characters
21 bytes

Header names do not depend on uppercase and lowercase: content-type and Content-Type are the same. You make a dictionary that compares this way with new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase).
HTTP/1.1 is text. The newer HTTP/2 and HTTP/3 send the same methods, headers and statuses, but in binary form and with many requests over one connection. For your program the meaning is the same.

Try it

A header with a colon

The program should print the name and value of the header, but instead of the address it prints only https. Which line is the bug on?

Pick the line with the bug.

The idea

HTTPS: the same, but locked

HTTP is plain text. Any router along the way can read your password or change the price. HTTPS is HTTP over TLS (the older name is SSL):

Handshake. The browser and the server agree on how to encrypt and exchange keys.
Certificate. The server shows a certificate signed by an organization the browser trusts. That way you know you are talking to the real shop.example.bg, not to someone in the middle.
Encryption. From then on, the requests and responses travel encrypted. Along the way, anyone can see which server you are going to, but not what you are saying to each other.

The methods, headers and statuses are the same as in HTTP. The padlock in the address bar means the connection is over HTTPS.

The idea

What the browser does

The first response is HTML: the text and structure of the page. The browser reads it and finds more resources in it: styles (CSS), programs (JavaScript), images, fonts. For each one it sends a new request. An ordinary page makes dozens of requests before it is ready. That is why a site with a lot of heavy images loads slowly, even if its HTML is small.

Then the browser lays everything out and draws it on the screen. JavaScript can change the page after that, without loading it again.

More precisely, if you are curious

A browser has several parts. The user interface: the address bar, the buttons, the bookmarks, everything except the page itself. The browser engine connects the interface to the rendering engine, which understands HTML and CSS and draws them. The networking part makes the HTTP requests. A drawing layer for basic controls (drop-down lists, windows), the same across all operating systems. The JavaScript interpreter runs the scripts. Storage keeps cookies and site data (localStorage, IndexedDB).
How drawing works: from the HTML, the browser builds a tree of elements (the DOM), and from the CSS, a tree of styles (the CSSOM). Together the two give the render tree. Then it works out where each element goes and how big it is (layout), and finally it draws (paint). JavaScript changes the DOM, and the browser redraws only what was affected.
To read more: How the internet works, An overview of HTTP, How browsers work and HTTP flood DDoS attacks.

Exercise

Addresses, statuses and requests

Solve these in your editor.

1. Parts of an address. For https://shop.example.bg/search?q=espresso&max=30&page=2, print the host, the port and the path. Then split the query into pairs in a dictionary (TrimStart('?') removes the question mark, then Split('&') and Split('=')), print them and get the page number as a number:

Host: shop.example.bg, port 443
Path: /search
q = espresso
max = 30
page = 2
Page: 2

2. What the status means. A method Meaning(int code) with a switch returns the meaning of the known codes 200, 201, 301, 404 and 500. For the others it gives the kind by the first digit (code / 100), and for an unknown kind it returns "unknown code":

200: success
201: created
301: moved permanently
404: no such resource
418: request error
500: server error
503: other server error

3. Build a request. A method BuildGet(Uri address) returns the text of a GET request with Host and Accept: text/html, with \r\n at the end of every line and a blank line at the end. address.PathAndQuery gives the path together with the query. Print it, replacing every \r\n with the visible text \r\n and a new line:

GET /products?category=tea HTTP/1.1\r\n
Host: shop.example.bg\r\n
Accept: text/html\r\n
\r\n

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·

How a page gets to you

Cookies