Deploying globally-available and distributed applications is tricky and involves a lot of steps. Luckily, we're better off using services already made for this situation, such as serverless platforms or managed DNS with Anycast capabilities. But what does this actually mean and how is today's software deployed to be accessed from around the globe without performance implications? Let's dive into the underlying architecture of the Internet.
As outlined above, when we deploy our software, we most probably want it to be available around the globe, with the lowest-possible latency
regardless of user and server location. In the classic sense, with one static IP per server and one server deployed, it would be impossible
to point example.xyz to multiple deployments solely based on the requester's geolocation. But is it really impossible? Let's step back for a
second and check out how the internet is run today.
All about the DNS
You've probably already heard about the Domain Name System, DNS in short. Being one of the most important building blocks of the
internet, its main purpose is to resolve or translate literal domain names (like www.example.com) to IP addresses (like 1.2.3.4 or 2001:db8::ff00:42:8329).
This task is achieved through multiple layers of resolving: When you try to access this blog, for example, the first step will be to pass through your browser and OS caches to finally
contact your configured DNS resolver. This might be your router, ISP or a custom service like 1.1.1.1. The resolver will potentially reach out to other resolvers,
a process called recursive resolving, and then contact one of thirteen root nameservers deployed around the globe and operated by independent organizations supervised by the DNS Root
Server System Advisory Committee, a part of the ICANN (Internet Corporation for Assigned Names and Numbers). This is where the process of resolving your
DNS query starts.
Since hostnames are represented in a tree structure with the TLDs as their first branch nodes, the root nameserver will take a look at the domain's TLD, in our case .com and respond with
the TLD nameserver linked to the .com top-level domain, or rather its address. Now we continue our journey to the .com TLD nameserver, which will in return tell us how to contact the
authoritative nameserver assigned to example.com. An authoritative nameserver is the final DNS server that knows about the domain's configuration, for example web and mail servers available
for example.com.
Although DNS isn't tied to a specific transfer protocol, the User Datagram Protocol (UDP) is primarily used to transfer DNS queries over the network.
Anycast & BGP
Previously we saw how hostnames are resolved into routable IP addresses using the Domain Name System. This allows your browser and applications to know which machine to connect to when you access specific domains. However, right now we'd always get sent to the same machine, regardless of where we are (geolocation-wise). If you're not nearby, you're out of luck. Surely, this isn't in line with today's globally-distributed infrastructure. That's why we've got a couple of helpers at the foundation of the Internet routing system: Anycast and the Border Gateway Protocol.
Going by the definition, Anycast is "a routing methodology like unicast, multicast and broadcast, selecting the desired path on the basis of rules like the number of hops, distance, etc.". This means, that Anycast allows your packets to take the optimal route to their destination, in our case we want the least amount of hops which equals the closest Point of Presence nearby. Since the design of an Anycast-based network will potentially result in requests switching their destination, a connectionless/stateless protocol is used to transfer self-contained data packets without the need for an initial handshake to configure a connection. This is where UDP comes in again, and this is why DNS traffic is a perfect fit for Anycast-based systems since DNS queries are, unlike usual web traffic, sent over UDP instead of stateful protocols like TCP.
So at this point, we know that we want to leverage an Anycast-based routing method for accessing our authoritative nameservers to determine the closest deployment to each user. In practice, this is implemented by using the Border Gateway Protocol, BGP in short. Another core part of how the Internet works and why it breaks sometimes, BGP allows autonomous systems (AS), which are entities like your ISP or cloud provider (GCP, AWS, etc.) and other organizations connected to the world-wide-web, to announce IPs they're able to route to their peers. Seen as a whole, this creates a routing mesh network, which I've simplified below. You'll enter this network through one autonomous system (your ISP) and leave it at your destination AS (your hosting provider).
If you wanted to reach 2.3.4.5, BGP could, in theory, take two paths, going through AS2 to AS3 (the shortest solution), or through AS6, AS5 and AS4. Based on the router's configuration, it might
take the quickest path determined by the least amount of hops, so the former to route your traffic. The last step to achieve Anycast is for an address to be announced twice across a network,
this can happen inside of an autonomous system but also externally if we wanted to use multiple AS. Having the ability to announce whichever addresses we want to serve is one of the big
attack vectors of BGP named BGP hijacking, allowing malicious actors to reroute traffic for users in reach by announcing an existing address
twice, which happens quite often: Last summer, a huge portion of European traffic was routed through China
after routes leaked from a Switzerland-based autonomous system.
We basically want to run through the same process, announcing our authoritative nameserver IP addresses from multiple sources so consumers will get directed to the closest location, network-wise that is.
Say we've got our authoritative nameserver for example.com deployed to the IP 3.4.5.6. We'll also announce this address multiple times for each point of presence we want to serve. Now that
a German user, for example, wants to resolve the address details for our webserver at www.example.com, they'll go through the DNS chain and once the TLD nameserver points them to 3.4.5.6, they'll take the shortest
path to our authoritative nameserver, located in France. American users will be routed to our North American instance.
In the end, the authoritative nameserver will point our users to the region-specific datacenter where our services are running in.
Additional Learning Resources
If you want to learn more about the Domain Name System (DNS) and Border Gateway Protocol (BGP), I've selected a few resources below which I've used to write this post and learn about these systems as well. Helpful sites include
- Cloudflare: DNS basics
- Cloudflare: Anycast DNS
- Cloudflare: BGP
- DE-CIX: Traffic routing basics
- NS1: Anycast DNS
and countless Wikipedia articles.
I hope you enjoyed this post and maybe even learned something from it! As always, if you've got questions, suggestions or any feedback at all, don't hesitate to send a DM or write a mail.