How can Facebook's data center design apply to your data center plans?
Over the past year, Facebook has thrown some interesting wrenches into the
gears of the traditional networking industry. While mainstream thinking is to
keep most details of your network operations under wraps, Facebook has been
freely sharing its innovations. For a company whose business model is built on
people sharing personal information, I suppose this makes perfect sense.
What makes even more sense is the return Facebook gets on their openness.
Infrastructure VP Jason Taylor estimates that over the past three years Facebook
has saved some $2 billion by letting the members of its Open Compute Project
have a go at its design specifications.
But what really turned heads was last year's announcement of Wedge, an open
top-of-rack switch developed with the OCP community. Wedge was followed eight
months later by 6-Pack, a modular version of Wedge purposed for the network
core. Added to these bare-metal switches is FBOSS, an open Linux-based network
operating system (well, not exactly an operating system – more on that in a
later post), and OpenBNC for system management.
Why this openness matters to the rest of us is that all of this is not just a
mad-science project within Facebook's lair. You can soon buy Wedge through
Taiwanese switch manufacturer Accton, bringing switches into your data center
for a fraction of the cost of proprietary switches with integrated operating
systems. And you're not locked in to running FBOSS on the switch either. You can
shop around, choosing the NOS that makes the most sense to you, such as Open
Network Linux, Cumulus Linux, Big Switch Switch Light, and possibly others such
as Pica8's PicOS or even Juniper's JUNOS. If you have an intrepid team of
developers with time on their hands you can even build your own.
I'll write more about open switches and open software in subsequent articles,
but for now I want to focus on what Facebook has been sharing about their
innovations in data center network design and what it means for you. Last
November, between the announcements of Wedge and 6-Pack, Facebook opened its
newest data center in Altoona, Iowa. And as it has done with its other network
innovations, Facebook openly shared its new design.
It turns out that there are some valuable takeaways from the Altoona design that
can be applied to data centers of any size.
Hyperscale Misconceptions
Say "hyperscale data center" to most anyone who keeps up with such things,
and they'll reflexively name Facebook, Google, and Amazon. And because of this
association, people think of hyperscale as something that applies only to
mammoth data centers supported by an army of developers.
In reality, hyperscale just means the ability to scale out very rapidly. A
hyperscale data center network might be small, but it can grow exponentially
larger without changing the fundamental components and structures of the
network. You should be able to use the same switches and the same interconnect
patterns as you grow – just more of them. You do not need to throw out one class
of switches for another just to accommodate growth.
You can have a data center consisting of just a few racks, and if the network is
designed right it is a hyperscale data center. Hyperscale is a capability, not a
size.
Another misconception about hyperscale data centers is that they are optimized
for one or a relatively few applications at massive scale across the entire data
center. This stems particularly from the Facebook and Google associations.
Hyperscale designs are in fact ideal for very heavy east-west workloads, but
hyperscale design principles can apply to an average enterprise data center,
supporting hundreds of business applications just as easily as it supports a
single social media, big data, or search app.
Hyperscale also conjures up images of do-it-yourself networks built from the
silicon up by a cadre of brilliant young architects commanding salaries far out
of reach of the average network operator. That might be true of the innovators,
but because Facebook has laid its work right out on the table, mere mortals like
you and I can put their design principles to work in our own data centers.
To appreciate the significance of the Altoona network, let's first have a look
at the network architecture Facebook is using in its earlier data centers.
Good is not good enough: Facebook's cluster design
Figure 1 shows Facebook's pre-Altoona aggregated cluster design, which they
call the "4-post" architecture. Up to 255 server cabinets are connected through
ToR switches (RSW) to high-density cluster switches (CSW). The RSWs have up to
44 10G downlinks and four or eight 10G uplinks. Four CSWs and their connected
RSWs comprise a cluster.
041415 figure 1
Four "FatCat" (FC) aggregation switches interconnect the clusters. Each CSW has
a 40G connection to each of the four FCs. An 80G protection ring connects the
CSWs within each cluster, and the FCs are connected to a 160G protection ring.
This is a good design in several ways. Redundancy is good; oversubscription is
good (generally 10:1 between RSWs and CSWs, 4:1 between CSWs and FCs); the
topology is reasonably flat with no routers interconnecting clusters; and growth
is managed simply, at least up to the 40G port capacity of the FCs, by adding
new clusters.
But Facebook found that good is not good enough.
Most of the problems with this architecture stem from the necessity of very
large switches for the CSWs and FCs:
With just four boxes handling all intra-cluster traffic and four boxes handling
all inter-cluster traffic, a switch failure has a serious impact. One CSW
failure reduces intra-cluster capacity by 25%, and one FC failure reduces
inter-cluster capacity by 25%.
Very large switches restrict vendor choice – there are only a few "big iron"
manufacturers. And because these few vendors sell relatively fewer big boxes,
the per-port CapEx and OpEx is disproportionately high when compared to smaller
switches offered by a larger number of vendors.
The proprietary internals of these big switches prevent customization,
complicate management, and extend waits for bug fixes to months or even years.
Large switches tend to have oversubscribed switching fabrics, so all ports
cannot be used simultaneously.
The cluster switches' port densities limit the scale and bandwidth of these
topologies, and make transitions to next-generation port speeds too slow.
Facebook's distributed application creates machine-to-machine traffic that is
difficult to manage within an aggregated network design.
The individual pods are connected via 40G uplinks to four spine planes, as shown
in Figure 3. Each spine plane can have up to 48 switches. Key to this topology
is that the fabric switches each have an equal number of 40G downlinks and
uplinks – maxing out at 48 down an 48 up – so the fabric is non-blocking and
there is no oversubscription between pods. Bisectional bandwidth, running to
multi petabits, is consistent throughput the data center.
The diagram in Figure 3 shows the color-coded connections between fabric
switches and their corresponding spine planes, but doesn't do justice to how it
all ties together. And something that surely strikes you is that there are a lot
of links between fabric switches and spine switches. Optics and cables can
become expensive, so it's important to manage the distances between pods and
spine planes. (If you're interested in learning more about Facebook's
architectures, here are the source documents I used for cluster architecture
(PDF) and the Altoona architecture.)
If you rotate the pods and line them up, the way the 48 racks of each pod would
be arranged into rows in the data center, and then do the same with the spine
planes – but lining them up perpendicular to the pods – you get the
three-dimensional diagram shown in Figure 4, with the fabric switches becoming
part of the spine planes. Distance between fabric switches and spine switches
are reduced. Note that there are also edge pods, which provide external
connectivity to the fabric.
Facebook network engineer Alexey Andreyev describes the fabric this way: "This
highly modular design allows us to quickly scale capacity in any dimension,
within a simple and uniform framework. When we need more compute capacity, we
add server pods. When we need more intra-fabric network capacity, we add spine
switches on all planes. When we need more extra-fabric connectivity, we add edge
pods or scale uplinks on the existing edge switches."
If you want to hear Andreyev describe the Altoona architecture himself, here's
an excellent video:
Altoona Takeaways
You might be wondering by now what any of this has to do with you and your data
center. After all, Facebook is supporting more or less a single distributed
application generating machine-to-machine traffic spanning its entire data
center. You probably don't. And while a 48-rack pod is a scale-down from their
earlier clusters, most enterprise data centers in their entirety are smaller
than 48 server racks.
So why should you care? Because it's not the scale. It's the scalability.
The fundamental takeaways from the Altoona design are the advantages of
building your data center network using small open switches, in an architecture
that enables you to scale to any size without changing the basic building
blocks. First look at the switches. You don't have to wait for Wedge or 6-Pack
to go on the market (Accton will be selling Wedge soon). You can pick up
bare-metal switches from Accton, Quanta, Celestica, Dell, and others for a
fraction of the cost a big-name vendor will charge. For example, a Quanta switch
with 32 40G ports lists for $7,495. A Juniper QFX5100 with 24 40G ports lists
for a little under $30,000. Is that a fair comparison? That JUNOS premium gives
you a pretty awesome operating system, but the bare-metal switch gives you a
bunch of options for loading an OS of your choice.
As for the pod and core design, that can be adjusted to your own needs. The pod
can be whatever size you want; while the "unit of network" is a wonderful
concept, it's not a rule. You can create a number of pod designs to fit specific
workflow needs, or just to start a migration away from older architectures. Pods
can also be application specific. As your data center network grows, or you
adopt newer technologies, you can non-disruptively "plug in" new pods.
The same goes for the core part. You can build it at layer 2, or at layer 3. It
all depends on the workflows you're supporting. Using a simple pod and core
design you can manageably grow your data center network at whatever rate makes
sense to you, from a new pod every few years to an explosive growth of new pods
every few months.
