|
July 1997
By Eric Brewer, Inktomi Corp.
Clustering
Clustering Multiply and Conquer
Clustering enables net managers to add resources as needed----without resorting to forklift upgrades
|
|
Net managers know that a straight line is the shortest distance between two points, but they seem to spend a lot of time going around in circles--especially when it comes
to servers. They deploy a shiny new box, but somewhere down the road it chokes on the amount of traffic that's thrown at it. So it gets replaced by a bigger machine. But eventually that maxes out, leading to yet another forklift upgrade. And the beat goes onÉ
That's where clustering comes in. By teaming multiple machines to
work on one task, clustering allows nearly infinite
scalability--without the forklift. And because clustered machines work
together, availability increases: If one node fails, any of the others
can take up the slack.
What's more, clustering eliminates the sort of guesswork that
passes for capacity planning for servers. The technology makes it
possible to add resources as needed--think of it as just-in-time
deployment. That's a lot more efficient--and less expensive--than
getting stuck with a server that's underpowered out of the box or
paying for more performance than needed at present.
There are more than servers at stake, though. Clustering also is a
natural for
emerging network-aware apps, like Web caching, smart
firewalls, and push servers--where the goal is to reduce or
redistribute traffic. Think of it this way: The technology's
unsurpassed scalability makes it possible to deal with traffic that is
growing at a steep but unpredictable rate. And clustering's
availability can help keep the network up and running no matter
what.
|
MORE INFO
Clustering vs.SMP
|
Sounds great. But clustering is still fraught with confusion and
complexity. Putting it to work means understanding different
architectures, along with their strengths and weaknesses. It also
means coming to terms with some serious challenges: To begin with,
clustering hardware and software aren't easy to implement. On top of
that, existing applications may need to be substant
ially rewritten.
Further, the technology isn't the only (or necessarily the best)
solution. Symmetrical multiprocessing (SMP) servers may be adequate,
and in some cases, they might even be preferable (see
"Clustering vs.
SMP"
).
CLUSTER CLASSIFICATIONS
Unfortunately, the term "cluster" has broad and ambiguous usage.
There are at least three types of clusters in use today: server farms,
failover clusters, and coupled clusters.
Server farms, the oldest and simplest scheme, are a collection of
machines (nodes) that draw work from a central unit, known as a
manager. This approach is well-suited to brute-force applications,
when massive amounts of processing need to be performed--as long as
there's relatively little need for internode communication.
Pixar Animation Studios used a server farm to render the frames of
the movie Toy Story. In this case, the overall application comprises
many small, processor-intensive tasks, each of which can be execu
ted
on one node within the cluster. And simple fault tolerance ensures
that if a request on one machine fails, the manager reassigns the
request to another machine .
But server farms could be characterized as "embarrassingly
parallel." The model doesn't work well when (as is the case with most
applications) machines need to collaborate on a given task.
Failover clusters represent another relatively old and extremely
simple architecture. Here, the focus is on availability rather than
scalability. Typically, there are two nodes, a primary and a backup,
that together provide continuous service. If one fails, the other
takes its place.
Coupled clusters go beyond simple scaling and availability by
harnessing multiple machines that work very closely together on the
same problem. Unlike server farms, the work is not independent, and it
requires a significant amount of internodal communication.
And unlike failover clusters, availability is a group
responsibility. That means that couple
d clusters have the potential to
handle multiple failures gracefully.
COUPLED CLOSE-UP
There are two types of coupled clusters: Shared-nothing and
shared-memory. Each of the nodes in a shared-nothing cluster has its
own memory space and communicates by passing messages. This is a very
tricky architecture to program for, since it requires an awareness of
each node in a cluster. For example, the application must keep tabs on
which CPU is running which process and be smart enough to reassign
processes if a node fails.
Shared-memory is the Holy Grail of clustering. All nodes share the
same address space. Thus, an app doesn't need to be aware of
individual nodes. It treats the entire cluster as a single entity.
Nodes communicate using the IEEE's scalable coherent interface (SCI).
This standard, although troubled and inconsistent, has led to the
emergence of shared-memory hardware from multiple vendors.
Coupled clusters represent the future of this technology. They
scale seamless
ly, handle nearly any application, use commodity
hardware, and offer high availability under an extremely wide range of
scenarios. The downside is that they also are the most complex and
difficult type of cluster to program. To understand how best to put
them to work, it's helpful to take a closer look.
THE FAB FOUR
|
MORE INFO
Getting Clued In to Clustering
|
Coupled clusters provide four primary benefits over single larger
machines such as SMPs: absolute scalability, incremental scalability,
high availability, and superior price/performance.
The most obvious benefit is absolute scalability. It's possible to
create large clusters that dwarf the power of even the biggest
standalone machines. For example, the Wal-Mart retail chain uses a
cluster from AT
&T Global Information Solutions (Dayton, Ohio) with 768
processors and up to 24 terabytes of online storage.
Incremental scalability--the ability to grow clusters over time as
capacity needs increase--is an even bigger benefit. This is a
tremendous advantage in areas of unknown growth (nearly everything
related to the Internet). Incremental scalability eliminates the
painful choice between a smaller, lower-cost system that will soon be
outgrown or a larger machine that may remain underutilized for some
time. Clusters also do away with the need for forklift upgrades in
which the investment in a current machine is lost when it's replaced
with a larger one.
High availability is at least as important as scalability. Since
each node has its own buses, power supply, and disks, there is no one
central failure that can take down the cluster. And in many cases,
fault tolerance can be handled in software. Trouble is, developing
such software is a major challenge.
Clusters also tend to win over h
igh-end, low-volume machines in
terms of price/performance. By using commodity building blocks, it's
possible to put together a cluster with equal or greater computing
power than a single large machine at a much lower cost. The most
reliable building blocks (whether it's CPUs, disks, memory, or power
supplies) tend to be the high-volume units of the previous generation.
This equipment typically features much better service and support,
particularly for PC hardware--which translates into lower cost of
ownership.
CLUSTER CONUNDRUMS
Clusters have significant advantages, but developing cluster-aware
apps is more of job for application developers than network managers.
Still, corporate networkers need to be aware of the issues (after all,
they're the ones who'll get the calls when the system doesn't perform
as promised). Further, the application developers may need a lot of
information about network traffic if they're going to deliver
optimized code.
Essentially, a well-written cluster-
aware app needs to deal with
three issues: partial failures, sharing memory, and managing the
cluster.
Handling partial failures--the ability to survive and adapt to
failures of subsets of the system--is the first order of business.
Standalone workstations and SMPs never face this issue, since the
machine is either up or down. But any part of a cluster may fail--and
different apps respond in different ways. It's important to understand
that "fail" in this case doesn't simply refer to hardware: If an app
requests a block of memory space and is told that it's not available,
the process can fail.
There are two ways to deal with partial failure. Highly available
clusters offer a high probability that all resources will be in
service--without furnishing any guarantees. In the rare case that a
node fails or a requested resource isn't available, the queries in
progress are lost. But if retried they will be serviced by a different
node.
Fault-tolerant clusters ensure that all resources are al
ways
available--no matter what. This sort of service is a must for
transaction-processing systems: All committed transactions must
survive faults.
How do corporate networkers decide which approach to take? For
starters, highly available systems are easier to implement--and often
good enough. The first job is to define the requirements carefully. Is
it OK to lose access to a subset of the data during failures? Is it
easy to retry failed queries? Retry is by far the simplest and most
general approach. Typically this is done by having some type of query
manager that takes responsibility for retrying failed requests until
they complete successfully, thus isolating failures from clients. Of
course, another mechanism is needed to recover from manager
failures.
|
The Buddy System
|
One way to deal with fail
ed requests is the so-called buddy system,
which achieves full access despite multiple faults (see
Figure 1
). In
this scheme, each node is connected to a RAID (redundant array of
independent disks) subsystem, which is then linked to another node
(the buddy). During normal operation, a node only accesses its own
RAID. But if a node fails, its buddy handles all accesses to that
node's storage. Thus, the requesting node knows to try the buddy when
the primary fails. This gives high fault tolerance for nodes and disks
at minimal cost (an extra connection to each RAID). Faults will
increase the load on the buddy node, which may affect performance, but
data remains accessible.
Sharing memory also is an issue-- at least for coupled clusters. As
noted, the goal here is for the entire cluster to appear to the app as
a single entity.
Several companies sell hardware optimized for shared-memory
clusters (SMCs). These machines simplify the porting of applicati
ons;
there's no need to make the application aware of individual elements
within the cluster. But there are two drawbacks. First, SMCs
exacerbate the problems of partial failures: The failure of even one
request typically causes the whole program to fail. It's possible to
rewrite the app to handle these failures by redirecting failed
requests to other nodes within the cluster, but that defeats the
purpose of using shared memory. Second, application performance varies
by orders of magnitude depending on whether the data requested resides
in local memory or cache or if it must be called from another node.
Managing a cluster effectively also is a challenge. The basic
problem is that the available tools are relatively immature. Ideally,
tools need to do two things. First, they should ensure that all
machines deliver the same functionality--or that there are relatively
few variations. This isn't to say that the hardware needs to be the
same but that the functionality of the nodes is symmetrical--or close
to it. From a management perspective, this means any node can "cover"
for any other, and that monitoring and configuration is the same for
all nodes.
Second, the tools should be able to monitor and configure all nodes
from a single, potentially remote screen. The common alternative is to
manage each node individually by logging in to each-- an approach that
doesn't scale well.
Eric Brewer is cofounder and chief technology officer of Inktomi Corp. (San Mateo, Calif.), a maker of clustering software. He also is a professor of computer science at the University of California at Berkeley.
[
Home
]
[
Registration
|
Subscriptions
]
[
Contact Us
|
E-Mail
]
|
|
|
 |
|
|
