Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

SGI -- 64-way SMP with ccNUMA, IA64 only

By kmself in News
Wed Apr 05, 2000 at 09:44:59 PM EST
Tags: Technology (all tags)

In the "Hey, cool!...waitaminute" dept. SGI has announced intent to enable up to 64-way SMP support into Linux, according to the Register and ComputerWorld. This is a huge step up from current 4-to-8 SMP capabilities...but it's only for Intel IA64 (aka Itanium) processors.

I've done some lay research into clustering and SMP type issues. There's a good general book (if slightly aged, In Search of Clusters, by Gregory F. Pfister on the topic, one of the lessons taken from it is that SMP is a complex problem, though it does trade off complexity of hardware design for complexity of software design vis-a-vis clustering solutions such as Beowulf and Mosix. Pfister is a huge fan of clustering.

So the questions are: is SMP a problem inherently linked to hardware and chip design, making generalized SMP solutions an impracticality? SGI had previously announced intent to scale Linux systems to 16 CPUs. The 64 CPU initiative is cool, but would put IA64 and all other chip support on different tracks. Is there a reason that proprietary support for clustering tends to be among OSs which are strongly tied to a particular hardware architecture: Solaris / SPARC, HPUX / PA-RISC, Irix / Mips, OS/390 / S/390? Are the solutions inherently bound to the hardware they run on?


Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure


Related Links
o Register
o ComputerWo rld
o In Search of Clusters
o Also by kmself

Display: Sort:
SGI -- 64-way SMP with ccNUMA, IA64 only | 14 comments (14 topical, editorial, 0 hidden)
Hey, how about a review of that Pfi... (3.00 / 1) (#3)
by marlowe on Wed Apr 05, 2000 at 04:07:14 PM EST

marlowe voted 1 on this story.

Hey, how about a review of that Pfister book?
-- The Americans are the Jews of the 21st century. Only we won't go as quietly to the gas chambers. --

Interesting, informative and way ov... (1.00 / 1) (#2)
by nascent on Wed Apr 05, 2000 at 04:29:53 PM EST

nascent voted 1 on this story.

Interesting, informative and way over my head. =)

To answer your questions, kmself, t... (4.00 / 1) (#1)
by fluffy grue on Wed Apr 05, 2000 at 05:24:00 PM EST

fluffy grue voted 1 on this story.

To answer your questions, kmself, this sort of clustering approach isn't at all tied to any sort of hardware or OS. All that ccNUMA requires is having an MMU, an OS which lets you hook in to segmentation violations (such as UNIX), and some sort of (preferrably-fast) networking hardware. Basically, ccNUMA uses a network as a shared bus, and functionality which has been in UNIX for a long time (mmap, mprotect, and signal) to implement distributed shared memory in software. There are actually quite a few distributed shared memory schemes, though ccNUMA (short for Cache-Coherent Non-Uniform Memory Architecture) is by far the most efficient one out there, is rather flexible (each node doesn't need to have the same amount of memory), and isn't too hard to implement.

This is actually little different than a ccNUMA-based Beowulf clustering solution, except that it's SGI-branded. I don't see why they've limited it to 64 processors. I also don't see why they're apparently intent on putting it into the kernel - this is the kind of thing where userspace libraries have existed for a relatively long time and do just as good a job. Mostly this just seems like marketing to me; there is no reason this should be tied to IA-64 at all, since the whole point to ccNUMA is that the processors *aren't* all on the same bus, and *don't* have physical access to the same chunk of memory.
"Is not a quine" is not a quine.
I have a master's degree in science!

[ Hug Your Trikuare ]

Oops, somewhat of a correction (4.00 / 1) (#4)
by fluffy grue on Wed Apr 05, 2000 at 11:37:34 PM EST

Before I posted this, I was under the impression that ccNUMA was a standard approach for cache coherency on network distributed memory, and not something which is SGI's intellectual property. However, there's still nothing special about the technique, and certainly nothing which ties it to Itanium. I believe there are other network distributed memory models which DO have userspace libraries, and again, there's no reason ccNUMA need be in kernel space. Most likely SGI feels they can only be buggered to port a binary-only library with a 64-CPU license to Linux, though, in the way that SGI usually does these things. If it hasn't been done already, though, I'm sure it's only a matter of time before ccNUMA is implemented as a Free(speech) library, just under a different name.

Again, there's nothing special about this approach which ties it to Itanium or the R10k or whatever. It can be implemented just fine on anything with an MMU - that is, a cluster of 386-based Linux boxen can go as many ways as you want, though of course performance is limited by the network.

Really, distributed memory can be implemented without too much code. The hard part is the cache coherency, and ccNUMA's algorithms really aren't that complex or difficult to implement.

On second thought, there IS one place where ccNUMA is useful to be implemented at a kernel level for SMP-type stuff: automatic spawning and load-balancing of threads. However, properly-written stuff (such scientific computing applications, which is what this is really for) is written to do that stuff on its own damned self. This isn't for webservers or playing games, folks, and there's better ways of dealing with both anyway.
"Is not a quine" is not a quine.
I have a master's degree in science!

[ Hug Your Trikuare ]
[ Parent ]

sgi and usually (none / 0) (#7)
by mattdm on Thu Apr 06, 2000 at 05:35:45 PM EST

I'm not sure what the basis for your "binary-only ... in the way that SGI usually does these things" comment is. Their open source projects include kernprof, bigmem, lkcd, nfs3, GLX, xfs, and more. In fact, SGI's "usual" method of porting things to Linux has been in the form of Free(speech) software.

[ Parent ]
Re: sgi and usually (none / 0) (#9)
by fluffy grue on Sat Apr 08, 2000 at 12:43:21 AM EST

Not for their cash cows, such as Performer and ccNUMA.
"Is not a quine" is not a quine.
I have a master's degree in science!

[ Hug Your Trikuare ]
[ Parent ]

Re: sgi and usually (none / 0) (#11)
by mattdm on Sat Apr 08, 2000 at 12:53:53 PM EST

Well, they haven't released ccNUMA yet in any way (not counting the kernel patches on their web site, which are GPL'd), so that hardly seems reasonable to use as an example. And Performer really seems to be the exception, rather than the rule.

[ Parent ]
Re: sgi and usually (none / 0) (#14)
by fluffy grue on Mon Apr 10, 2000 at 02:29:50 PM EST

ccNUMA is their cash cow for scientific and massive-server computing. It's speculation that it would be released binary-only, but the only thing I can think of which stands in the way of that is legal issues with the GPL, and they could always work around it by doing scary hacks with kernel modules, or by doing it as a client-side library, which is what I've been rambling on about being possible this whole time.
"Is not a quine" is not a quine.
I have a master's degree in science!

[ Hug Your Trikuare ]
[ Parent ]

hmmmm... (4.00 / 1) (#6)
by mattdm on Thu Apr 06, 2000 at 05:30:59 PM EST

I'm not an expert on this by any means, and don't pretend to be, but from reading the info from SGI on this, it looks like this involves changes to the memory manager, which is certainly a kernel thing, not userspace.

[ Parent ]
Re: hmmmm... (none / 0) (#10)
by fluffy grue on Sat Apr 08, 2000 at 12:48:34 AM EST

All of the ccNUMA functionality can be implemented in userspace. Granted, some of the functions need to override existing functions, but IMO it doesn't make it any easier to implement the important stuff (sharing/mapping file descriptors, process table stuff, etc.) in the kernel, and all that doing it in the kernel buys you is being able to migrate processes between nodes a little bit easier. These functions will have to override the UNIX system library no matter what, and whether it's done in the kernel vs. done the same way that Electric Fence overrides the malloc functions doesn't make any real difference, IMO.
"Is not a quine" is not a quine.
I have a master's degree in science!

[ Hug Your Trikuare ]
[ Parent ]

More information from SGI (5.00 / 1) (#5)
by mattdm on Thu Apr 06, 2000 at 05:25:17 PM EST

There's more information on this direct from SGI. (Including code.)

Re: SGI -- 64-way SMP with ccNUMA, IA64 only (none / 0) (#8)
by your_desired_username on Thu Apr 06, 2000 at 10:19:42 PM EST

From reading this, I get the impression that although ccNUMA support has only been implemented for IA64 and IA32, the authors have been careful to not to rule other architectures out.

Glancing over the patches, the code seems to be divided into a platform-independent patch, and a platform-specific patch.

Re: SGI -- 64-way SMP with ccNUMA, IA64 only (none / 0) (#12)
by Eldritch on Sun Apr 09, 2000 at 05:30:10 PM EST

As a home user, what I'd like to see is a software standard for SMP. The way that I see it, games could make excellent use of SMP for a home user (just buy another CPU instead of a more powerful computer) but they _rarely_ support more than 1 CPU. And its games that are driving the x86 makert. If some kind of standard was reached, the CPU manufacuturers would make more money. This enables them to make faster CPU's and makes the home consumer uprades cheaper. I might just be talking a load of bolls, but I'm suprised that this hasn't been mentioned.......

Re: SGI -- 64-way SMP with ccNUMA, IA64 only (none / 0) (#13)
by Anonymous Hero on Mon Apr 10, 2000 at 04:10:34 AM EST

you mean threads? They're standard on both win32 and linux. The reason games don't support SMP has nothing to do with library availability, it has to do with the fact that parallelizing a renderer or a physics simulator is not an easy task, and a very small % of gamers have SMP hardware. As SMP catches on more and more, expect to see more developers follow id's lead and thread their games.

[ Parent ]
SGI -- 64-way SMP with ccNUMA, IA64 only | 14 comments (14 topical, 0 editorial, 0 hidden)
Display: Sort:


All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!