Beowulf Home > Allen Advice

Allen Advice

Hello Joe/Mario:

This is in reference to advisory committee meeting earlier. I have included e-mail from Bruce for completion:

MEMORY/STORAGE

  1. Minimum amount of RAM: 256MB
  2. Minimum amount of hard disk: get as much as you can.
  3. Type of hard disk: ATA-66
  4. NFS-mounted disks: Local hard disk in conjunction with NFS-mounted volumes
  5. RAID: Too expensive
  6. LIGO data distribution method: Read the White paper
  7. On-site data storage: As much as possible

PROCESSORS

  1. Pentiums vs. Athlons: Depends on benchmarks... repeat with optimization switches on 2, 3, & 4: Follow the benchmarks

NETWORKING

  1. 100 Mbps or more: One can only afford 100 Mbps
  2. Mutiple NICs: If you can afford it
  3. Switches: Get switches with enough backplane (10/20 Gbps)

SOFTWARE

  1. MPICH vs. LAM Do not have enough experience
  2. SSH vs. RSH No preference

RACKMOUNT VS. BOXES

  1. Answer to all the questions: Test and evaluate price/performance

Joe,

You're very welcome. I think this will work out fine. Here's what's stuck with me after a night of sleep:

(1) You should work out a performance/price ratio for each option and compare the numbers. Our had units of FFTs/$-sec, for FFTs of length 256k points. To do this fairly, you need to make every effort to get fftw to run fast -- ie experiment with compiler optimization options.

This applies as well to questions like: what kind of memory chips should one use?

[By the way, since we may be doing something similar, I'm interested in the results of this -- a faxed copy of your final results table would be useful for me.]

(2) Prototype! Meaning: when you are fairly sure what you want, get (as loaners) two machines, two 100 switches, and 1 Gbit switch, and test the entire configuration together, before you issue the big purchase orders.

Do a networking test where you simultaneously transfer files in both directions over the same networking line -- it's a good test of full duplex connections. You should get 7-10 Mbytes/sec in both directions at the same time.

(3) Heat. Make sure you can keep it cool, and have a backup plan for cooling, if needed. Talk to your university HVAC people -- and listen to what they say.

(4) Disk/tape. Aim for balance. Meaning: spend some meaningful fraction of the total budget on this. Roughly speaking, the cost is split between CPU, memory, disk, tape, networking. For a good, general purpose system, you'll want a reasonable balance between these.

(5) Network bandwidth: try and avoid oversubscription of the central switch. It's intersection bandwidth should be at least as great as the total bandwidth of all the NICs. After thinking about your design a bit more, if you can manage it, it might be wise to use three 12-port 100 switches for each set of 32 nodes. Then have 12 gigabit lines going to a 12 port gigabit central switch. This should have 12 gigabit/sec intersection bandwidth, and avoids any oversubscription anywhere

(6) Two thing I forgot yesterday: Will you have a cdrom or floppy on each system? and, make sure you get sufficient electrical service. You'll probably want something like twelve 15 amp circults or ten 20 amp ones. If you are thinking about eventually doubling the size you'll want twice as many circuits (and 6 tons of cooling rather than 3).

Cheers, Bruce