darkness

This is part of a series of entries about building a new 2TB array. Or trying to, at least.

First thing I had to sort out was all of my I/O options.

• PCI: my current array sits on three HPT302 ATA/133 controllers. All on the same PCI bus. This is not a special PCI bus. I suspect it is your usual 32-bit, 33MHz PCI bus. There are, apparently, 66MHz PCI buses, and I’m quite certain there are 64-bit buses. There are 5V PCI slots/devices and 3.3V PCI slots/buses. I suspect I have the later. Anyway, PCI bus bandwidth: in the realm of 132MB/s. Five drives, even at, say, 60MB/s each: 300MB/s. Not so good.

  (Of course, what’s even worse are the HPT302 quasi-open-source drivers. I think recent kernels don’t need them, but at the time of RH8.0 I certainly needed them. From time to time the drives “stall”—I just don’t know what else to call it. Everything blocks for 10-20s, then it comes back and all is right in the world. Plus it’s slow according to hdparm, but I don’t really trust hdparm.)

• PCI-X: I’d say it’s PCI 2.0… but that’s really just the new version of PCI. Most everything you get these days in regular ol’ PCI is going to be PCI 2.2, I think. No, PCI-X is (in my words) an evolution of PCI. Cards and slots are both backwards compatible. PCI-X running at the same width and speed as PCI, is supposedly still faster than PCI.

  You’re most likely to find PCI-X to be 64-bits wide. I don’t think I’ve seen a current motherboard that has 32-bit PCI-X slots.

  PCI-X can run at 66MHz, 100MHz, or 133MHz. Technically, some newer version of PCI-X can run at 233MHz or even 533MHz, but I’ve never seen one faster than 133MHz.

  If you’re keeping up with math, you may have noticed some nice numbers: PCI-X 64-bit/100MHz = 800MB/s, PCI-X 64-bit/133MHz = 1GB/s. Thems fast. There are limits though:

  • A 133MHz bus can only support one device. A 100MHz bus can only support two devices. I believe a 66MHz bus can support four devices. On some motherboards I was looking at, things like the gigabit NIC would be on the same bus as one of the 64-bit/133MHz slots. So if you put a card in that slot, surprise! Your NIC and your device are both running at 100MHz now.

  • PCI-X is backwards compatible with PCI, and that goes for both slots and devices. This sounds nice, until you (almost immediately) realize that it means everything on a bus is going to run as slow as the slowest device. In reality, I read one source that said a PCI-X card on a PCI bus will always fall back to 33MHz-66MHz PCI isn’t an option for a PCI-X device. So, I’ve got three PCI PATA controllers I’d like to keep, and (for example) two new PCI-X SATA controllers I’d like to add. I need a board with at least one 64-bit/100MHz PCI-X bus with two slots (and just two slots on that bus), and then another three PCI slots on top of that. Hope you’ve got on-board NIC and video.

  The vast majority of motherboards I’ve seen don’t really seem to give me enough I/O options. I really want something insane like four PCI-X 133MHz buses with one slot each, and then a few more PCI slots. Maybe a PCIe (see below) or two. Oh, and under $200 too, please. I might go $250 if I have to.

• PCI Express: not to be confused with PCI-X! Yes, they’re different. Genius naming guys. “Like, these guys couldn’t have devised names that sound just a little more unique than ‘PCI-X’ and ‘PCI Express’?”

  Whereas PCI-X is based on PCI, PCI Express (PCIe) is apparently quite different. Serial instead of parallel, and capable of several metric assloads more speed. Slots and devices are no where near backwards compatible AFAIK. Also, PCIe used to be called 3GIO, in case you run into that term. PCIe and 3GIO are the same thing.

  PCIe is usually specified as “x1,” “x8”, etc. This specifies the number of “lanes” that the slot offers. A lane is a bi-directional serial channel capable of around 250MB/s in each direction (sending and receiving). So an x16 slot should be 4GB/s in each direction. It took me a little while to realize this was in contrast to PCI, which goes in one direction at a time. So where your PCI-X 64-bit/133MHz slot has 1GB/s bandwidth, your 4x PCIe slot really has 2GB/s (1GB/s transmit, 1GB/s receive, simultaneously).

  I think PCIe also offers some other features, like hot swap. It seems poised to be “the expansion bus of the future.” Of course, the main class of peripherals I see using it currently are graphics cards. I suspect this is often why you find a motherboard with one PCIe 16x slot and one PCIe 1x slot: the 16x is for your video card (P.S.: AGP 8x does 2.1GB/s! Let me stick a storage controller in that slot, baby) and the 1x slot is to make you feel slightly more manly. Or maybe it’s for a GB NIC.

This study of buses was necessary for me to decide what kind of storage controllers I’ll be interested in. Accordingly, I plan my next installment to cover the various options one has for SATA controllers.

I do have one question remaining from all of this talk of buses: what’s the speed of the connection between the various I/O controllers and buses and what not? Hopefully no one is running a 4GB/s bus to your expansion card, but a 1GB/s bus back to the rest of the system. I’d have a little trouble figuring out the point of that, probably.

Some links on the above topics:

• HP/Compaq’s PCI-X Industry Enablement Program
• PCI-X boosts bus bandwidth to 1 Gbps
• A page from an article presumably called “Server System Architecture - 2002” which helps give you a visualization of how many motherboards have their I/O buses put together
• In case you were unconvinced of the effect of putting a slow card on a fast bus, I/O Speeds Steam Ahead gives more information about PCI-X backwards compatibility
• I also got a lot of the PCIe information from the PCI Express entry on Wikipedia