August 19, 2005

PCI vs. PCI-X vs. PCI Express

This is part of a series of entries about building a new 2TB array. Or trying to, at least.

First thing I had to sort out was all of my I/O options.

  • PCI: my current array sits on three HPT302 ATA/133 controllers. All on the same PCI bus. This is not a special PCI bus. I suspect it is your usual 32-bit, 33MHz PCI bus. There are, apparently, 66MHz PCI buses, and I’m quite certain there are 64-bit buses. There are 5V PCI slots/devices and 3.3V PCI slots/buses. I suspect I have the later. Anyway, PCI bus bandwidth: in the realm of 132MB/s. Five drives, even at, say, 60MB/s each: 300MB/s. Not so good.

    (Of course, what’s even worse are the HPT302 quasi-open-source drivers. I think recent kernels don’t need them, but at the time of RH8.0 I certainly needed them. From time to time the drives “stall”—I just don’t know what else to call it. Everything blocks for 10-20s, then it comes back and all is right in the world. Plus it’s slow according to hdparm, but I don’t really trust hdparm.)

  • PCI-X: I’d say it’s PCI 2.0… but that’s really just the new version of PCI. Most everything you get these days in regular ol’ PCI is going to be PCI 2.2, I think. No, PCI-X is (in my words) an evolution of PCI. Cards and slots are both backwards compatible. PCI-X running at the same width and speed as PCI, is supposedly still faster than PCI.

    You’re most likely to find PCI-X to be 64-bits wide. I don’t think I’ve seen a current motherboard that has 32-bit PCI-X slots.

    PCI-X can run at 66MHz, 100MHz, or 133MHz. Technically, some newer version of PCI-X can run at 233MHz or even 533MHz, but I’ve never seen one faster than 133MHz.

    If you’re keeping up with math, you may have noticed some nice numbers: PCI-X 64-bit/100MHz = 800MB/s, PCI-X 64-bit/133MHz = 1GB/s. Thems fast. There are limits though:

    • A 133MHz bus can only support one device. A 100MHz bus can only support two devices. I believe a 66MHz bus can support four devices. On some motherboards I was looking at, things like the gigabit NIC would be on the same bus as one of the 64-bit/133MHz slots. So if you put a card in that slot, surprise! Your NIC and your device are both running at 100MHz now.

    • PCI-X is backwards compatible with PCI, and that goes for both slots and devices. This sounds nice, until you (almost immediately) realize that it means everything on a bus is going to run as slow as the slowest device. In reality, I read one source that said a PCI-X card on a PCI bus will always fall back to 33MHz-66MHz PCI isn’t an option for a PCI-X device. So, I’ve got three PCI PATA controllers I’d like to keep, and (for example) two new PCI-X SATA controllers I’d like to add. I need a board with at least one 64-bit/100MHz PCI-X bus with two slots (and just two slots on that bus), and then another three PCI slots on top of that. Hope you’ve got on-board NIC and video.

    The vast majority of motherboards I’ve seen don’t really seem to give me enough I/O options. I really want something insane like four PCI-X 133MHz buses with one slot each, and then a few more PCI slots. Maybe a PCIe (see below) or two. Oh, and under $200 too, please. I might go $250 if I have to.

  • PCI Express: not to be confused with PCI-X! Yes, they’re different. Genius naming guys. “Like, these guys couldn’t have devised names that sound just a little more unique than ‘PCI-X’ and ‘PCI Express’?”

    Whereas PCI-X is based on PCI, PCI Express (PCIe) is apparently quite different. Serial instead of parallel, and capable of several metric assloads more speed. Slots and devices are no where near backwards compatible AFAIK. Also, PCIe used to be called 3GIO, in case you run into that term. PCIe and 3GIO are the same thing.

    PCIe is usually specified as “x1,” “x8”, etc. This specifies the number of “lanes” that the slot offers. A lane is a bi-directional serial channel capable of around 250MB/s in each direction (sending and receiving). So an x16 slot should be 4GB/s in each direction. It took me a little while to realize this was in contrast to PCI, which goes in one direction at a time. So where your PCI-X 64-bit/133MHz slot has 1GB/s bandwidth, your 4x PCIe slot really has 2GB/s (1GB/s transmit, 1GB/s receive, simultaneously).

    I think PCIe also offers some other features, like hot swap. It seems poised to be “the expansion bus of the future.” Of course, the main class of peripherals I see using it currently are graphics cards. I suspect this is often why you find a motherboard with one PCIe 16x slot and one PCIe 1x slot: the 16x is for your video card (P.S.: AGP 8x does 2.1GB/s! Let me stick a storage controller in that slot, baby) and the 1x slot is to make you feel slightly more manly. Or maybe it’s for a GB NIC.

This study of buses was necessary for me to decide what kind of storage controllers I’ll be interested in. Accordingly, I plan my next installment to cover the various options one has for SATA controllers.

I do have one question remaining from all of this talk of buses: what’s the speed of the connection between the various I/O controllers and buses and what not? Hopefully no one is running a 4GB/s bus to your expansion card, but a 1GB/s bus back to the rest of the system. I’d have a little trouble figuring out the point of that, probably.

Some links on the above topics:

Comments (8)

  1. August 20, 2005
    Andy said...

    “A 133MHz bus can only support one device. A 100MHz bus can only support two devices. I believe a 66MHz bus can support four devices.” Hmm… Sounds like the VLB of the PCI world.

  2. November 22, 2006
    Gary Quiring said...

    The line “PCI-X is backwards compatible with PCI, and that goes for both slots and devices.” is not always true. I have a v20Z Sun server that uses PCI-X and it won’t physically fit a PCI card. The slot is backwards preventing them from fitting. PCI-X compatible cards fit fine.

    • September 8, 2013
      Romania II said...

      Yeah there is different amperage of pci cards. One is 3.3 Volts and the other is 5v. The 3.3 is compatible with the pci-x cards.

  3. December 4, 2006
    xmoe said...

    AFAIK – PCI-X is backwards compatible, except where it is 3.3V … you can have PCI cards that are 5V only … or ‘universal’ cards that should work in any PCI slot – that is 5V or 3.3V … I think the majority (90%) of PCI cards are universal, but then the devil is in the details and what you actually have.

    Also I would not characterize PCIe bandwidth as “2GBps” when its actually only 1GBps ‘full duplex’ .. certainly this has some nice performance boosts over PCI, but its not like you can push 2GBps in one direction … its simply like ethernet 1Gbps (bits – yes) full duplex … I think they should keep it like this for any other serial bus.

  4. June 19, 2008
    Nick said...

    This is a great article. I’d like to see another on PCI/PCI-X/PCI-E commpadibiliy. I’ve been researching for a year in preparation for building my own computer for about a year. I’ve found the perfect motherboard but it is running low on PCI slots. It was made for 3-way nVidia SLI so it has… 3x PCI-E x16 slots, 1x PCI-E x1 slot, and 2x PCI slots. So i really want 2 PCI-E x16 video cards, 1 PCI to USB PCI card, 1 PCI to wireless network card, and 1 PCI-X RAID 0/1 controller card. That means I have one spare PCI-E x16 slot left and need to put in my PCI-X RAID card. SO HEARS MY QUESTION!!! Can you put a PCI-X card into a PCI-E x16 slot? Please email me. again great article. Would like to see more.

  5. June 19, 2008
    darkness said...

    As far as I know, no, you can’t put PCI-X into a PCI-E slot of any type.

  6. August 20, 2009
    Jon said...

    There were some experimental 233mhz PCI-X cards made, but they found that any higher frequency was simply not going to work because the distance for the wave got too short.

    This is all kind of moot now, of course, since PCI-E has completely taken over and is currently working towards the 2.0 standard (3.0 apparently has been delayed until 2011 now)

  7. July 16, 2012
    PCBONEZ said...

    “A 133MHz bus can only support one device.” – Only true in the sense that PCI-E can only support one device. Just as with PCI-E, a board can have more than one PCI-X BUS. Supermicro X6DHE-XG2 is a great example.
    PCI and PCI-X cards/slots are always interchangeable -AS LONG AS- the voltages are compatible. Both the slots and the cards come in 3 flavors.
    | 5v only | 3.3v-and-5v ‘universal’ | 3.3v only. |
    The cards and slots are ‘keyed’ according to their compatible voltages. – So, as long as you aren’t mixing 3.3v only with 5v only it will work.
    PCI-X cards also work fine in standard PCI slots if there is enough clearance for the over-hang of the card but they will throttle down to that PCI BUS’s speed. [More correctly, the speed of the slowest card installed in that BUS.]
    This interchangeability is ~required~ by the PCI/PCI-X Standards, though not well publicized.
    PCI-X is still preferred by many that build servers because a 133MHz/64-bit PCI-X RAID card moves as much data as a 4x PCI-E RAID card but it costs around 25% as much. – Due to this PCI-X is still in use in servers as of socket LGA 1366. See Supermicro X8DTN+-LR.
    The 3Ware 9500 series was the last 3Ware series that was compatible with 5v slots. These work GREAT in standard 33Mhz PCI 32-bit slots as long as you have clearance and can deal with the slower transfer rate. These cards are now dirt cheap too. [12-ports for like $40.] Great way to get ‘real’ hardware RAID in that older equipment you have laying around or when PCI-E isn’t an option.