Exchange on NetApp & Disk Layouts

It came up today, in a good way, as to how a customer who has a beast of a box (FAS3170 with two PAM2 cards and lots of shelving) and is migrating to Exchange 2007. How to lay out the aggregates? Whoah, WTF is an "aggregate"?
OK, reminder time. In NetApp you take (between 11 and 28 really) disks and you make a RAID Group. NetApp only do RAID-DP which is akin to RAID4 with an extra parity disk and akin to RAID6. RAID-DP has two parity disks and the parity is calculated differently for each parity disk; parity disk one is not a mirror of parity disk two. Then you take one, two or more RAID groups and lump them together into an aggregate. Rather than have one RAID group and have to restrict yourself to the IOPS inherent in that one RAID group the aggregate allows you to stripe all the data across all of the disks. Other vendors use a concept called Metaluns where they take small RAID groups and create LUNs then blend those LUNs together before they present them to a server.

In conventional designs you would take a RAID group for the logs and a RAID group for the databases. So far so good. Well, no. Look what you just did.
Logs:
You made a RAID group. Do you have enough spindles? Do you have too many? If you add a spindle you have to wait for the RAID group to re-balance. Time and processor demands! You can't take away disks so you're stuck with it. If your requirements change you're in for major surgery. One thing is for certain with today's disks. You just assigned a gargantuan number of gigabytes simply because you needed the spindles. That's a Royal PITA to explain to management.
Databases:
You've spent all your money on spindle provision for the logs and now are scrabbling around for the money to fund the capacity. Life is just a beach, ain't it.

NetApp and the single aggregate.
I say a single aggregate because I'm just going to talk about a single server which is sufficiently small right now. Size your requirements. You're going to come up with a number of GB (TB these days, obviously) and a number of IOPS. Provide the IOPS you need and see if you've met the capacity. If you meet that capacity within the number of disks required for IOPS then you're good. If you have more IOPS than required by meeting the capacity you're home. If you have the capacity but not the IOPS then you just add more disk until you do meet the IOPS. Yes, you're going to have the tired old "why did I just buy 20TB when I only 'need' 10TB of space" conversation (trust me, I have it more often than you so cope!). What all this means is that you're more than likely giving your transaction logs far, far more disks than they need so the performance is going to scream. Isn't that going to contend for I/O? Not the way that modern storage systems perform. Writes to NetApp are sent to NVRAM and thence to disk once the parities have been worked out and the optimum disk stripe has been calculated. Reads for Exchange 2007 are optimized anyway and reads from NetApp are better than from DAS systems because 1) there's probably a PAM card and 2) the data is probably lumped together and accessible from a fewer number of passes.

Very often the Exchange environment will need multiple aggregates because in the current (outgoing) version of the NetApp operating system the aggregate is limited to 16TB. What this will mean is that there's going to be an aggregate for stores one through (say) ten and another one for 11 up and so on. Alternatively you could have an aggregate for server one, another for server two and so on. It could go either way depending on your Exchange user-count-per-server parameters to name but one.

To ease troubleshooting keep an aggregate dedicated to a server. If something goes wrong you want to isolate things to one set of spindles and one server. If you do use multiple aggregates on a server (capacity requirements) don't put logs of A on 1 and logs of B on 2 and databases of A on 2 and databases of B on 1 (you get the message, right?) Even within the server keep the number of things you could potentially troubleshoot down to the minimum. If you do decide to go with separate logs and database aggregates then keep them on the same system controller. This, again, keeps you down to the minimum components to troubleshoot.

Reading? TR-3578 for Exchange 2007 and TR-3824 for Exchange 2010 are your friends.