37.561% Garbage With added Exchange and Storage

SQL Server in VMDKs over NFS on NetApp

2012-01-09T15:33:00.005-05:00

I'm going to remove this when the full documentation set and some case study architectures are released but here's your starter for ten (University Challenge, look it up!)

FlexVol 1. Create one VMDK per SQL Server Instance. Put the temp DB from each Instance into its own VMDK in that single FlexVol. This is a FlexVol you certainly want on SAS, not SATA backed Flash Cache. You can also put VMDKs in that volume to host your guest page files. You will not be doing any snapshots of that volume whatsoever. ever. No really, never.

FlexVol 2. Create four VMDKs in that volume. One for the Logs for DBs in Full Recovery. One for the DBs, one for SNAPINFO and one for the System Databases. This is your minimum package of volumes per SQL Instance.

IF..... you want a Consistency Group across servers or even Instances on the same server you may put eight (2 x 4) VMDKs, or multiples thereof, into FlexVol 2. Use SnapCreator to generate the synchronized backups.

For the Operating System.

(1) FlexVol 3. Drop your golden OS VMDK into it and SIS Clone it for multiple servers (Use this for Hyper-V)

(2) FlexVol 3. Drop templates into the FlexVol and dedupe the hell out of it. Theoretically you can get over 200 Operating System / SQL Binaries into that one FlexVol and then drop it down so that it uses less space than one VMDK file.

This is based on the principle that all of your databases in a given SQL Instance have similar SLAs. If you have databases with a 15 minute SLA and some with one hour and some with four hour SLAs then you will want to use three SQL Instances. Whether that's one SQL Server name or three of them is entirely up to you.

The use of a database in a VMDK on a FlexVol does not move the granularity of backup; it's still the FlexVol. This isn't a NetApp restriction or unique problem, it's applicable to everyone. If you were doing this on DAS and MS DPM you'd still backup all the databases on the same disk at the same time because the level of granularity is the volume.

Obviously the NAS protocol provides a quantum leap in the flexibility but there are still some VSS/VDS APIs that you have to obey. Marrying one technology with another can make an old SQL hand need a double take but it's a massively popular solution so there are plenty of SQL DBAs and Admins who are completely down with it.

That's your basic end to end architecture. Please check with your local Virtualization or MS Consulting SE at NetApp because this could easily get out of date real quick. As at January 2012 it's good to go.

PowerShell FlexVol Creation & LUN Clone

2011-08-25T10:17:00.003-04:00

You're bored. You want to create something. You're creative.

See previous post about connecting to the controller.

Create the Volume:

New-NaVol -Name ma_test1 -Aggregate demo_aggr -Size 200G -SpaceReserve none -Controller $Filer -LanguageCode en_US

Create the LUN:

New-NaLun -Path /vol/ma_test1/na_test_lun1 -Size 100G -Controller $Filer -Unreserved

See previous post about the "$Filer" bit

Create a Snapshot:

New-NaSnapshot -TargetName -ma_test1 -Controller $Filer -SnapName snap_with_lun1

Great. So what?

You might want to clone the entire volume to a separate space because you want to do some work that might fill the volume up and you don't want to risk taking the LUN offline **

But you might just want to clone the LUN because you're just having a quick delve in to the snapshot to see some data or restore something you took a snapshot of at some previous time.

Clone the LUN:

New-NaLunClone -ClonePath /vol/ma_test1/ma_test_lun1_cl -ParentPath /vol/ma_test1/ma_test_lun1 -ParentSnapshot snap_with_lun1

** The point here is that you might be being careful about where you put data or the storage admin might not have given you the right to create a cloned volume. Perhaps the storage people want to keep you fairly cleanly inside your own box. A FlexClone creates a new volume whereas a LUN Clone creates a zero space clone inside the existing FlexVol.

Connect to your FAS / N-Series

2011-08-20T19:18:00.002-04:00

Spin up your PowerShell. do the import-module DataONTAP and then do this:

$FilerName = "Name"

$narootpasswd = "p@ssword"

################################

$password = ConvertTo-SecureString "p@ssword" -AsPlainText -Force

$Cred = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList "root",$password

$Filer = Connect-NaController $FilerName -Credential $Cred

There are a bunch of ways but this is easiest for me.

Long term FlexClone

2011-08-20T18:53:00.004-04:00

So you have your SQL database and you are keeping a few snapshots around, mirroring them to the DR site and vaulting them to another location for true backup protection.

Ah, but you want to take an SMSQL snapshot and then keep it around for a good while to do testing and/or development. Easy enough but what you need to consider is that you are going to lock a snapshot and it won't be able to be aged out of the production. Normally you'd have to call up the storage people and ask for the volume to be split. How's about you just do the job yourself and save everyone time. Obviously you need permissions on the controller (storage admin will provide easily enough and you need to be responsible for your own aggregate. Don't fill it up !!

Two PowerShell commandlets will FlexClone the snapshot so that you can start using it. Then, later, if you want to have that around on those disks for longer than your normal snapshot retention pattern you can execute the split. The FlexClone Split command is a background process so the LUN doesn't go offline ever, the underlying FlexVol just silently inflates on the aggregate.

New-NaVolClone -ParentVolume test1 -ParentSnapshot snapshot101 -CloneVolume testclone -Controller $Filer

Start-NaVolCloneSplit -name testclone -Controller $Filer

Windows patches and NetApp CIFS

2011-02-09T18:29:00.003-05:00

Another thing that comes up with predictable regularity is when ill-informed security professionals at a company engage people like eEye Digital Security (but by no means exclusive to them) to do vulnerability audits on their network. Shock, horror, oh the disaster, the nightmare, the sky is falling (you now I can do hyperbole and cliche's until the cows come home) and all of the NetApp controllers are insecure because they are vulnerable to MSnn-nnn and on and on and on. Well, not to worry. It's the audit software that is faulty and should be trained to properly recognize what it's interrogating for vulnerabilities.
If you want to see whether or not it is OK to apply a Windows patch to your PCs and whether or not vulnerabilities apply to any aspect of the NetApp platform you should simply click on: http://now.netapp.com/NOW/knowledge/docs/olio/MS_security/index.shtml. Obviously you will need your own NOW account to log on.

Exchange "Evolution"

2011-01-24T07:09:00.017-05:00

Thousands of years ago I rambled on about where MS had missed the boat a couple of times in their evolution of the flagship server product, but given what's happened with other products and that I seem to have pruned out some old posts the topic bears a redux and an update.
Ye Olde News (1):
Exchange 5.5: One 16GB store or one 'unlimited' store (not that large really since Windows couldn't cope in terms of either memory or raw storage and the size of backup tapes were very small, even in the large libraries. 50GB was about as large as many dared go) The RTO in hours of Exchange could never a be number less than three.
Exchange 2000: 20 stores
Exchange 2003: 20 stores (16GB for a while and then up to 75GB but Microsoft never wanted you above 40GB)
Exchange 2007: 50 'unlimited' stores. Unlimited here largely meant 50GB except CCR where 200GB was oft mooted. I can point to endless customers who wouldn't take it above 40GB.
Exchange 2010: 100 unlimited stores. Here is better in that 2TB is the number. Imagine if you were backing that up and didn't have snapshot protection to get it restored in a number less than five minutes.

Ye Olde News (2) - and some newish thoughts.......
Exchange 5.5: Single copy clustering and the number of people who couldn't keep a cluster up for more than five minutes was scary.
Exchange 2000: Ditto. In some cases more so! SAN a requirement because of the increased storage capability.
Exchange 2003: Same storage story, lots more stability. Exchange 2003 was more than stable enough for one copy and the availability requirements of most businesses. Those that deployed clusters never really got a huge uplift in availability because the failover always used to take a little while and, maddeningly, admins always wanted to fail it back for some wacky reason, mainly because one box was called mail01 and the other was called mail02. Heaven forbid that they run on mail02, regardless of the fact that the cluster was called mail.
Exchange 2007: This is where it all went wrong.
There was nothing wrong with the availability model in 2003 but because the people who looked after the storage in the Exchange 2003 days did something "sub-optimal" and because they did that sub-optimal thing on a platform that wasn't forgiving of sub-optimal and was badly configured so that it couldn't handle sub-optimal anyway, they decided to take the storage out of the equation as a potential source of trouble. Exchange 2007 brought in two cool sounding but ultimately ill-conceived things; LCR and CCR. Nobody used LCR and everybody wanted CCR because it was the new thing. Nobody really needed CCR and in fact few NEEDED it's sister; SCC. But because it was new the admins just had to have it. MCS consultants were pushing CCR like someone in North Philly pushes little plastic bags and their rock-like contents. Now, SCR was a pretty cool invention. 9/10 for innovation but 2/10 for execution. Storage vendors that had replication solutions that did not force a full resync after a site failure had nothing to worry about. SCR had its place but it was pretty niche. Then of course there was the fact that the Exchange PG and MCS tried to shift everyone to DAS right at the point that everyone with architectural vision wanted to virtualize and/or implement blades to reduce their data centre footprints. Good luck with DAS on a blade or VMware without doing something inefficient like putting a storage chassis into your blades. #fail. Suddenly you had three copies of the storage running on three active servers when only a couple of years ago there was only one active server and some storage in the DR site. You went from a decent level of storage efficiency down to 33% or less. Remember that you also changed RAID types from RAID5 down to RAID1+0 (0+1) (10) etc. so there were more disks for less overall data. Overall efficiency is now probably in the 20% area compared to Exchange 2003.

Having just seen the industry fly off at a tangent to their own world view the Exchange product group looked at Exchange 2010 and decided what they had done in 2007 wasn't nearly enough. Firstly they improved the IO again and then said to use SATA rather than SAS (three versions of the product, three disk types). That wasn't so bad but what came next was, err, umm, interesting. Not content with ramping up from one copy to three active copies from 2003 to 2007, Microsoft went from three copies to four copies in 2010. The original idea was for that fourth copy to be an hour out-of-date and have logic to bring things up to date as and when necessary. Papers came out citing a fifth copy that was even further (a day, two, etc) out of date in order to eliminate backup. It's a credit that they have now* stepped back a little bit but I assure you the number of customers who are planning a fourth copy where there is zero need is amazing. Luckily I am usually able to talk them down from that particular cloud (ahh, cloud, that reminds me - more later) So where n=1 with Exchange 2003, n=4 with Exchange 2010 (25% efficiency) - except that it wasn't. As part of the IO reduction effort the database was flattened and single instancing had to go, although compression did come in. With a lagged copy you were into 20% efficient on storage and 25 to 33% efficient on servers.

So, where are we?
Exchange 5.5 was DAS because SANs were expensive.
Exchange 2000 and 2003 was SAN because the storage requirements of Exchange exceeded the available DAS chassis availability in a lot of cases.
Exchange 2007 was suddenly local SAS because MS wanted nothing other than to eliminate the SAN because they couldn't manage one particular model themselves and assumed that everyone else was less competent than them.**
Exchange 2010 was suddenly local SATA rather than SAS because MS made some serious improvements to the IO model.
Pro DAS Exchange 2007 was released at the point where many customers were looking at virtualization. Exchange 2007 single-handedly delayed the tipping point of virtualization. It is perhaps no coincidence that Microsoft did not have a production-grade virtualization product at the time and didn't really know enough about their own product to accurately predict the IO profile in a virtual environment.
Pro DAS Exchange 2010 was released at the tipping point to virtualization but because Microsoft had missed the boat on Hyper-V they still didn't want you operating a virtual environment because that meant putting money into the hands of VMware. Same blocking factor, different day. Only this time people aren't wearing it. Virtual environments are winning in the dynamic data centre and centralized storage is back where it should be; efficiently managing the integrity of customer data.

What's coming?
What's coming might be what's happening with other Microsoft products. The SharePoint people have hit an absolute home-run with 2010 and the RBS. Having a small database containing metadata and a big old BLOB store on a file share is the next big thing. In fact, it's the now, not the next. But there's a problem. If Exchange 20xx did this, right now, the use of DAS is useless. The use of Windows file servers is useless (Windows & SMB (CIFS) is not an optimal file-serving platform). What is SharePoint looking like in the SAN world? It's small databases stored on the SAN with the disk metadata on SSD based victim-cache. It's data files on NAS and the disk metadata on that same SSD. It's database and data files on SATA disk, SAS at worst. It's high density, high efficiency, highly performing storage. Exchange 2010 is this thoroughly annoying step-child in the corner that everyone hates. The data centre people hate it because of the footprint. The server teams hate it because they have to manage physical resources in a time where most of their assets are virtual guests. The storage people hate them because they are lowering the ROI of the SAN and increasing overall TCO for storage and support across the board. The finance people hate the Exchange people because the Exchange people blindly follow what Microsoft tell them and insist on three or four copies and bloat the cost of their environment. It's a hard life being an Exchange operations guy in 2011 when you don't see the writing on the wall because there are too many of your own servers in between you and the wall!

To summarise.
Exchange had it almost right with Exchange 2003. They went just that little bit too far with 2007 and missed virtualization and the new data centre. They compounded that problem with 2010 and hurt their customers bottom lines, their own DPM and Hyper-V product groups and their own credibility. They have positioned themselves at the opposite pole to the coming raging success within Microsoft; SharePoint.

The bottom line here is that anyone who looks back more than one year into the past should see what a success the Exchange product group has had at seeing the trends and exploiting storage capabilities. Right about now the numbers you are looking for are (1) 408 822 6000. (2) 866 438 3622 (3) 877 426 2233.

*edited 'not' for 'now' - my bad.
** I will point out that not having a SAN in use with such high turnover environments such as Windows / Exchange builds and dog-fooding is not a bad thing. HBA drivers and firmware are all on a particular support matrix for Windows builds. If there's a version of SP of Windows or Exchange on the box there could very well be problems until the HBA vendor updates drivers or firmware. The same applies, of course, to system boards, disks and chip sets but to a lesser and more manageable degree.

SMSQL Verification on Cluster

2011-01-21T09:08:00.003-05:00

Is now possible, but you have to do some work up-front.
Previously you would have to have a workstation, standalone server or a separate non-clustered SQL Instance on a clustered SQL Server in order to mount the FlexClone and conduct the automated DBCCCHECKDB and verify the backup.
What you can now do is to specify the path to where the verification is to take place and if that path is a folder (which becomes a mount point) on a clustered LUN you're all good to go. You can use the same process for FlexClone restores and (LUN) copy restores. That last part is for where you have a bunch of LUNs in a single FlexVol and only want to do a restore of the database(s) in one of the LUNs.

What some customers also used to do was to manually "connect" to the snapshot, creating it as a shared LUN then do the verification process manually. That required quite a lot more thought than simply using a non-clustered box.

As always you are best off having SMSQL5.1 and SDW6.3 - Both have recent P releases so check with communities.netapp.com for the current release.

Many thanks to colleague Alan Parker for drawing our attention to that gem.

Not free lunch but..........

2011-01-02T15:52:00.002-05:00

You could find yourself heading off to lunch on time........
Quest have released something free:
http://www.open-a-socket.com/index.php/2010/12/23/gal-sync-with-quest-quick-connect-express-for-active-directory/
http://www.quest.com/activeroles-server/quickconnect-express-for-active-directory.aspx

Because as we all know, the first one is always free and you're sure to go back to Quest for something you'll need to pay for.

More Microsoft Pro-DAS Jihadist FUD (and a counter)

2010-11-22T15:25:00.003-05:00

Mr Perry Clarke has been at it again. http://blogs.technet.com/b/perryclarke/archive/2010/11/16/responding-to-migration-vs-in-place-upgrade-comments.aspx is his latest pro DAS, anti Virtualization and anti networked storage communique. This follows on from: http://blogs.technet.com/b/perryclarke/archive/2010/04/14/storage-performance-and-my-take-on-virtual-storage.aspx which has been taken apart in a number of forums over the past few months so I won't bother doing so here. It's getting to be depressingly routine but so long as he keeps saying things I get to talk to customers to tell them why you don't do something "just because" Microsoft tells you to and "just because" a Microsoft employee proffers an opinion.

If you want a strategic solution you put enterprise networked storage into your data centers. Depending on what your business requirements are you implement such a solution from one or more vendors. EMC and NetApp will tell you, it's all about your business and what business problem you are trying to solve and what service you are trying to deliver to the rest of the business to enable them to work more efficiently and with greater productivity. You don't abandon a strategy "just because" an employee at a company who doesn't do storage says one of your applications should be done the way he likes. Thinking in a silo'd manner is several years out of date and the Microsoft Exchange product group do themselves a gross disservice by continuing to do so. Please, if you see something from them that says anything other than "large, low cost mailboxes" or you encounter a sales rep or MCS consultant that says anything else or recommends DAS please email me with their name and you can be hooked up with the official line, direct from Microsoft. (I won't even get involved!)

Similarly virtualization is a solution to a business problem. Do it or don't do it, it doesn't really matter to me as an applications professional working for a storage company. But if you do implement it there are some things you are going to want to think about. If you do implement a virtualization platform there's a good chance that your requirement for a highly available infrastructure didn't just go away. There's every chance you want both. There's every chance you want to have highly available guest servers as well as a highly utilized host server environment. So would you put the data onto disks internal to the host? Sure, if your guests had no high availability requirement. But only if. If you need that data highly available you're going to store it on a networked platform. Of course, if you have two hosts and put a DAG guest on each and a load of disks you're getting that HA but you still don't get storage efficiency and lost flexibility as well as being constrained into buying hardware that can take lots of internal disk. And who in the enterprise is buying big iron servers for little silo jobs these days? If you have a couple of servers and that's it then yes DASDASDAS. If you need a dozen or more server names and isolated roles then a blade farm is going to be an attractive option to you. Again, more with the business requirements and strategic direction. Good luck with running Exchange and DAS in a Hyper-V blade host.

NetApp commissioned and released a document:
http://media.netapp.com/documents/esg-wp-netapp-exchange-storage-efficiency.pdf on, as the file name suggests, storage efficiency. It talks about what to look at when calculating the TCO of an Exchange environment and little of it has anything to do with the data inside the EDB files.

Remember what a showcase from Microsoft is. Remember what it is not. It is a "how we did". It is a "how we, with unlimited resources and an inability to use anything other than Microsoft software solved a business problem". It is not a reference architecture. It is not a prescriptive set of steps to follow to generate the perfect environment. If you or your staff have printed out a showcase then you are already in trouble and your business is ready to spend more than it likely needs to.

Bottom line? Think about it. Microsoft release a good application with their Exchange product but those people aren't paid to look beyond the application and you would do yourself a favour if you put anything they said to one side of your desk and got the other side of the story from your infrastructure architects.

Nice Sh*t-storm

2010-11-16T15:38:00.003-05:00

The blog about 'misleading advice' http://msexchangeteam.com/archive/2010/11/09/456851.aspx seems to have generated way more comments than I thought it would. As usual there are the kool-aid drunkards who leap to the defence of Microsoft on every little thing and who seem to have weaved in SAN=BAD, possibly because there is no understanding of the true arguments here.

VMware have responded in a candid and balanced manner: http://blogs.vmware.com/virtualreality/2010/11/virtualizing-exchange-on-vmware.html
As you can see, VMware HA is not VMotion or DRS but many people wrap all of the capabilities into one colloquial term; it's human nature.

I don't know who 'Alagba' is but he makes some pretty cutting remarks about the capabilities of the Hyper-V solution that Microsoft has; i.e. it can't really do VMotion like activities and that might be why it's not supported. Live Migration does in fact exist but it's less mature, significantly so.

Key points from the comments. Microsoft fanatical (can't argue). Microsoft schizophrenic (can't argue) Jim Lucey backpeddaling somewhat. Microsoft spiteful (jannie h) (might be true but well, it's a bit strong.

I'm learning quite a bit from these Exchanges as it helps me in my job counter all of the must-use-DAS that some Microsoft field staff still maintain.

One hopes that Jim's post can be safely buried and forgotten about since it's not credible.

Microsoft, DAG, VMotion, Crankiness, and stuff.

2010-11-15T09:29:00.003-05:00

VMware recently released their best practice advice for Exchange 2010 and VSphere. The link is here: http://www.vmware.com/files/pdf/Exchange_2010_on_VMware_-_Best_Practices_Guide.pdf It's all very sensible advice that links out to and uses Microsoft best practices all the way through it. Unfortunately someone at Microsoft had a problem with the documentation and the Product group have launched forth with a long old rant: http://msexchangeteam.com/archive/2010/11/09/456851.aspx that bears little resemblance to reality. Some of the comments have tried to get MS honest on it, others have just swigged another mouthful of Kool-Aid and supported everything that was written.

So folks, the VMware document is good. The Microsoft blog post is revealing but, as you can read on page 64 of the VMware document, is calling out misleading advice that isn't actually present as advice in the first place. Please read the VMware document carefully. Once you have properly read the document there is no way on earth that you will ever think you could VMotion a running DAG member. Really. Honestly.

Server Side File Indexing

2010-10-29T11:20:00.004-04:00

With regard to Windows Search on servers. One particular question regularly comes up, driven by a customer query. It goes like this.
The customer has seen a TechNet presentation and seen a link or two: http://biztechmagazine.com/article.asp?item_id=416.
Then the customer wants to do it against the shares he has hosted on NetApp CIFS volumes.
What the customer simply didn't read is that, on the Windows feature setup screen, Microsoft specifically call out the fact that this feature is NOT FOR THE ENTERPRISE.
NetApp storage controllers host untold millions of files. They serve the files out to tens of thousands of clients. If indexing a 'handful' of files on a Windows server is too much for Microsoft to recommend for customers just think of the challenges it would pose for NetApp and EMC Celerra systems.

So if you see it as a feature, read the warnings on the Windows screen in front of you. If you are in an enterprise and need to remove your socks to keep track of how many users are accessing the file server you don't want to implement the indexing feature on the server.

There are many client-side features where you can archive server data - the Library to name but one.

NetApp Snapshots & Exchange 2010 -Which Copy?

2010-10-16T09:34:00.005-04:00

A vendor-and-partner email conversation with Devin Ganger (http://www.thecabal.org/) in his role as Exchange Grand Poobah at Trace3 (http://www.trace3.com/) about Snapshots brought a few things to light. He has a customer who wants to take backups of the active Exchange 2010 database and then spool the data off, using NetBackup (NBU) to DataDomain. Well, that's as simple as falling off a log. SME to snap the stores, FlexClone the snap, mount it to the NetBackup server, drag the data off and then destroy the FlexClone. Done and done. Everyone home in time for Vimto and NAAFI sandwiches.

Not so fast. What happens if the active copy changes from node1 to node2? No problem. SME gives you three options. (1) backup the stores on a specific server. (2) backup the active or passive copies. (3) backup a specific activation preference. So in this particular case you use SME to connect to the DAG rather than the node which brings up a screen with those three options. Specify the active and walk the rest of the way through the wizard. What SME does is create a job on all the servers in the DAG that have a copy of the stores you selected.

That's all very easy but Devin had some extra requirements. He wanted to verify the snapshot and get it to the DD box with NBU. There are a few ways to achieve this. SME has an after-process capability to kick off something else such as an alert to some other system or, as in this case, telling a different server to do something. In this case SME will run the snap and its last act will be to instruct the verification server to kick off the scheduled verification task of the most recent snapshot. That's going to do its thing and then the 2nd to last act of that job, before it tears down the FlexClone is to stream the flat file that constitutes the snapshot off to DD.

It's true there are quite a few steps in that process but they can be integrated if you want or left isolated. None of it is rocket science and it's all PowerShell or SDCLI both of which are readable in pretty much plain English.

What I didn't point out but possibly should have done is the extra IO that's being generated. It might not matter because there may be plenty of spindles in the aggregate to handle the load, especially if the customer is using legacy FC disks following an upgrade from (say) 2003 and the disks are still in support but the verification is going to consume disk IO and the transfer to DD is going to prolong it. Unless you have a business requirement it might be a better idea to snap the passive or a particular preference and then to the verification and transfer on the passive or another copy attached to a different aggregate.

Doom thy Windows File Servers

2010-10-05T10:07:00.001-04:00

With SMB2, Windows 7 and NetApp storage. Take a look at the statistics in:
http://media.netapp.com/documents/tr-3869.pdf.
Shiny

NetApp 7.3.x Simulator - Squeezing Space

2010-10-02T10:55:00.003-04:00

The default installation of the NetApp simulator creates you three 100MB disks as aggr0 and puts vol0 into it. You can have up to 56 1GB disks in the sim which isn't many but it's the sim, there aren't supposed to be enterprise grade applications on it. You might want to add that little bit more space, all three gigabytes of it. Here's what you do. It's here more for my own reference because I run about five of these things at home because I really don't want to pay $200 a month for a FAS2040 electricity bill and still not have SnapMirror and SnapVault between system capabilities.

Stuff you need to do:
Edit /sim/setup.sh to change maxdisks to 56. (gedit is fine)
Set up filer as per instructions, make disks 56. Paul Hargreaves wrote comprehensive instructions so I won't bother here.
Create a new aggregate.
Create a new volume 'vol1'.
License all the extras from the license.htm file - including SnapMirror.
Configure SnapMirror to accept connections from all the sims you need.
Configure SnapMirror.
Initialize the SnapMirror and note the rejection works properly.
Restrict vol1.
SnapMirror the volume which will work properly this time.
Wait for mirroring.
Quiece the mirror.
Break the mirror.
Run: vol options vol1 root.
Reboot the filer.
Remove vol0.
Rename vol1 to vol0.
Destroy aggr0.
Remove the three small disks that used to be aggr0.
Shut down the sim.
Delete the three 100MB disks from the /sim/disks directory and also the 'reservations file'.
Boot the sim. (just run bash runsim.sh from the terminal window)
Make sure all is well and only 53 disks exist.
Shut down the sim.
Run setup.sh again and add the three disks.
Start the sim.
You're done. You have 56, 1GB disks.

Exchange 2010 Single Item Recovery

2010-09-21T15:51:00.005-04:00

Today, again, a customer went away from a call feeling that the people they had been speaking to previously were doing a smoke-and-mirrors job on them. The topic in question was single item recovery in Exchange 2010 (http://msexchangeteam.com/archive/2009/09/25/452632.aspx) and the ability for individual users to get their hands on retained mail and what implementing the feature means for your servers. Taken at its simplest:

Implement SIR and your store sizes increase. Implement SIR and, because the storage isn't in an 'archived' state, the server resources are commensurately higher. Either you need more servers or you need more stores or you need more disk. Implement SIR and the users still cannot do their own restores. They need assistance from IT services staff and that means staff beyond first line 'ticket crew' type people. Implement SIR and you have significantly more storage to cope* with. Larger/more stores and maybe online background maintenance 24/7 means more IOPS. To summarize, implement SIR and the already problematic Exchange 2010 DAG storage efficiency takes another hefty knock.

As usual there's an alternative. Snapshots are here to help you. If you keep your primary site copies nice and minimal - and you've seen how deduplication helps a lot there - you can replicate nice, small (relatively speaking!) databases to your secondary site and retain long term snapshots at that location. Using far less storage allows for store-size recoveries to be carried out quickly but that's the secondary result. Using the money saved on your storage you can re-invest a small part of it towards a single item recovery tool that can connect to your snapshots and drag single messages, folders or mailboxes back from what was previously considered to be oblivion.

Ohhh, I can hear it now. OMGOMGOMG, more software. Money. Spend. Microsoft gets me that for free. I think I've explained above that it's far from free so go re-read what I just typed, just to make sure. Once you've done that you then take a look at the soft costs. If my front office team are using a GUI to do single restores it's instantly down-skilled so you are going to employ fewer 3rd line admins, need fewer people at your outsourcing organization leveraged to you or alternatively you can increase your technology offerings to the business. The cost of the products and support is a tenth to a fifth of the annual cost of an admin so you've hit ROI inside three months and reduced the time a case is open by an order of magnitude.

So implementing snapshots isn't just about being able to restore a store that, frankly, is probably never going to be necessary. What snapshots mean to you is flexibility; Efficient and reduced storage footprints. The option to down-skill mundane tasks. There is nothing that anyone can come up with that does not lead you away from the fact that snapshots make business sense. The cool technical stuff is just the sauce base on which the good stuff sits.

Go for it, bother your local NetApp rep. Tell him the wastrel from Philly sent ya!

*Note "cope" does not necessarily mean backup/restore; it simply means cope with, as in monitor, maintain, look-after.

Exchange 2010 - The Complete Solution

2010-09-20T08:41:00.004-04:00

There you sit, An IT Professional with an interest in Microsoft Exchange. You have this old Exchange 2003 architecture and you look longingly at all your drinking buddies who have a nice corner of a SAN and lots of copies. They're still not happy because they keep getting beat up by their managers because they blew the budget on servers and storage. They keep getting beat up by their virtualization team because they're still a bit legacy and were too afraid to put the Exchange on a hypervisor. Basically they may well be crowing down the pub but it's the Triumph ale talking because they're not much better off. That's why they go there during happy hour because it's $3 rather than $5 a pint at all other times.

What's a guy to do? Well, put it all together.

1. Take your Exchange server plans and give them to the virtualization team. If you have (say) 10 servers at 20GB of C drive each they're going to be able to give you those same servers, increase their disk performance and do it all in less than the space for one copy; 18/20GB rather than 200GB of spinning.

2. Get your storage guys to co-locate the LUNs for DAGStore1 in the same FlexVol. Co-locate the LUNs for DAGStore2 in a new FlexVol and so forth. Allow your storage guys to deduplicate them on condition that they load the disk metadata into FlashCache (also here).

What have you done?
1. The virtualization guys owe you a beer because they increased their influence.
2. The storage guys owe you a beer because you didn't consume tons of unnecessary disk and floor-tiles.
3. You owe you a beer because you have a great 2+1 DAG.
4. The owe you another beer because you still 'own' the DR plan and don't need to worry about VMotion and SRM.
5. The VMware guys owe you a second beer because Exchange DR isn't their problem.
6. The business owes you a beer because you dun good.

Six beers. That's a decent night out. Make it a Friday or a Saturday though so you're clear-headed for work.

Lagged Exchange 2010 Databases - Just Say No

2010-09-03T09:35:00.006-04:00

It's time to put some recently risen silliness to rest. When Exchange 2010 was fresh out of the box only the most experienced administrators were on the case and not a single one of them in nearly a year brought up the topic of whether they should lag an Exchange 2010 database copy their NetApp storage. They all saw that there was just no point to it and the storage platform could handle things so much better. That relatively pain-free honeymoon didn't last.

But now Service Pack 1 is in the wild the product is getting the attention of the next tier of Exchange IT professionals and it's time to answer the question on why there is no point to a lagged database copy. There have been three instances lately where the customer has stated to the Microsoft (employee) consultant that the upgrade WILL be done to NetApp storage and for that to be a central part of the planning..... So, why on earth do they still shove the lagged copy down customer's throats? They simply have no idea about the storage because they think their job is just to make the application work. Well, what is it that the Microsoft people don't get?

Two things; Snapshots and 'frequent recovery points'. Available on all storage platforms to a greater or lesser degree and by various names.....

Background. You can lag an Exchange 2010 database so that it can be as far as a fortnight out of date so long as it has all of the logs to play itself up to date. If it doesn't have the logs you will find that there is another server with the logs that the lagged copy needs. Leaving the incoming logistics aside the bottom line is that the lagged copy has a shed load of logs sat on it, waiting to be played into the store as and when instructed.

When you decide that it's all gone Pete Tong and you need to activate your lagged copy there is going to be a short period of time to wait before you're up and running. Not so bad really.

But hold on. What if you had a competent Shapshot enabled storage solution? Would you need the lag? If you were taking snaps of the stores every hour, four hours whatever, and were taking snaps of the logs every 5, 10, 15 minutes or so in between.

So what, I want my Lag. It is the one with the bigger, err, GBs. Cos Microsoft told me so. Well, no, not so much really.

A ton of logs on a given server takes up a lot of space. It is an inherent risk. If you follow the guidelines from Microsoft you're not going to back anything up in your Exchange environment so those logs are all that stand between you and a P45/pink slip when the Visigoths are sighted.

The other thing is that, usually, that copy is going to be copy four rather than three so you're inherently adding 25+n% more storage than you really needed.

And as if that wasn't enough you can only restore to the last log file unless you roll up the sleeves and get activating... I've never been one for shying away from the command line but if someone has gone to all the trouble of coming up with a GUI that does it for you and that GUI manages exactly when you want the database to abandon playing logs and mount, you should probably pay them the compliment and use it.

Our customers get to save themselves the extra storage and they get to have a GUI that takes snapshots, does database restores, does zero-space clones to test whether the thing you think want to restore is indeed the thing you need to restore and does that restore for you to the point in time you have told it to.

All very cunning really and very space efficient. Map that with my previous post and the thing that Microsoft tell you is going to take 400+GB for every 100GB of email is changed. Now you've just got yourself a multiple copy solution for Exchange that takes about 100GB per site for every 100GB of email.

Oh but, oohhhhh, ahhhh, noooo, SAN bad. SAN expensive. SAN complex. Backups bad. Whatever guys.

Exchange 2010 & NetApp Deduplication (2)

2010-08-27T17:04:00.005-04:00

Following on my from ramble at: http://markarnold.blogspot.com/2010/03/exchange-2010-netapp-deduplication.html there is more.....

Early on in the thought process the customers were keen to retain one copy of the storage on one set of disks and another set of storage on an entirely different set of disks. This met the Exchange requirement of having a two copy DAG in production before consideration was given to the DR site. However, the storage people really thought the Exchange people were, to put it charitably, somewhat excessive in their storage requirement. How could the Exchange people be satisfied that they had their two copies and the storage people be satisfied that they were getting the best use out of their highly resilient storage platform.

Some bright soul had a plan (there's a clue for you). Let's fool the Exchange guys because, being one, I know they're far easier to fool than the storage boys. Take two servers. Create a FlexVol and put a LUN in it. Create a database. Get the users to load up the store with normal data. Deduplicate it. As you've seen you're looking at between 10 and 25% saving in physical, consumed, space. My home lab is sat at around 13% which is typical for a store that receives emails with body but few attachments.
Now for the next part.....
Set up a replica. You now have two LUNs. Put those two LUNs into the same FlexVol. Run deduplication.

The picture is just a screenshot from System Manager showing that the two copies as far as Windows is concerned, is only taking the physical space of one.

Ohh, I can hear you now. Single point of failure. Sky falling. Visigoths. Barbarians. End of days. Not at all. Keep calm and carry on. It's all on RAID-DP. Two disks can fail and nothing happens. And anyway, if you're doing a 2+1 there's still that 3rd copy in the remote location.

Exchange 2010 SP1 & SnapManager 6

2010-08-25T16:13:00.003-04:00

Speaking as a private individual whilst conscious of being a NetApp employee....

As you all know SP1 is out on the streets. I won't post a link to it because, frankly, I really can't be bothered since it's late in the day and everyone else has posted links to links, blogged about blogs, faceyspaced, tweeted and retweeted themselves beyond the will to live.
SnapManager 6 for Exchange is under test with SP1 and you will hear official news in due course but testing in the lab here at home has so far gone without hitch. Standalone servers, DAG members, restores, single mailbox recovery and everything has gone without a single error. Everything that should have got purged did and those logs that needed to stick around for whatever (legitimate) reason, did.

Keep an eye on the Interoperability Matrix for supportability. Contact your local SE for news in about late October if you haven't heard official news.

Exchange 2010 Virtualization Webcast

2010-08-08T14:20:00.003-04:00

NetApp, like everyone else, is running Exchange 2010 webcasts. The topic here: http://communicate.netapp.com/forms/verify?seminarID=20100729WOD&REF_SOURCE=TTsite was of virtualization.
Take a look at the poling answers:
How likely is it that your organization will virtualize your Exchange 2010 server environment? (378 votes)
a. We have already virtualized Exchange 2010 7%
b. Very likely 64%
c. Somewhat likely 22%
d. Not very likely 5%
e. Not likely at all 2%

That's some encouraging numbers. The webcast was hypervisor agnostic so there's data on which virtualization was favoured.

Snapvault - It's all about the "hyphen"

2010-06-21T12:01:00.005-04:00

As part of the ongoing quest for filling #superphastmopho with NetApp simulators and Exchange servers I decided that SnapVault as well as SnapMirror would be something great to implement. Why? Goodness knows why. Because it's fun perhaps.
Anyway, there is no GUI for SnapVault at all. It's all command line. I spent a long time messing around with it because I just couldn't manage to punch the right words into NOW to get the necessary syntax out. So here's whatchadoo to get it all working.

Make sure you're licensed. The source is the primary and the target is the secondary.
Create a volume on the secondary
On the primary execute options snapvault.access and options snapvault.enable
On the secondary execute your equivalent of this:
snapvault start source:/vol/vol_name/- target:/vol/vol_name/qtree (note that the /- is essential and the qtree should not actually exist already.
If you don't remember the /- you're going to be troubleshooting for a long time. Right up until the point you remove the destinations and start again.

In case you too are also having search problems, this: http://now.netapp.com/NOW/knowledge/docs/ontap/rel733/html/ontap/onlinebk/GUID-67A4A9AB-DBD5-4040-B52B-B62EB06F2BBD.html is what you need.

IPv6 and Exchange 2010 - Exterminate

2010-05-25T20:58:00.003-04:00

1) Yeah, I keep losing the links. 2) No Daleks were hurt during the making of this post.

Exchange 2010 Pre-requs and getting rid of Exchange 2010
Pre-reqs:
http://msunified.net/2009/10/30/exchange-2010-prerequisites-on-server-2008-r2/

KB Article:
http://support.microsoft.com/kb/929852/en-us

Exchange 2010 on Windows Server 2008 R2 is slightly different than on Windows Server 2008 in that No IPv6 is required at all, right from the get-go.

DPM 2010 and SharePoint

2010-05-25T08:56:00.002-04:00

DPM 2010 is looking fairly interesting with SharePoint 2010.
Now, with SharePoint 2007 you needed a separate recovery server so that made three boxes involved in backups. (your running environment, your DPM and then your recovery farm). That's a lot of processor, memory and storage.
With SharePoint 2010 you don't need that extra recovery server. However, it's not all plain sailing. You do need all that DPM space: http://blogs.msdn.com/b/russmax/archive/2009/10/21/sharepoint-2010-granular-backup-restore-part-1.aspx and http://blogs.msdn.com/b/russmax/archive/2009/10/21/sharepoint-2010-granular-backup-restore-part-2.aspx you do have to use a lot of storage in the first place.
There's also some interesting stuff at www.microsoft.com/systemcenter/en/us/data-protection-manager/dpm-2010-overview.aspx so you can see all those steps that you need to go through.

It all begs the question as to why Microsoft still haven't got around to an AvePoint-like solution to do all this in a couple of clicks.

Exchange on NetApp & Disk Layouts

2010-05-18T16:24:00.004-04:00

It came up today, in a good way, as to how a customer who has a beast of a box (FAS3170 with two PAM2 cards and lots of shelving) and is migrating to Exchange 2007. How to lay out the aggregates? Whoah, WTF is an "aggregate"?
OK, reminder time. In NetApp you take (between 11 and 28 really) disks and you make a RAID Group. NetApp only do RAID-DP which is akin to RAID4 with an extra parity disk and akin to RAID6. RAID-DP has two parity disks and the parity is calculated differently for each parity disk; parity disk one is not a mirror of parity disk two. Then you take one, two or more RAID groups and lump them together into an aggregate. Rather than have one RAID group and have to restrict yourself to the IOPS inherent in that one RAID group the aggregate allows you to stripe all the data across all of the disks. Other vendors use a concept called Metaluns where they take small RAID groups and create LUNs then blend those LUNs together before they present them to a server.

In conventional designs you would take a RAID group for the logs and a RAID group for the databases. So far so good. Well, no. Look what you just did.
Logs:
You made a RAID group. Do you have enough spindles? Do you have too many? If you add a spindle you have to wait for the RAID group to re-balance. Time and processor demands! You can't take away disks so you're stuck with it. If your requirements change you're in for major surgery. One thing is for certain with today's disks. You just assigned a gargantuan number of gigabytes simply because you needed the spindles. That's a Royal PITA to explain to management.
Databases:
You've spent all your money on spindle provision for the logs and now are scrabbling around for the money to fund the capacity. Life is just a beach, ain't it.

NetApp and the single aggregate.
I say a single aggregate because I'm just going to talk about a single server which is sufficiently small right now. Size your requirements. You're going to come up with a number of GB (TB these days, obviously) and a number of IOPS. Provide the IOPS you need and see if you've met the capacity. If you meet that capacity within the number of disks required for IOPS then you're good. If you have more IOPS than required by meeting the capacity you're home. If you have the capacity but not the IOPS then you just add more disk until you do meet the IOPS. Yes, you're going to have the tired old "why did I just buy 20TB when I only 'need' 10TB of space" conversation (trust me, I have it more often than you so cope!). What all this means is that you're more than likely giving your transaction logs far, far more disks than they need so the performance is going to scream. Isn't that going to contend for I/O? Not the way that modern storage systems perform. Writes to NetApp are sent to NVRAM and thence to disk once the parities have been worked out and the optimum disk stripe has been calculated. Reads for Exchange 2007 are optimized anyway and reads from NetApp are better than from DAS systems because 1) there's probably a PAM card and 2) the data is probably lumped together and accessible from a fewer number of passes.

Very often the Exchange environment will need multiple aggregates because in the current (outgoing) version of the NetApp operating system the aggregate is limited to 16TB. What this will mean is that there's going to be an aggregate for stores one through (say) ten and another one for 11 up and so on. Alternatively you could have an aggregate for server one, another for server two and so on. It could go either way depending on your Exchange user-count-per-server parameters to name but one.

To ease troubleshooting keep an aggregate dedicated to a server. If something goes wrong you want to isolate things to one set of spindles and one server. If you do use multiple aggregates on a server (capacity requirements) don't put logs of A on 1 and logs of B on 2 and databases of A on 2 and databases of B on 1 (you get the message, right?) Even within the server keep the number of things you could potentially troubleshoot down to the minimum. If you do decide to go with separate logs and database aggregates then keep them on the same system controller. This, again, keeps you down to the minimum components to troubleshoot.

Reading? TR-3578 for Exchange 2007 and TR-3824 for Exchange 2010 are your friends.

Interesting Google Apps / Exchange Online Post

2010-05-18T14:04:00.003-04:00

Curses. Comments are disabled on the blog due to the platform migration. I'll ask my questions when they come back online but in the meantime, here's my thoughts and what I don't understand because it hasn't been explained.

The company made the decision last year to go with Google Apps. All well and good. Despite being an Exchange MVP I don't, today, have a horse in that race because I also work for a storage vendor who stands to make a very tidy sum from people outsourcing services to the cloud.

Now the company has made the decision to go with Exchange Online services.

Holy heck Batman. After a year? Lemme take a look and see what's going on. At first I thought that Google trumpeted a sale rather than an agreement to do a trial. Nope. The company pulled the trigger and said as much.

1. June 11 2009 the Senior Manager of Global IT and his sysadmin posted on Google.
2. May 18 2010 the Director of IT posted via the Technet, Online Services blog.
Ahh, interesting. A power play?

An on-premise Exchange environment for 800 people will not cost you a million dollars. There's the first indication of something fishy going on numerically.

25GB of mailbox space? No. You don't get 25GB each. You get storage. You get told that you have 25GB each. If you go and look you're not going to see a 1:1 capacity correlation between 25GB and users.

But hang on, all that went before was nit picking at tactical points. The real question is what changed in the 11 months between Google Apps coming in and Office 2010 coming in. What were the business requirements in June 2009 and what are those utterly new business requirements in May 2010? Can you, in the business that the company is in, have such a radical change in business requirements over the course of 11 months that caused them to throw Google Apps out and put Office, and Office 2010 at that, on the desktops all over again.

I am genuinely flummoxed at this. Personally I think it's the right solution because the feature-rich messaging client is backed up by a comprehensive back end. But I go back to the original point. What the heck changed in a year? Can you even operate it for long enough before you decide it's not for you and then make the plans to migrate to something new. Can you even plan and execute that migration in, say, 6 to 9 months?

There's way more to this than meets the eye and it's fascinating to me, geek that I am.

Shiny Space Utilization Post

2010-05-18T10:09:00.005-04:00

A colleague: http://recoverymonkey.net/wordpress/2010/05/07/netapp-usable-space-beyond-the-fud/ has posted about space utilization; i.e. the amount of terabytes of spinny you must buy to get 'n' terabytes of LUN.

(Staggering over-simplification follows.....)
An example goes something like this:

With NetApp you get 11 1TB disks and that gives you 9TB of space for data and 540 IOPS. Only it doesn't.
With "others" you buy 18 disks and that gives you 9TB of disk space for the same IOPS (RAID10). You can also buy for RAID5 (10 disks) but you get nowhere near the same number of IOPS and therefore need to beef up the disk count by an undetermined number. Safe to say, RAID5 requires far more than the original 10 disks for 9TB visible. So unless you go RAID0 or implement RAID6 there's simply no way to get a better utilization ratio than RAID-DP - as an up-front raw number.

Now, remember that a 450GB disk isn't 450GB. At best it's 438, in Base2 it's 402GB. Ahh, but it's not. All that pesky 1000/1024 marketing masks reality. All manufacturers and storage vendors ensure that additional space is held back to account for physical blocks (all of which are 4KB chunks) that go bad. If you have one bad block the block is marked bad and a block out of the reservation is activated and the data put back there from parity (no, that's not exactly what happens but you're Windows people and the explanation is, to quote the Wicked witch from CA, perfect enough). That brings the number down some more from 402, usually by about 10%. So you're "450" disk is actually about 360. And before all you DAS jihadists go off on one, it's a disk. All the vendors do the same. Do this on DAS and the numbers are the same. And if the numbers aren't the same you're, by definition, sacrificing resiliency for capacity - and that's bad.

Go read Dimitris. Excellent and cool (if you love this kind of stuff)

Create mailbox store and can't mount it?

2010-05-14T16:57:00.003-04:00

Yes yes, it's here so I can find it.
Created a mailbox store in a crappy lab environment that keeps changing and it falldowngoboom but the fact remains that this can easily happen in production if systems change.

Create database and it fails to mount at the end of the creation process.
Hmm says you.
What's happened is that the configuration DC is not the same as your preferred GC

PowerShell to fix:
Set-AdServerSettings -ConfigurationDomainController -PreferredGlobalCatalog -SetPreferredDomainControllers

It's got a KB

NetApp FAS / V Series & Active Directory

2010-05-12T09:18:00.004-04:00

Oy, the amount of time this beauty gets asked - by Windows guys as well as storage guys. So, in the spirit of the usual "if it's here I can find it again" here's the skinny on NTLM & Kerberos authentication to NetApp systems for Windows NAS (CIFS (SMB)) access.

If you have a NOW account click here which says on the storage controller to execute:

Options cifs.LMCompatibilityLevel (1,2,3,4,5)

where:
1 - Accepts LM, NTLM, NTLMv2 session security, NTLMv2, Kerberos.
2 - Accepts NTLM, NTLMv2 session security, NTLMv2, Kerberos.
3 - Accepts NTLMv2 session security, NTLMv2, Kerberos.
4 - Accepts NTLMv2, Kerberos.
5 - Accepts Kerberos only.

As you can see, if you're in a secure environment there's every chance that as soon as you put the FAS into an Active Directory domain you're going to hike the number up to a five, or a four at worst. Obviously you need to check with the Active Directory staff to ensure that your methods match the group policy in place

Exchange 2010 Single Item Recovery

2010-05-10T16:20:00.006-04:00

Microsoft's Ross Smith has just written a very detailed post on Exchange 2010 single item recovery. It can be found here: http://msexchangeteam.com/archive/2010/04/26/454733.aspx. I can hear you now; wow, single item recovery in Exchange. Cool. Now I don't need all these products from Quest or Kroll. Err, well, no. Not so fast.

What Exchange 2010 can do for you natively is at best a corner case. Leaving aside the jolly awful scenario which Michael neatly solved in one line of text let's see about this.

Enabling the feature has to be done in advance. Enabling this feature will eat gargantuan amounts of storage on your Exchange servers for no valuable purpose. Whilst Microsoft might be drawing back slightly from fanbois flag-waving about DASDASDASOMGDASDASDAS and instead talking about "large mailboxes at low cost" there does have to be some common sense applied. What's the point of having massive mailbox potential if he end-user doesn't actually get their hands on all that much more storage?

Do you really need your admins doing all these steps which, even if scripted, takes an age to accomplish? You can rest assured that deploying space-efficient storage, a decent protection product will cost you a few dollars you will recover the benefits in a very short length of time, especially if you're managing the IT for an organization whereby admins frequently Shift-Delete and then purge items instead of clicking on recovery. It's not as if the buttons look alike.

The single item restore feature solves a problem that doesn't exist and doesn't solve the actual problem that organizations have; how do you provide simple, single item recovery at low cost without firstly utilizing large amounts of storage and secondly knowing that you'll want to do it. The only answer to that is using 3rd party tools to mount up snapshots and recover the item you need.

SnapManager for SQL Server - System Databases

2010-05-06T16:04:00.002-04:00

Now remember this one folks.
By all means SnapMirror your system databases (except temp obviously, always except temp!!) but you can't just do it in the conventional manner.
Since the system databases have to have ye-olde-ancient streamy backup (to be consistent anyway) you don't get snapshots in the FlexVols that host these databases.
What you do therefore is to SnapMirror the FlexVols where the Snapinfo is. That FlexVol contains all the goodness for you to do the restore.
When you create the SMSQL job and select to update the mirror you will find that SMSQL is bright enough not to try and replicate the snaps from the database FlexVol but will replicate the FlexVol where the Snapinfo resides. It's pretty clever like that.

Cunning really. It's a couple of extra steps but the whole lot can be done through PowerShell so all you have to do is find the script and hit go.

SnapManager for SQL Server - Disk Configs

2010-05-06T14:48:00.004-04:00

Here's a quick one-pager for you.
Create three 1GB FlexVols and put a 500MB LUN in each.
Assign three drive letters (E, F and G in this case).
Databases are E. Transaction logs are F. SnapInfo is G.
System Databases:

Create a FlexVol and LUN for each of them and mount those LUNs into the root of E.
Create a FlexVol and LUN for all of them except temp and mount into the root of G.
Run SMSQL Configuration wizard and move the files into the relevant locations in E and G.

There are no files in F because you keep the logs with the databases.
User databases:

As above but create a FlexVol and LUN for the transaction logs.
Run the configuration wizard and move the files.

Options?

To keep your LUN count down you can use the same FlexVol and LUN for all the Snapinfo directories. Or not. The trick is to decide what RPO you need to maintain and lump everything that has a similar SLA into the one FlexVol. SnapMirror the FlexVols to suit your RPO.
Remember that NetApp can do anything, anyhow and not to look at LUNs so much as look at what protection you want for your databases.
Remember that in a DR environment you're going to want your Master database in your DR location so that needs to be SnapMirrored but you don't necessarily need the other system databases.
Remember not to SnapMirror the temp database. You can't actually back it up so that's no problem.

And finally. I'm sorry. Yes. The GUI. Yes. Sorry. Engineering are on it.

Exchange 2010, SnapDrive & SnapManager Security

2010-05-05T08:45:00.004-04:00

Here's what you need to know.

The SnapManager service account needs to be a member of the Exchange role "Organization Management" (Yes, that's the current instruction from Engineering but by all means read it here and ask your local SE to confirm (June 5, 2010))
The SnapManager service account needs to be a member of the local administrators group on the Exchange server and, obviously, it needs to be able to log on as a service - it being, well, a service.
The SnapDrive service account needs to be a local administrator on the server and be able to log on as a service.
On the firewall allow COM+, HTTPS and SnapDrive (swsvc.exe) through.

Did a lot of testing with various Exchange 2010 management roles (i.e group membership) and because they're not very granular when it comes down to the nitty gritty the only workable solution is to put the SnapManager account into the Org role.

Prepare to meet thy doom

2010-05-04T07:41:00.002-04:00

Multi site DAG failover prevention

2010-04-21T19:26:00.004-04:00

Another one of those "gotta post it here otherwise I'll never find it again" things.
You have a 2+1 DAG. The +1 is in a second data centre - well it would be wouldn't it, it's the +1!
Trouble is that the (in your case) second data centre isn't really there for an active/active configuration, i.e. you want control on whether to activate the third copy or have an outage whilst you fix things up. Ordinarily if the two copies in the primary location are down the third is going to come right up. Well, if you've had a link failure in a particular scenario you don't absolutely, positively want to invoke DR. You might want to think about it for a while.

Use: http://technet.microsoft.com/en-us/library/dd351074.aspx to configure the copies so that replication and replay work as designed but never automatically activate.

A Cunning First?

2010-04-16T19:08:00.008-04:00

I read this: http://www.infostor.com/index/articles/display/0355658971/articles/infostor/backup-and_recovery/disaster-recovery/2010/march-2010/emc-integrates_replication.html today and at first it seems like a great idea but then I got to thinking.

In summary there are APIs that you can leverage in Exchange to replicate the databases from primary to secondary sites - or even within sites if you want - rather than leverage the DAG in the conventional manner. Well, that's great, up to a point.

As you know I work for NetApp and EMC came out with something first so it's bound to be bad, right? Well, sometimes they just package up stuff we do and re-invent it like Microsoft do on occasion (virtualization, OCS to mention but two). But this is genuine and something we don't (yet?) do. Whether we intend to do it in the future is not for this blog or for me to discuss - even if I knew.

The use case here is going to be marginal at best. Why?

1. The Exchange guys are best when their egos are massaged and if you play to that you're far more likely to sell them on the idea of NetApp, performance guarantees, snapshots in the "DR" site, single mail item recovery without major restoration work and getting rid of tape.

2. If you use a 1+1 or 2+1 DAG you can implement Exchange in a manner where DR planning (almost) goes away. That's a great boon to the planners and whilst you might think that improves the case for DAS you should read (1)

3. Everyone is doing virtualization. Yes, everyone. Yes, even the mailbox. If you have a 2+1 DAG you have to disable all client side HA features on the host but that's not a real problem to the VMware guys and frankly, they all seem to like the idea of not having to worry about keeping space somewhere for the DR boxes to run etc. It also allows you to worry far less about DR: see (2) and because you're virtualizing you need a SAN (and there's no better SAN/VMware solution than NetApp due to the multi-protocol capability, dynamic caching (PAM2) and sundy other technologies.

4. If latency is over 25ms or there are no servers at the DR location then I'm in to sell customers SnapMirror (EMC will do the exact same thing so I'm not claiming superiority there (although I am because I am reliably informed that SnapMirror is easier than competing technologies)) because no destination server means a requirement to do Database Portability. This happens a lot more than you might imagine. I can tell you that there are a lot of NetApp systems in 401 North Broad but there are nowhere near as many servers plugged into them as you might think! SnapMirror is a massive market for NetApp.

The API replication technology is certainly great. SAN vendors wanted the technology in there to do the replication but given advances in the last year or so I'm now pretty convinced it's going to be a corner case solution.

So if I was EMC walking into a crowd of ambivalent and pro-Microsoft kool-aided Exchange guys I think that I'd be choosing my battles. I'd let the Exchange guys do their stuff with the DAG. I'd let them have total control about what went where. I'd sell them on the idea of snapshot retention and tape elimination. Why? Because it's worked - to the tune of a half million mailboxes this calendar year alone.

*update*
Dustin mentioned "choice" and I thought a little more. There are going to be times when you're in a 2+1 situation, or rather would like to be, and you find yourself constrained by network capabilities. There is nothing to stop you leveraging normal Exchange replication on-premis and then handing the off-site replication to the array. Obviously I'd like you to use SnapMirror but since this is a pro Exchange and (kinda, sorta) kudos to EMC you can use their new announcement for it.
Either way, the customer gets additional flexibility and can do so much more with the technology at their disposal.

VHD De-duplication - Can Haz

2010-04-14T06:05:00.007-04:00

I was having some trouble with system state backups and deduplicating the resultant VHD files on a NetApp controller. When I put a single system state backup onto the storage I was getting around 7% space back on the volume. The backup was around 7GB in size since it was just a lab system. Having put another system state backup from the same system I didn't see any benefits which, if you know the process, instantly shouts alignment at you. Well, I deleted the copy and did a backup of another DC and re-ran a full volume deduplication. The returned space was a mere 5%, hardly a compelling story.
At a time when I was thinking that the VHDs may have been the problem I hit upon the idea of putting two identical VHDs of 2008 R2 that are our sysprepped images. That's better, 55% deduplication. Now I'm copying a third one in there and will see what I get..

The problem exposes something wrong with the system state backup process of creating a VHD and laying the backed up files "inside" it. Going to see who I can ask but I'm not sure where to start!

Image 1. Two Sysprepped VHD files in the volume.

Image 2. Three Sysprepped VHD files in the volume.

SnapManager 6 For Exchange - Things To Note

2010-04-02T09:41:00.003-04:00

SnapManager for Exchange, that application with the worlds 3rd worst GUI (only beaten by SMMOSS and SMSQL) can, in common with every other VSS aware Exchange 2010 backup product can protect the passive copy of an Exchange 2010 database. However, and here's the thing; you have to be careful on your timings.
If you deploy SME to protect the active copy once a day in case you need to do an active restore and also do longer term protection on the passive you will want to pay very careful attention to the contents of your snapinfo directories.
If you take a backup of the passive the transaction logs will be purged from the active (since a backup happened) but a snapinfo structure will ONLY be created on the passive. So if you're on the active and want to do a partial restore to a particular point in time for whatever reason you may not be able to go all the way up to that point because, although you have all the logs SME thinks you should, SME doesn't have every one of them.
Obviously there is no point at which an active restore without data loss cannot be done. You will know where the latest backup is and can always restore up to date. It is only those partial restores for (say) forensic examination or single mailbox recoveries that might be affected.

Rules you can't escape from

2010-03-24T14:00:00.001-04:00

If your Active Directory is borked the resolution is going to be DNS
If your ADFS is borked the resolution is going to be PKI (just ask the ADFS Kitteh)
If your SnapDrive doesn't work it's going to be your MSCS that's not working properly.

Trust us on this.

VSS Confusion and Blank Stares

2010-03-23T16:13:00.003-04:00

Oh my. A prospective customer recently made hand wavey gestures about VSS and, candidly, their lack of understanding about it. Since you can only do VSS backups with Exchange 2010 here's the scoop.

There are three components

Requester. This would be the thing that asks for the backup to happen. In my case it's SnapManager for Exchange. It could be BackupExec or any other application with an Exchange aware agent. It's software. Microsoft have one and it's SCDPM 2007/2010.

Writer. This would be the thing that gets the application ready for the backup. Exchange has one built in, as does SQL. You'll see it as a service.

Provider. This would be the thing that actually takes the VSS backup. In my case it's the snapshot function on the storage platform. In the case of DPM it's the product that drags the changed data off disk and over the network.

Article: http://support.microsoft.com/?id=822896 does a good job in terms of it being competition for paint drying but it's useful for those people willing to spend five minutes reading rather than posting questions on the Internet asking for it in potted form.

Exchange 2010 on NetApp - Best Practices Guide

2010-03-22T09:18:00.001-04:00

The best practices guide is out and being downloaded in large numbers :) http://media.netapp.com/documents/tr-3824.pdf

Exchange 2010, DAG & NetApp SnapDrive

2010-03-15T13:46:00.005-04:00

There I was, perfectly lovely functional Exchange 2010 DAG. Stores on NetApp, stores on DAS (bleugh, how do you people cope!) and everything was cool. Then I decided to play............. Why do I insist on playing. Oh, wait, I know. See last post.....

The end situation was that there was a copy of a database on the DAS and the other two copies (SAN storage being cheap donchakno') on NetApp. Suddenly SME started failing and SnapDrive was throwing "class not registered" errors. Never having been one for change logs I was lost as to when it started happening and what I had done. You immediately see why I'm in a consultancy role rather than an operational one. Lots of wild and frankly random clicking, lots of insane messing around with NIC settings, DNS and lord knows what else.

Bottom line is that in my quest for global domination I had ended up not installing SnapDrive onto a box that was in the DAG. SnapDrive needs to be installed on all of the servers.

If you click "connect to disk" or "create disk" in SDW and you get (again, for the sake of Google hits) "Error: Class not registered" your first, primary, number one check should be that you have SnapDrive installed on every node you can get your hands on.

Moral of the tale? Wouldn't have happened to any Exchange customer since they all do it on NetApp SAN and it only happened to me because I'm a random clicky fool who isn't allowed near anything production-ready. With good cause.

Now I have to put all the NIC and security settings back to how they should be. If I wasn't on te way to Florida I'd have a long week ahead of me.

Meet me on Tuesday at Bright House field in Clearwater.
Goooooooooooooooooooooooooooooooo PHILLIES.

Hyper-V Snaps? Jeez H C NO.

2010-03-15T09:26:00.005-04:00

So here's something for you.
1. Don't touch Hyper-V
2. See rule 1.
If you have to touch Hyper-V don't ever take snapshots using the inbuilt tools. I was recently undertaking a load of work on my simulator (7.3.2) pair (two clusters) at home. Lots of SDW6.2, SME6, SnapMirrors and SnapVaults. Why? Cos our customers want to do some seriously wacky things sometimes and it's good for me to be able to explain what works and why. So I decided to take a few snaps of the Ubuntu guests on which the sims run.

Ubuntu on Hyper-V? Shiny, The NetApp sim inside that lot? Uber-shiny.

My big problem is with Microsoft and their horrific approach to snapshots. Apart from the "don't rely on snapshots as a backup mechanism" (Ben Armstrong) BS the whole creation of massive files was unbelievable. It's an utterly ridiculous approach to snaps.

No wonder Microsoft are muting their utter loathing and hatred for NetApp in favour of an all out love-phest for SnapManager for Hyper-V.

The moral of the story is that if your VHDs reside on DAS you're a fool. Put them on the NetApp. Don't have one? Get NetApp or use VMware. Then you can put behind you the disgrace that is Hyper-V snaps and use NetApp snaps for both for the guest base system and also the guest apps.

If you do take Hyper-V snaps, expect huge outages as you need to save, merge and restart the system. If you do a shutdown of the guest be prepared for an extended outage as the snaps are merged. If you do allocate space to the system make sure you allocate enormous amounts of space so that you can play the snaps into the base file.

or get NetApp. Seriously. Get some. Our mortgage needs it.

Exchange 2010 & NetApp Deduplication

2010-03-08T07:18:00.005-05:00

Lots of testing of various stores under various situations. As you know SIS has been eliminated from Exchange 2010 and a form of store compression introduced. This, along with the other capabilities of the NetApp platform have played right into our hands. NetApp is the only vendor to offer deduplication on primary data. The deduplication is not an in-line process but rather is scheduled. Hourly, twice daily, daily, weekly, whatever. You set it to suit your requirements.
Oversimplification alert: The process runs through a particular volume and checks all the blocks to ascertain whether or not the checksums are identical. Once done a walk is done through the bits to make sure that the blocks with identical checksums are indeed identical (a certain vendor has suffered lost customer data by failing to do that part)

What does this mean for Exchange?
Nothing. Exchange doesn't understand storage, doesn't see a difference and doesn't experience any kind of a performance drop. In fact, leveraged with PAM, the performance of Exchange actually goes UP!!

What does this mean for your storage?
Stores which experience a lot of attachments get around 23% back. Stores which have an awful lot of message body content but very little in the way of attachments experience a return of around 10%. YOUR MILEAGE WILL VARY!!!!!

What does this mean for you?
$$$
If you've got ten stores of about 1TB each the broad-brush returns are that you only consume space for about nine of them. As your stores get large the amount of space returned goes up, even if the ratio hovers around the same place. You buy enough storage for the ten stores and you get the log space for free. That's in addition to the amount of space you gain by spreading your stores across an aggregate of disks. An alternative way of looking at it is if you experience a daily change rate of 5% - not an unreasonable number - then you get a couple of days worth of snapshots for free since you don't need to factor that extra space into your capex. Given that only NetApp can cope with multiple online snapshots without performance degredation the amount of space regained by deduplication really does come in handy.

Ddeduplication on small Exchange 2010 stores that have a high turnover and policies that delete old mail within a smallish number of days probably isn't too productive but consider that against the online archive and the fact that people with huge mailboxes are never going to touch most of their mailbox and you're seeing a lot of 'wasted' storage, about which you can do nothing. Except on NetApp.

Shiny, huh? And one of the many reasons I enjoy working for the company. Cool Exchange-enhancing technologies.

Exchange 2010, DAS and SAN - New Research

2010-02-20T07:01:00.003-05:00

Interesting research article over at Wikibon: http://wikibon.org/wiki/v/Should_Exchange_2010_use_DAS_or_SAN

What do I seee in that article?
I see that the cost for the SAN is higher (than it could have been) because they selected a pretty expensive platform. A certain storage company of my aquaintance would be even cheaper. Go SAN.

I see that they (deliberately) left out such things as de-duplication. A certain storage company of my aquaintance would be even cheaper. Go SAN.

I see that the discussion left out certain key features that optimize space consumed in the data protection sphere. A certain storage company of my aquaintance would be even cheaper. Go SAN.

Whilst it's natural that I would find this kind of stuff interesting the amusing part to me was the fact that I just constructed an identical TCO graph as part of a 150k seat project I'm helping out on.

Shiny, one might say.

Exchange 2010 - Native Single Item Recovery

2010-02-09T19:25:00.004-05:00

For all my NetApp customers who wonder why they need to buy SnapManager for Exchange here's the bill-n-ben...

PowerShell you'll need.
Set-Mailbox -Identity "user.name" -SingleItemRecoveryEnabled $True -RetainDeletedItemsFor 200

What you're going to get.
The user portion of the dumpster will keep for 14 days and then there will be an extra 200 days worth of deleted mail and versioning to changes in those messages availabe for recovery. The user can't do it so it's a helpdesk call just like it always was.

Benefits.
Well, not many really. You're getting the stuff in the one place but that's downsided by the bloat in primary storage. Yo have to manage protection or at least additional storage for that information.

Better?
SnapManager for Exchange, SnapVault. You can keep your litigation hold stores away (remember that Exchange 2010 moves the dumpster items as well during mailbox moves) from all the heavy data protection infrastructure. It's far easier on the eye.

DPM 2010 RC and Single Mailbox Restores

2010-02-09T17:43:00.004-05:00

Well, the good news for NetApp and all the other vendors is that using DPM to recover a mailbox is just an utter nightmare.

New-MailboxDatabase -Recovery -Name "RecDB" -Server - "mbx1.domain.com" -EDBFilePath "path" -LogFolderPath "path"

Be careful not to mount the store otherwise you'll ruin the "allowfilerestore" part which will mean you need to go into the GUI (where you can see the store but not where you can create it)

Once you've done that you need to go to the DPM console, into Recovery and find the mailbox you want to restore. You tell the wizard what store you want to put the recovered mailbox into and then let the job run. The job will restore the entire database (up to a terabyte in Exchange 2010 remember) to the good old SATA disks at 65 IOPS (rather than your SAS at 180/200).

Finally you can export the mailbox you need from the Exchange console.

Elapsed time? A good 20 minutes, minimum, before you get your messages back and a whole Terabyte of space you have to reserve.

OR..........
SnapManager for Exchange. FlexClone/LUNClone (SDW will work it out for you) the snapshot (8 seconds, zero space). Use SMBR to get the items out of the mounted store. (about two minutes total but there's a dollar cost for the SMBR software)

OR.........
SnapManager for Exchange. FlexClone/LUNClone (SDW will work it out for you) the snapshot (8 seconds, zero space). Use the recovery database to mount the store and export-mailbox to get the items out of the mounted store. (about five minutes but it's free)

Bottom line is that the recovery features of DPM 2010 still make you recover far more core data than you need, make you restore that over the network and make you use different tools on different clients and workstations to get at the data. It's more complex.

As luck would have it you can eliminate some of this by using the hardware snapshots feature and I'll post a little bit about that when the lab boys have finished re-jigging and can give me a new 64bit guest for DPM.

2010-02-06T14:27:00.002-05:00

About a foot landed. Drifted to about two feet on my street.

Betsy Ross Lives Here

2010-02-06T11:57:00.003-05:00

Home

2010-02-06T11:55:00.003-05:00

A Great Dane - Confused

2010-02-06T11:53:00.001-05:00

Snowpocalypse II

2010-02-06T08:44:00.004-05:00

Holy hell. This doesn't happen every day of the week.

That was Philadelphia International Airport at 8:46 Eastern. Fort Lauderdale and Tampa. To be fair the phlight to Vegas was always going but you just couldn't cancel the Vegas shuttle could you? I mean, would you want to handle 200 87 year old Jewish grandmothers from New Jersey? You'd want to get them the hell out of Dodge, sharpish.

What else? Oh yeah. The city has a snow emergency in place so the Phascist Phuckers at the PPA are being nice and charging $5 for all day parking. We have no idea why since there is no money in the budget to do any ploughing. Maybe the planned trip to Libs for brunch might be a bit ambitious.

Our friends at the BBC put Snowpocalypse and Snowmageddon on the map by using the words on their website so it's only a matter of time before they're in the OED. Ah, yes, and http://snowpocalypsedc.com/ has been set up with helpful tips for DC residents.

It's only a matter of time before Snowpocalypse III - Revenge of the Drift hits town.

That's it. There is no more. I am now going to find a way to work "eaux" into every written sentence today.

EMC Blows (CIFS Record)

2010-01-27T08:07:00.003-05:00

Good news for the competition based out of leafy Framingham MA. They exploded the SPECsfs2008 CIFS throughput tests. Previously Apple had held it with just over 44k ops/sec using 65 disks and got a response time of 1.89ms.
What did EMC do?
They got their Celerra gateway, naturally. Then they threw away the Clariion that usually sits behind the Celerra and put a stonking great Symmetrix V Max behind it instead. And they 100 disks. Oh, and did I mention that 96 of those disks were SSD rather than spinning plates? Exactly what result did they expect with ~50% more disks and tens of thousands more IOPS potential? Yeah, cos a Celerra behind a V-Max and a bank-breaking SSD array for file shares is a commonly adopted approach.

I mean, why frikkin bother?
Seriously, dudes. What-evv-ahh.

Come on. Let's see you do the same with NFS. You know, the format that you can actually do something useful with. Like VMware. Which you OWN.

CNN in Country #fail

2010-01-25T16:34:00.005-05:00

CNN seems to think that Congress is providing stimulus money for roads projects in the United Kingdom:

Yes, the lines are double yellow meaning no parking and the sign is not a man putting up an umbrella but is a man with a shovel in a pile of something.

New Job New Posts

2010-01-19T09:24:00.004-05:00

No. Not me. I'm still in my dream job until gaffer finds out that I have no clue what I'm talking about and cannot tell the difference between a 3170 and a 6080, a PAM and a PAMII, a DS14 and a DS4243, a SAS and a SATA, an ESH and, err, something else.

Young (barely) Jimmy has a new job (see previous posts) and has taken to blogging prolifically. See him at http://jimmytheswede.blogspot.com/. The recent post about the God Mode thing rocked and new things about the recycle bin in Active Directory (using something called PowerShell, whatever the hell that is) are really interesting. It's almost SnapManager for Active Directory which is something I'm banned from talking about in the house due to something called USN Rollback which, apparently, has nothing to do with the Navy.

Oh Lord. Did I just blog about a blog? Kill me now.

New Outlook / Exchange Tools

2010-01-15T14:54:00.004-05:00

I got pointed at this little lot: http://www.codetwo.com/ the other day. I normally send people towards DidItBetter but it's always good to have some diversity.

The category sharing thing looks particularly useful and the folder sync thing looks like a doozy (and is free. We all like free)

Check it out.

(Update)
Check it out. Or not, if you have Outlook 2010. oh well, never mind.

Shifting Hyper-V

2010-01-06T13:01:00.006-05:00

Why in gods name would I, an Exchange guy working for a storage vendor, evangelize Hyper-V? What on earth am I doing? After all, it's an immature product that has some shortsighted architectural decisions and a feature roadmap that would make a block of Emmental look densely packed. But it's sufficiently stable and has enough of a feature-set since R2 to make it worth a serious look.

Hah, but there's method in the madness. Read on, gentle disciples of my ramblings.
If you don't want High Availability (HA) in your Hyper-V environment you're going to get a lot of looks from people; "Seriously, for real?" type looks. If you just want one server in your organization and have a few applications that don't sit well together because they're from different, competing vendors (Like, say, SQL, Exchange, SharePoint don't work well together and are from differ. oh, wait.) then you're fine. You aren't my target audience and can go back to your nice cup of Rosie Lee.
However, if you've been looking at VMware and now Hyper-V for multiple servers and want HA to guard against host or site failures then your ass is mine. In this case, central to Hyper-V is storage. If you leverage a server virtualization solution it is an all-but-given that you are going to use networked storage. I say that because whilst you can use SCSI attached shared storage why would you restrict your scalability? Seriously. Dudes.

So now you've got your Hyper-V and your NetApp storage (why would you go anywhere else)
you can put your Exchange and SQL and SharePoint on it. You can deduplicate volumes containing the system drive VHDs to reduce the space that your 20 C drives consume, probably into slightly less than one would take up. (Storage Efficiency, shiny!) Then you can leverage SnapManager for Everything (SME, SMSQL, SMMOSS etc.) to take application and environment consistent snapshots of your data. You can also use SMHV to take snaps from the host of the guest operating systems and soon orchestrate everything including replication of all that data to your disaster recovery site.

No. i'm not finished. All that testing you need to do in order to make sure the latest patch from (insert name of random software vendor) you can do so. Quickly. Without consuming tons of storage. FlexClone the volumes you need, present them up to your sandbox and have at it.

Bottom line? Microsoft have their agenda to shift Exchange 2010 and shift it on direct attached storage. Microsoft have another agenda to shift Hyper-V and all the System Center stuff that wraps around it. So Microsoft can either use Exchange on DAS or they can use Hyper-V. If I can persuade customers to virtualize I can guarantee a NetApp sell and if I sell NetApp I'm in business and the customer gets an enterprise environment rather than a narrow point-solution

More Hyper-V

2009-12-26T18:22:00.002-05:00

Not content with going halves on a new TV for xmas the decision has been made to go halves on a brand new shiny server. On the way is a twin dual-core beastie, 24 of the RAM and four 15k SAS kiddies.
Processor - check. RAM for ~10 mid-power concurent guests (two DCs and eight Exchange servers perhaps (I wish!!)) and IOPS - check. Stick a bit of SATA in there for some basic capacity on top of the 3TB of capacity in the other host and we're hot to trot.

Now all we gotta do is take out a second mortgage to pay the electric bill.

OMG - No BlackBerry

2009-12-23T02:10:00.002-05:00

Cold sweats.
Shivers.
No BB service across USA.
Oh, company just mass-emailed us. No BB service globally.

Wondering why left Windows Mobile.

Oh yeah, I remember, UN-fracking-believably unstable builds.

Wondering why not on iPhone.

Oh, yeah, I'm on Verizon.

Booooo.

Mixed Workloads on NetApp

2009-12-14T09:33:00.004-05:00

Some very bright people have been beavering away. http://media.netapp.com/documents/tr-3804.pdf
Ever wanted to run your data on one efficient platform? Ever wanted that data synchronously replicated to somewhere else? Fancy having complete 100% seamless failover? Fancy having the data in one place but the servers in another? (after all, why fail it all over if only half it went titsup!)
It's an awesome paper that describes Microsoft solutions on a NetApp MetroCluster platform and gives you an idea of what you can do for your customers in terms of data presentation and protection so long as you are prepared to get out of your comfort zone and think innovatively.

Exchange 2010 - The DAG is blown

2009-11-23T11:54:00.004-05:00

I'm doing a lot of stuff with current and potential customers around Exchange 2010. Every single one of them wants to use VMware and VMotion and every single one of them wants to use the DAG and database replica's.

Well folks, there's bad news: This link tells you that, just as in Exchange 2007, you can't have your cake and eat it. Since the Database Availability Group requires MSCS underneath it you can't also use Hypervisor based high availability solutions.

What does this mean for you? Let's take a look at some conversations and the internal (customer) reactions when it because apparent that both the VMware and Exchange teams were resume padding, the storage team was trying not to put more PB on the floor than necessary and NetApp were trying to get the customer to do the right thing, in spite of the customer's best efforts.

Customer 1 Exchange team: We're doing Exchange 2010 on blades and DAS. (Storage team enquired how the backups were going to work and the hardware team wished the Exchange team all good luck in finding someone to support that cluster-fisk-in-the-making because they didn't intend on buying more enclosure space for SB40's when there was a x*sb going in for the express purpose of attaching blades to the SAN they already owned).

Customer 2 Exchange team: We're having three replica's. (Storage team told the customer that they could take a flying you-know-what. Windows team said the same. Hardware team were polite and asked them who was paying for the servers. The VMware team were polite and said that since the Exchange 2010 was going to be virtualized the Exchange team might want to read the support statement. The Exchange team shut up)

Customer 3 Exchange team: We're doing Exchange 2010 on DAS. (Exchange manager said 'like hell you are'. The VMware team asked how they were going to do Exchange on DAS on an ESX box. The storage team sat back and shook their heads)

Customer 4 Exchange team: We're going DAS because Microsoft say they don't support Exchange on SAN. (Storage team threw (yes, threw) the Exchange team the NetApp and Microsoft paper from the MS website. The hardware team wished the Exchange team all the very best in securing the local disk for DL380's. The NetApp team asked for the name of the Microsoft rep so we could get him educated).

As you can see, everyone wants to do Blades for data centre footprint optimization or VMware for the same reason or for NetApp for storage footprint optimization.

Teams. Talk to each other for goodness sake. You shouldn't have to embarrass yourselves in front of your SAN vendor or their partner by not actually having an Exchange 2010 design ready. You don't have to know exactly what SAN you're going to put it on because that can be worked out later; you just have to know some fundamentals about Exchange and what can be supported.

Repairing Exchange

2009-11-17T12:35:00.003-05:00

One of those rare events happened recently; an Exchange database went south due to explosive hardware failure. In reality I doubt there was anything wrong with the database given whose storage it was on, but the customer did a range of things without refering to anybody's best practices and all manner of trouble happened in rapid cascading manner.

So here's what you might want to consider if you have an Exchange 2003 store blow up in your face.

Don't try to repair the database in the LUN. That's a number one, never deviate, cardinal rule.
Take a snapshot of the volume and FlexClone it.
Then do your repair attempt against that clone and mount it to the server.
If it mounts then all well and good. Conduct a FlexClone Split and eliminate the parent when it's complete.
If it does not mount you can go back to a previous snap and FlexClone that, then repair it if necessary. FlexClone Split when you're ready.

If you don't have FlexClone you can get an eval key from your local SE, CSE or partner SE. We don't mind; all part of the service. Once you see how useful it was I can guarantee you're going to buy it before the 30 day key expires.

(yet) Another VSS Hotfix

2009-11-12T11:16:00.003-05:00

There's another hotfix out: 975832 because VSS isn't correctly responding to the hardware provider reporting a failed snapshot in all cases leading to some disk operation restrictions.
Yes, it applies to Windows 7, Windows Server 2008 and the R2 release as well.
It goes with 953531 and 950267 and 957522

Microsoft are really, REALLY going have to pull their fingers out of their testing asses if they expect Hyper-V to be the latest vehicle for global domination.

Alternatively we can just shut down the servers and take copies of all the VHDs and the RDM data it's pointing to.

Can't beat 'em? Join 'em

2009-11-08T12:30:00.004-05:00

The Metropolitan Police failed to go into full abuse-of-powers mode on Thursday when 30-40 Guy Fawkes took a stroll around Parliament Square before wandering into the Commons Gallery to watch some proceedings. Her Majesty's finest did, however, politely inform the tourists that removal of masks and gowns would almost certainly be in their best interests whilst inside the Mother of Parliaments.

Photo credit Cassie Myers via the El Reg.

Exchange 2007 Support Flip-Flop

2009-11-05T15:45:00.002-05:00

Seems that Exchange 2007 on Windows Server 2008 R2 will very soon be a supported configuration. Exchange 2010 on either 2008 or 2008 R2 was always a supported config.

I'd rather MS spent the time on something else (using the "if you had $100 to spend, what would /you/ spent it on?" thing) but seems that lots of punters have been shouting loudly and you have to listen to the people who help pay your mortgage.

"I can go to Best Buy for these disks"

2009-10-29T07:17:00.005-04:00

This is one that ranks right up there with my recent one: http://markarnold.blogspot.com/2009/09/how-many-usable-terabytes.html

A thread cropped up initiated by a conversation with a customer and it bears bringing out.
1. Yes. Yes you can get in your SUV and mince down to Best Buy for a car load of 1TB disks.
2. No. You will not get the same 'kind' of disks that you get in a storage vendor's arrays.
3. Those disks you get retail are just that, retail. They fail. Often. Very often when running at full wack 24/7.
4. The disks you get from NetApp, EMC, HDS and everyone else like that are enterprise grade, not cheapo ones you get from a retail outlet - even if they have the same part numbers on the stickers.
5. The disks are manufactured by a couple of companies and have custom firmware on them that optimizes their performance. The disks that you get retail are the very disks (well, the very spindles) that storage people reject. Well, ok, they don't reject them because they're not of sufficient quality to be offered in the first place. You get the idea.
6. You're not just buying the spindles when you buy the disks. You are also getting all the R&D that goes into the shelves. Unlike the kit that Google knock together (google for shouting at disks) the kit from storage vendors isn't susceptible to being shouted at; they aren't going to go into a tizzy when some long haired yob from the Chocolate Factory gets the server-room on his swipe card.
7. Yes. You can buy a shed load of disks and put them into a, err, actually what are you going to put them into by the way? You're going to put them into some kock-up box and stick ZFS on it are you? Uh huh. Lemme know how that works for you when you have a disk failure, or you need to do something clever like application-consistent snapshot the data, replicate it somewhere or anything else clever. Oh, of course. You're going to put them into a Windows server. A Windows server. Really? Yeah, whatever boys. Enjoy the IOPS you get out of that puppy.
8. Ah ha, I hear you cry. You can negotiate direct with Seagate to get all your disks and get FC instead of those cheap and cheerful SATA disks. Good luck with that. You have to buy tens of thousands a year to get the discounts.
9 So you're left with BestBuy and a pile of 7200rpm, 1TB fail-a-minute disks. I can assure you that far fewer 450GB, 15k rpm disks in a nice vendor-supported chassis will work out far cheaper.

Guess what. If you find yourself with the whole "get them cheaper at...." thing just remember that you're going to make yourself look silly in front of your colleagues and you're going to give the storage consultant opposite you an opportunity to eviscerate you, politely.

Picture of the Day

2009-10-28T08:29:00.003-04:00

Look carefully.

Sigh. Look again.
Sigh. Who do I work for?

Ahhh, now you're getting it.

Giant snakes not on plane shock horror

2009-10-26T12:22:00.002-04:00

In other news: http://news.nationalgeographic.com/news/2009/10/091014-giant-snakes-invasion-us.html giant snakes have escaped that MoFo plane and are invading Florida where they are thriving on a diet of blue-rinse and bagels.

Sadly, this story will not be found on Fox until a way has been found to blame inaction by President Obama for this undesirable influx. It's only a matter of time. However, since Keith Olberman and Rachel Maddow are far brighter they'll be able to work out how to blame Gov Crist for this in far less time.

NetApp and Microsoft Vulnerabilities

2009-10-23T10:49:00.003-04:00

It often comes up whenever there's a slew of CIFS, SMB, etc. vulnerabilities. Since one of the many functions of NetApp is to replace Microsoft Windows file servers customers always ask questions on how to apply Microsoft patches to NetApp controllers. Clearly that's impossible and in any case the Windows vulnerabilities are generally not applicable to NAS platforms. However, all Micorosoft vulnerabilities are tested, partially because the NetApp system will join an Active Directory name and/or present LUNs to Windows server over iSCSI, Fibre Channel or FCOE.

To that end NetApp produce: http://now.netapp.com/NOW/knowledge/docs/olio/MS_security/index.shtml#oct09 which is accessible to NetApp customers, all of whom will have a NetApp On Web account, either individually or collectively. Change the bit after the hash to get the latest version.

And that's how customers get to know if they need to do anything with their FAS systems before implementing a patch.

Exchange and Thin Provisioning LUNs

2009-10-19T10:40:00.002-04:00

It came up on an internal thread so here's the take.
Everyone can do Thin Provisioning (TP). NetApp might have been first with it and it continues to be a great selling point in that you can tell the network (NAS) or server (SAN) that you have 100GB of storage available to your project (whatever) but you physically assign a mere fraction of disk. As projects grow you can purchase disk (if you've got such a model) or assign the disk space. That helps space efficiency and it helps smooth the cost model. In general it's a great idea.
There's a catch.
With Exchange it's pretty much pointless in most cases.
Exchange scales outwards, 100GB at a time - if you think about it. You don't go and assign all "40" (whatever, it's a number) stores at once. What are you going to do, migrate 4TB of mail onto the box overnight? I don't think so boys. You are going to assign a handful of stores to the box and then fill them up to the design level. Then you're going to create some more stores and ride the wave forwards.
If you're Thin Provisioning the Exchange server in terms of stores there's no point in also Thin Provisioning the storage behind it.
So you have 40,000 users to migrate. Do you purchase and commission all eight servers on day 1, fill them with stores and fill all of the stores with mail? You probably don't buy all the servers all at once. You certainly don't assign all the stores at once.

Remember, when a 110GB LUN has a 100GB store in it with perhaps 80GB of live mail and a bit of white space, it's not Thin Provisioned any more, it's provisioned and in use. If you think you need to TP your Exchange environment and you are so tight on budget that TP comes up you probably should be reviewing your whole plan because storage isn't your problem.

Exchange and Caching - Shiny

2009-10-09T12:04:00.009-04:00

** Yet another oversimplification coming **
Arrows not necessarily accurate, merely representative, broadly, kinda.

What the hell are you looking at? Are you sitting comfortably? Good, then I shall start.

How NetApp caching and the Performance Accelerator Module (the lovely PAM) work together.....

1. User A sends an email. Exchange sends data to disk. The cache stands between the write request and the actual write. Cache says cheers to the server. Cache gradually fills up and then stripes a write out to disk, complete with parity and double parity information. That's the thing, along with WAFL and RAID-DP that makes NetApp the reason I work for a storage solution vendor.

2. User B through Z etc. are the recipients of that mail. The message is delivered to the Exchange server and the disk map is updated so that when User R wants to see the message and the Exchange server has knocked it out of Server RAM the user can get it with a single IOP rather than two. User B through Q were in the office at the time and got it through server cache, but user R was having a liquid lunch since he's Italian and probably called Gianluca Hotz or something.

3. User Z is Jimmy Andersson and comes into the office the next day to get his mail. That message has aged out of everywhere and is called up from disk.

A. NVRAM (Cache) is split into parts. In this picture A is the bit that the controller is writing to.

B. This is the bit that is actively flushing data to disk whilst A is being written to. A and B will swap every 10 seconds at an absolute maximum but there are way too many parameters that are far beyond my understanding. If I understand that in depth I wouldn't be an Exchange MVP.

C. This is the bit that's nice. In addition to the controllers being clustered and able to take over each others workloads the memory (A and B) is replicated to the partner controller. If a head fails whilst stuff is happening on A and B there is another copy of it inside the C bit. Writes carry on. If there is some total power failure to the room then the whole lot is battery backed by one of those little rechargeable Johhny's and writes will sit there for, literally, days whilst you call Works Services (good RAF term there) to come change the fuse.

What does all this mean to you?

It means that you are closer to being back to sizing to meet capacity requirements rather than meeting IOPS (through spindle) requirements. But since it's all protected by RAID-DP you're getting full protection, only need one copy rather than three in your local data center and another in your remote data center.

It also means that we're better, and cheaper, than everyone else's DAS based ideas. We will do you less disk and manage that disk better than any DAS offering. It's more efficient and easier to use. Essentially, we rock and anyone who tells you that your Exchange 2010 - at a certain level - needs to be on DAS is no more than a glorified Exchange Admin rather than an Exchange or an Infrastructure Architect. It's a storage solution. It's a backup solution. It's a replication solution. It's extensible and scalable. It starts at a few Terabytes and goes to 1.1 Petabytes. The disk is just the disk. If you want something that stands out you need to look at the whole picture. Anyone can back up. Can just anyone backup inside seconds? Can just anyone synchronously write to the DR site without additional hardware or software? Can just anyone do an instantaneous restore of a backup from several days ago and have you up and running inside a few minutes? (No, is the answer to all those things in case you were wondering)

I either don't understand enough or I understand too much about the innards. Could go either way.

Complexity you don't need.

2009-10-09T10:27:00.003-04:00

One thing that the DAS Fanbois over at Microsoft do have a case about is the "complexity" of the SAN. But not in the way that they think. More later. But consider Compellent and (now) 3Par who: http://www.theregister.co.uk/2009/10/09/3par_adaptive_provisioning/ are talking about tiered storage. EMC have had idea for the Sym for ages but given that a Sym will cost you both arms, legs and your first born it's not a topic for Exchange aficionado's.

The other (Since you all know my mortgage is paid by NetApp) folks look at Exchange storage as tiers and in fact when I worked for the loved and much mourned Posetiv the Compellent story was, as El Reg said, compelling. That was before the debut of the lovely PAM onto your screens.

What does an Exchange 2010 environment look like on NetApp? It's as many SATA disks as you want, in a 24 disk DS4243 shelf attached to a controlling head. Inside that head is indeed SSD, and SSD thy name is PAM. But, it's innovative (yes, I would say that, wouldn't I? (copyright Mandy Rice Davis)). That SSD is read-only. You (as a user) never write to it. Information pushed onto disk is sent to disk.

The most important thing about any disk subsystem is the fact that one read is always two actions. One to read from the GPT/MFT and one to get the block off the disk. SAN or DAS, it's the same. Pulling the entire disk map into memory means that you get to half your read operations.

If you have Exchange data on the SSD you still have to do those two reads only now you need to provide an SSD for the disk map and waste money on an SSD storing a ton of data.

Compellent, 3Par and EMC are wasting customer's money by focussing on somewhere fast to store the data and somewhere to push the old data off to (remember that Exchange 2010 brings you truly gargantuan storage capabilities) slower and way cheaper SATA. It's all, if you will allow me, ass backwards. An unintended flaw with that model is where you store your snaps. The snapshot backups either have to be off-box or are shoved onto the slower disk meaning that your 15 minute RTO is blown. Remember that NetApp gives you the capability for 255 snaps on a volume. Not that anyone ever would do that with Exchange because it's expensive on disk, but that feature is there between 1 and 255. Three is a sensible number, seven if there isn't a tape-replacing SnapVault implementation.

We (yeah, like I'm really bright enough to be 'we') figure that if we can throw just the right amount of SSD (16GB, 256GB and 512GB per card - more than one card is possible) at the controller(s) and let it look after what goes onto it based on policies that users define, the whole Exchange - and this works for SQL in the same way - shebang just sits on SATA disks. All this works for Exchange 2007 as well but it's probably best to replace SATA for SAS. No tiering and a lot more simple.

OK, I was going to go off on one about this 'complexity' nonsense but I'll pass for now and do that in a coming-soon-to-a-screen-near-you post.

Exchange 2010 and NetApp - A Case Study

2009-10-08T16:37:00.001-04:00

Hot on the heels of today's RTM:
http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid=4000005382

Exchange Change Rate and SnapShots

2009-10-08T08:56:00.003-04:00

Something came up last week and, like a lot of things at that meeting, I thought was a given - it being pretty obvious - but apparently not.
It was all about change rate on Exchange databases.
If a user has a 100MB mailbox it's not unreasonable to expect 5% per day to change. There will be a few megs in, a few megs out (no shaking, turning or hokey-cokey though).
But what if the user as a 3GB mailbox? Is it still reasonable to work on a 5%? Sure you say, but how many megs is 5% of 3GB? Well, it will come as no surprise to learn that it's 153MB.

So, is 5% still reasonable? How much email can one person receive and delete in one day and still manage to do other things, like eat and pee?

When you're working out snaps to assign space you're going to need to take this into consideration. How much do you allocate for snaps? Well, a rule of thumb for me is to assign 5% for up to a gig, 2% for the next gig and 1% for subsequent gigs.

Aww hell, that's work. Yeah, it's why you get paid the big bucks. Cope.

This number is particularly important in NetApp environments that leverage SnapManager for Exchange and the snap_info directory structure. Oversizing the database volumes isn't such a big deal especially if you're thin provisioning but the logs are a little different. Working on a change rate that's too high will lead you to assume more transaction logs than will actually be the case. NetApp exacerbates this on the path to highly granular instantaneous restores by NTFS hard-linking logs into snap_info from the logs directory. So oversizing the log generation means you will oversize subsequent days snap retention space. Since that's in the live LUN rather than in a discreet place within the Flexible Volume you can end up assigning far more space than necessary.

Luckily you can shrink a LUN with SnapDrive and Windows Server 2008.

Exchange 2010, Windows 7 Launch Event Oops.

2009-09-28T16:36:00.006-04:00

As part of my duties working for the Microsoft 2009 Storage Partner of the Year I was required to attend the Exchange 2010 Launch Event near Philadelphia. Oh Dear.

We spent a lot of time talking to customers and potential customers about their plans for Exchange 2010. Actually most of them were planning on moving to Exchange 2007 from 2000 and 2003 but enough were thinking about 2010 to make the conversations worthwhile.

At 3pm I decided that the CSEs should go to the Exchange 2010 HA and Storage presentation. This was delivered by a BT Employee from their US Subsidiary. The presenter had some problems with the convoluted and somewhat disingenuous wording of the presentation but nothing compared to the implosion that happened when he did the demo. Rather than get a decent Hyper-V box (he works for BT and it's a Microsoft event FFS!!!!!) he was given a slightly beefy (8GB) but-not-beefy-enough laptop and ran Hyper-V on that. He had a DC and three MBX to form the DAG. No off-box HT.

What happened? Created mailbox, OWA'd into mailbox and sent a mail. Then turned that guest off and waited to get into OWA on the other box. He eventually had to come out of IE and go back in again. Guess what. The message was missing. The box hadn't had the time to close the log and throw it to the other server in the DAG with the mailbox copy. Result, lost data. I bet he was on the Atone Phone later.

A little prompt from the audience along the lines of if there had have been an off-box HT what would have happened (I knew, I was just throwing him a bone to prompt him to tell the audience). Nothing. Nyada. It took his Microsoft minder to come to his rescue and "To speak to this gentleman's point etc. etc".

I have spoken to a number of people after the event who questioned how the tagline "The New Efficiency" squares with the need to have at a bare minimum three copies of the information stores on servers that are, in the example, no more than 33% utilized. I agreed and gave the attendees our take on the situation which is, like Fox News, fair and balanced. The efficiency tagline really refers to the efficiency with which a Windows admin can now do his job and keep the server secure.

Well ok, maybe not. Microsoft are being the Fox News and I am the Keith Oberman or Rachel Maddow Show on this thing; Microsoft are shouting that the SAN sky is falling and I am shouting that it's not. The sky is up there and isn't coming down any time soon.

Do more with less resources = more efficient. Making the logo the new squigly light bulbs is quite frankly a lie.

The event seemed to have been a relative success and the punters got some good information about the technology coming up. It was a bit difficult for me to guage since I've been using the stuff for a couple of years by now and it's all old hat to me and I'm not very good at remembering that all this is new to most people. Still, I shall continue on my quest to evangelize Exchange on NetApp which is what I'm paid to do and what I enjoy.

Carry on.

"How Many Usable Terabytes"

2009-09-25T17:49:00.006-04:00

**Warning. Simplified Example Coming. Look Both Ways Before Crossing Tracks**

If you feel yourself asking that question when talking to any vendor; be it my very own employer NetApp, EMC, HDS or even HP with their MSA DAS boxes, be very careful. In fact, stifle it because I can assure you that there will be some people in the room who already think you're dumb and your question will remove all doubt from their minds.

Why?

I shall tell you. Exchange, SQL, Oracle et. al. are not about visible storage. You need to provide performance for all these applications to serve up data to their host servers. The disk capacity is largely incidental if you think about it carefuly. If you are planning on hosting 1TB of email you can't just mince down to BestBuy and wander off with $120 worth of SATA. Even with Exchange 2010 you can't do that. If you want 1TB of email there's a very good chance you are hosting 1,000 (for ease of mathematics) mailboxes at a gig each. let's say that all of them have BlackBerry devices and that means ~3 IO/per sec per user; 3,000 IOPS per box. That's a massive oversimplification but it's all based on mundane facts. An average FC disk (15k) feeds you with 150 IOPS and an average SATA (7.2) with 65 IOPS.

So, your 3,000 IOPS needs 46 SATA disks. You've just bought 46TB of raw data for one terabyte of storage. In reality you're going to buy quite a bit more. If you assume a (truly awful idea) RAID 5 you will perhaps use seven disks per array. So those 46 disks are going to have about seven parity disks and there's going to be a hot spare in there somewhere. We're up to about 74 disks, the same TB number. All for 1TB of actual stored email.

Obviously your mileage will vary. Your numbers will vary but as a baseline it's sound.

Oh, and FC disks? You'll be needing 20. Add another two for RAID-DP and another one for a spare and you're up to 23 disks. Naturally it's a lot more cash to outlay but they take up less space and power and of course, kick out less heat meaning less cooling required.

Way back in the day, a thousand years ago disks were 72GB (and a little slower at 10k rpm) and 125IOPS each. So your 3,000 IOPS needed 24 disks (plus parity) which meant that you needed more disks so that you could meet capacity rather than disks to meet the required spindle (IOPS) count. Thus the "How many usable Terabytes" was a valid question because you knew that you needed to take some disks away for RAID and some capacity away for marketing capacity numbers. Buy 20 72GB disks and you do not get 1440GB, you get closer to 1368GB. You have to take a pile more off for RAID and before you know it you have the spindles to meet your requirement but not the capacity.

Now you can only get 450GB FC/SAS and 500GB/1TB SATA disks. This means that you are going to pass your capacity requirement about 30 seconds into your journey on the road to IOPS.

So, next time you hear yourself thinking of asking about usable Terabytes you should bite your tongue, it will make you sound a little silly.

Honey, I Shrunk The LUN

2009-09-14T15:28:00.003-04:00

There are areas of Microsoft that are crazed DAS loons but others whose entire existence relies on a SAN (Hyper-V and DPM). The Hyper-V folk have been very pro NetApp because they see some pretty neat solutions that help them shift Hyper-V to try and catch up with VMware. What you see for Hyper-V here today you already have with VMware but it's important to note the likely market penetration that Hyper-V will get. Thin Provisioning and deduplication are just two of the components demonstrated that will be of great value to end users.

Shiny. Frea speaks whilst (probably) Chaffie drives.
Frea does not actually put her Storage Admin hat on though, since she's a Marketing non-droid-ette.
http://www.youtube.com/watch?v=ibOtcWrUkPw

See also: http://www.youtube.com/watch?v=AvkOJU4Iclk&NR=1 for some VDI stuff but the sound is crap. I must remember to report that.

SQL; Development, Testing And NetApp

2009-09-10T13:55:00.005-04:00

Here's your situation. You have this production database and you have this dev/test/whatever server that you want to do some work on. The tired old way that you dba's are used to involves taking a huge great backup of the production, shoving it across the network or onto a disk and then restoring it to the dev server.

What if you could do that inside 30 seconds, at the cost of zero storage and all in one long line of text?

If your source server has SnapManager on it (and it will because you will be backing up your production SQL servers) you also have access to the PowerShell command below.

clone-backup -svr production1 -Database Adventureworks -TargetDatabase yourdatabase -TargetServerInstance devserver -Backup sqlsnap__production1__remainder_of_snaps_name

All in one line you create a FlexClone of the SnapShot of the production store, create a database on the dev server, mount that flexclone as a mountpoint and then attaches the files to the database. Seriously cool and shiny.

There are some more parameters as well. If you don't want to bring the dev server entirely up to the same point in time you can use the SMSQL Installation and Administration guide to specify what point in time you want to roll that database up to, so long as there's a snapshot of the logs to go to.

What do you do if you like the look of the database and want to take a full copy of the information to put into production or generally do something else jazzy here's what you do:

rsh netapphostname -lroot vol clone split start volumename
the volume name will be garnered from your SnapDrive application and will most likely start "sdw_cl_". Make sure you split the right one!

Exchange 2010 Storage (part n of n*)

2009-09-09T11:02:00.006-04:00

I posted comments on Oz's blog after he responded with some neat information to Exchange 2010 soon-to-be-users. But I also wanted to expand on some things that I've said there.

In line with my other post we’re going to love Exchange 2010 deployments.
>For those organizations that are sufficiently large (starts ~50 or so users) where a centralized backup, recovery and data management solution is necessary the Exchange 2010 can now be used as a slot-in product rather than the centerpiece that all other applications were too afraid to dance around.

>The archiving mailbox is going to rock. One thing that NetApp and the other storage vendors suffer from is memory pressure inside the controllers. NetApp can typically put ~50/60k users on a FAS3170C with PAM1, more with PAM2. But, if the archive component makes a 10GB mailbox look like a 100MB mailbox then the number of users hosted on a controller is going to leap. That’s not going to be of interest to anyone with less than 100,000 mailboxes and frankly, at that point, the use of DAS gets pretty uncompetitive (remember that Microsoft are the first to admit that their solutions are Showcase rather than Best Practice).

>Traditionally NetApp has always said use the 15k disks and use FC rather than SATA (we've only recently introduced SAS into our mainstream product set since there was never any real call for it). That's going to change because it is, simply, pointless to use significantly more expensive diska than necessary. The relative inexpensiveness and greater capacity of the SATA (1TB available, 2TB coming) disks are the way to go.

>On that note, here's something that is going to play into storage vendors hands. Currently the largest SATA available is 1TB. That's kinda ok for Exchange 2010 because you could get yourself a store of ~700GB and use ~100GB for logs leaving the rest to save NTFS from getting in a fracked up tizzy. What do you do with 2TB disks. Do you want your Exchange store getting to 1.5TB in size? There is going to be some degree of discomfort among admins there - even if there are three copies knocking around. Co-host stores on the same disk? Well, that's IOPS profile interference and you're just asking for trouble. Using RAID-DP is going to help there in terms of financial probity and storage efficiency.

>Exchange 2010 on NetApp will be on SATA and there will probably be a PAM2 card attached somewhere. That makes the likelihood that even more users will get hosted on the storage platform which means that the storage will get used far more efficiently. One thing that everyone suffers from is capacity utilization. In the old days (yeah, a whole two years ago!) you could get 144GB disks. You could get all the IOPS you needed from a given number of 144GB disks and because the disks weren’t huge you weren’t wasting terabytes of space to meet spindle count. Spindle count applies to everything; file shares, SQL, Exchange and applies to SAN, DAS or NAS – EQUALLY. These days you have to provide the same number of spindles (allowing for 10k or 15k disk speed improvements obviously) but there is a ton of space you can’t use. 144GB disks are now like rocking horse sh1t, 300GB are going the same way. 450GB disks are the norm. For NetApp this is great because we can give you ‘free’ space for transaction logs and equally ‘free’ space for SnapShot retention without performance degradation. Although there comes a point where EMC are right when they say “Well, exactly how many snaps do you really need FFS?”

>NetApp, EMC and Dell all have challenges ahead to evangelize that sticking stores on disk is only a fraction of the story. Back it up? Well, use multiple copies of the DAG or use Snap*? Could do either way, and indeed will.

>If you’re a NetApp house you, as the Exchange guy, carve your own LUNs up. Exchange 2010 will actually make that process easier as you will see when the Best Practice guide comes out.
>Exchange 2010 will hit our relationship with SQL and Enterprise Vault. I haven’t seen where they’re going with that yet. It’s funny though; Exchange is going to negatively impact SQL sales. I want to be a fly on the wall for that conversation when it happens!

>It's never about the base cost of the kit. It's rarely about the (admittedly) cool software. It's always about efficiency.

Great article from Oz and I feel sure he owes Fullbright and me a beer at summit - if I get renewed next month which is far from certain since I've been busy earning a crust this year.

How many disks?

2009-09-08T19:27:00.005-04:00

Let's tale a look at the SATA disk design du jour for a random Exchange 2010 solution.
In DAS, this is your flow:
I want 20 stores. Let me buy 20 1TB SATA disks for my stores and maybe another five 1TB disks to play safe and use for my logs. Maybe just go with the 20. Anyway, 25TB of JBOD. Good enough for anyone really.
But with SAN:
Let me use a FAS2050. I can use 24 1TB disks, have one spare, four parity disks and 19 data disks for 14TB of available storage. At this break point NetApp has only 14TB of capacity and DAS has 20/25TB.
Damn, to the DASmobile Robin...........
Oh, but with SATA/DAS/JBOD you have to have three of them so NetApp gives you 14TB but you have to buy 75 1TB disks for DAS. And three servers. Three servers with 66% more memory and CPU than necessary.
With NetApp you have full protection and are supported on SATA with a single server. Even if you follow an early hair-brained scheme by putting the database and logs on the same disk that's still 60 1TB disks so don't try to toss that one over the fence.
Ahh, but..
You have to pay for the storage controller and software. Yes you do, but in line with previous posts we've already proven that the FAS is actually cheaper to buy and operate than all that disk tagged onto those servers.

60/75 disks and three servers versus 24 disks and one server.
Remind me, how is DAS cheaper again?

Time to stop the porkies boys and girls.

Don't believe the hype

2009-09-08T18:49:00.005-04:00

The hype is crazy. Oz puts it at his most fanbois best here: http://smtp25.blogspot.com/2009/09/what-is-business-justification-going.html when he, like a lot of others, rails that for the first time in history a SAN is off the plate, allowing cheap storage rather than expensive SAN. Now, to use the cheap storage you HAVE (because otherwise you're not supported) to use three or more copies of the Exchange store in your DAG. If you don't use three servers that can operate at no more than 29% of potential (in case one server has to take over the other two) you're replacing 'cheap' (he means SATA) storage for SAS storage and you're not a great deal further forward than you are today.

Even though there are some pretty impressive improvements in I/O with Exchange 2010 you're going to have to be very careful how many users you put on a disk - whether you use a RAID level or not. You will have to ensure that your BlackBerry's don't overload the disk and that's allowing for the leap in optimization that the latest BES software brings.

What about your storage efficiency. Do you really want to have a 1TB disk spinning at 7200 rpm and only use 300GB, even 500GB of it. That's 50 to 70 percent wastage and you can't do anything else with the space otherwise you over-complicate your IOPS profile. Do you really want to implement the Exchange 2010 archive for no other reason than you have the space? Do you actually need to store that data?

And Oz, where are you going to put those backups? I had to blog: http://markarnold.blogspot.com/2009/01/netapp-microsoft-data-protection.html some while ago to help all those Microsoft / Exchange 2007 / DAS zealots who said that DAS was the greatest but needed somewhere to store the backups. NetApp NAS, deduplication, FlexClone and SnapMirror to the rescue. Really, all yous fanbois gotta be careful. I and my colleagues know both sides of the fence whereas anyone Jihading against SAN is either a liar, stupid or knows only the Exchange side.

SAN is not expensive. We believe that http://blogs.netapp.com/msenviro/2009/06/fas-the-new-das-using-fas-in-a-das-configuration-for-exchange.html showed that quite clearly. Even with a SAN we have shown that Microsoft will use nearly 4,000 disks to deliver a solution that NetApp does in less than 600. Oh, and NetApp deliver more IOPS on their 600 than Microsoft's 4,000. Oh, and Microsoft deliver less available storage than NetApp. Yeah, SAN is really more expensive. STFU.

And SAN replication? Microsoft recognise that storage vendors have a very solution on their side of the fence so have opened APIs to allow EMC, Compellent, NetApp and all the rest to plug in to that the DAG can be used across site without having to use the still somewhat bloated and inefficient transaction log shipping method built into Exchange.

Here's something that many haven't considered. So many people are deploying NetApp (and other) vendors' solutions for Oracle, SQL, File sharing and other things that the SAN is ubiquitous; indeed, the NetApp 2000 series is designed for those 50 to 2000 seat orgs and the partners can't shift 'em fast enough. If Exchange 2010 is so good on IO load we're just going to see people drop Exchange onto a DS4243 shelf of SAS and have done with it. We expect to sell more Exchange 2010 on NetApp than Exchange 2007 because now Exchange is merely a 'something else' to add onto the storage. Anyone who says that Exchange 2010 means the death of the modular SAN is a fool and hasn't done their research.

Oz and others like him are doing a disservice to the community when they say nothing more than Exchange 2010 = SATA = DAS. Sure, I work for a SAN vendor but I have a responsibility to all Exchange customers, more often than not to save them from Microsoft and themselves.

Update:
Oz published: http://smtp25.blogspot.com/2009/09/what-is-business-justification-going_08.html today. It's a pretty good article and I've posted some positive comments along with it to show where we are going with the solution.

CCR and NetApp – Design Questions

2009-09-08T12:24:00.001-04:00

The first thing to say here is that NetApp and CCR is not something you should consider if you are only looking for storage resiliency. NetApp will provide you more resiliency than you can shake a particularly large stick at when you host Exchange data on the FAS platform.
However, there are times when you absolutely must have your Exchange server up and running 24/7 with only the briefest of service outages. In this situation you use CCR so that you can run service packs and patches on the passive node, implement a failover and then run the patch on the (newly) passive node. At no point will you ever do a fail-back because that would be an unnecessary amount of outage for no valid purpose. CCR nodes are supposed to have identical processing and memory capabilities and are supposed to have the same disk (read I/O) capabilities as each other. If you are designing a solution and have to consider different server and disk capabilities you clearly have less of an SLA than you think you do and shouldn’t be considering CCR in the first place; SCC is the thing you need.
Moving on then to your back-end storage design if your NetApp representative (your friendly neighborhood CSE’s (Luis, John, John, Steve, Jen and Mark – no Paul, George or Ringo though)) has nodded sagely and said CCR is the solution you need. The controller design changes a little bit. With SCC NetApp will always say that you need a clustered FAS platform. In the unbelievably small scenario that the controller takes a nosedive you want to make sure that the storage controller fails over to its partner so that you are not affecting the dozen or so servers that are interacting with the disks through that controller. NetApp failures from head to head are in-line with Microsoft timeouts so the Exchange will not barf if a controller does. Note that fail-backs are always done under controlled situations and are designed to take longer.
But if you have two copies of the storage do you really need all that redundancy at the back-end? The answer is often “Yes” because a FAS controller head will happily serve 50 to 60 thousand active users (8 or nine beefy servers hammering the kit). You don’t really want a controller to fail causing some (half perhaps?) of those servers to fail over to their CCR passive partners. The right way to do a CCR environment where the disk is provided by a storage network is to have a total of four heads, two clusters and each clustered pair serving either the Active or Passive Exchange servers.
Sometimes though, because you want to take advantage of the NetApp software, the answer might be “No”. Perhaps you might want to do a CCR/NetApp design that does not utilize clustered storage pairs but single heads. If the worst happens and you have a head failure then all that will happen is that the Exchange servers will fail over to the second CCR node. Logically in this case you’re going to have each FAS head manage half of the active mailboxes and half of the passive, so there will be 30,000 mailboxes or so that have to tip over. Obviously all nodes will lose any further failover capability until the storage controller is fixed.
Whilst I, as a NetApp CSE would want you to use NetApp clusters all over the place because I am a commissioned employee it’s also my responsibility to sell the right thing to the right customer to meet the right business requirement. If the Exchange admin is a crazed geek it’s my job to talk him down to reality and use our shiny software to achieve what he wants to achieve in hardware alone.
Perhaps it’s the economy but we are doing a number of deals where CCR on the local site on single heads is backed up by SnapMirror on the remote site rather than SCR. Gone are the days where the simple A/P NetApp cluster can be assumed; the redundant head is quite a lot of money to have sat there doing nothing.

Elmo Goes into Politics

2009-09-01T15:26:00.001-04:00

Kathleen Sebelius recruited a new deputy at HHS.

Elmo will be curing flu.

TV Re-runs

2009-08-25T12:55:00.001-04:00

Opera, according to Drop the Dead Donkey's very own Henry Davenport:

Fat tart comes on, fat tart sings, fat tart dies.

Must-Have Exchange Book

2009-08-24T20:09:00.003-04:00

Jim McBee's recent blog post reminded me.
I've had the book a while now and made sure that any customer I was in front of recently had the name and book title of the rockin' book, Exchange Management Shell by Ilse Van Criekinge. So far I'm 100% and the Red Spectacled Lady From FlemLand is probably the princely sum of about five Euro's richer for the royalties. The books are positively flying off the Amazon warehouse shelves on the eastern seaboard.
Since I have no imagination I always found the learn PowerShell books very challenging and had to go through those books to learn syntax and then the Exchange 2007/2010 help to learn the commands and exactly where to put the millions of quotation marks so that the command wouldn't collapse into a failed heap. Ilse helped by combining three things into one. Perfect for the Exchange professional.

And not long now until all the author copies of the AD cookbook are out of the utility room. I've Fedex'd a fair number of copies around PA, NJ, FL and MA.

Exchange 2007 and DAS - The Truth

2009-08-24T15:42:00.003-04:00

Just compared a DAS solution for a customer to a NetApp SAN solution. Let's take a look at those numbers.
The customer went (hasn't but it's an option) for 15 active CCR nodes and 15 standalone SCR nodes. They decided that HP MSA 70 was the way to go and four shelves - 100 disks would be attached to each node. So there were three copies of the Exchange data attached to three servers; 300 disks. Multiply the 15 by 2 by 100 (for CCR) and you get 3,000 disks. Then you multply 15 by 100 and add it to the original number; 3,500 disks.

At no point in those calculations do backups get taken into consideration. Each of the 15 nodes can have up to 7.5TB of storage (let's be kind and say 3.8TB of data since it's going to be RAID0+1) so that's 57TB of day zero backups that's going to have to be on some disk somewhere because you can't spool that lot off to tape in a hurry.

Approximately seven NetApp 3170C systems containing a total of 1,100 450GB disks, uses 1/3 of the raw disk numbers AND there are SnapShots included in that. There is SnapMirror to a DR location and there is a NetApp 3104C with a ~80 of 1TB SATA disks hanging off the end of it for 30 days backups (all backups die after 30 days, never to be needed again) At an upper end of the scale the NetApp solution will use ~1500 disks, provide 2 days local snaps for rapid restore and 30 days snaps in SnapVault for just-in-case the auditors come asking questions.

Oh, and the customer gets to use FlexClone to test service packs and patches against live data. Oh, and the customer gets to have multiple roll back points. Oh, and the customer gets to de-dupe their SnapVault destination (the only way to do that in MS DPM is with NetApp as a backend (on which topic we have a paper)).

And DAS is less complex? Are you looking at the back of the servers by any chance? SCSI spaghetti anyone?

And DAS is cheaper? STFU. Just how much do you think all that floorspace, power and cooling to chill all those disks is costing.

Oh, and if you wanted to turn those 3170's into a DAS-esque solution you could. In fact it would be damn easy to do.

Your Exchange 2010 won't work right?

2009-08-21T14:53:00.002-04:00

You've installed Windows 2008 R2 and have the Exchange 2010 RC in your sticky mitts?
Then you will need to make sure you run "winrm quickconfig" from the command line and say yes when prompted.
If you don't you'll run into horrific PowerShell problems, more so than the problems you're going to have to endure on a day to day basis anyway.

The Intertubes are awash with it right now since the Exchange installation process has skipped that little step. If you use The Google you'll see how it's going.

Snap-what Now?

2009-07-26T21:07:00.008-04:00

Following on from my hugely successful foray into Flex* which surely someone must have read, it's time to have a look at the other ubiquitous NetApp-ness, that of Snap*. Let's see how it affects the daily life of an Exchange administrator and, to some extent, an Exchange architect.

SnapDrive. A shiny bit of software that runs out-of-band (i.e. you don't actually need it to connect a NetApp LUN to your server). this software allows you do expand and contract the LUN (i.e. drive) without bouncing the OS. You can create, modify and delete LUNs on Volumes that have been given to you by Mr Storage.

SnapManager. SnapManager for Exchange, SQL, SharePoint, SAP, Oracle and Domino (yes, there's one for Domino) does need SnapDrive to run because the SnapManager is little more than the fancy front end or scheduler or whatever for the back end storage. It's clever though, SnapDrive tells it all fancy stuff about the underlying storage and SnapManager offers you lots of features based on all that gleaned information.

SnapMirror. This is the cool bit of the back end that mirrors a volume, and in the case of Exchange it's a LUN inside the volume, to some other location. It's fiendishly clever because it replaces Exchange 2007 SCR and in some cases Exchange 2010 DAG copies. If you use SCR you can't do anything with the destination. If you use SnapMirror you can mount stores at points in time to do testing, all without causing the replication to pause or generally go awry.

SnapVault. Lots of people like to maintain old copies of the volumes containing the Exchange data so that they can get discovery operations underway without using tapes. Sure, you still need stuff like DigiScope from Lucid8 or the similar products from Appassure to do your discovery but if you can do that discovery without restoring hundreds of gigabytes of data you can do your discovery action far faster. Oh. And you can deduplicate that data so not only does it take seconds to mount up a LUN to find stuff, you're doing so from core data that's only a fraction of the size it started out at.

SnapLock. When you need to provide M'learned friends with data that is certified not to have been tampered with you retain the archive information in SnapLock volumes. Again with the beating tapes. Again with the beating actual Worm media. Shiny.

SyncMirror. OK, not exactly a Snap* on account of it not having Snap in the name but it kinda goes here. Anyway, wonderful stuff. Perfect for occasions where you just have to waste vast amounts of cash because you simple have to have RAID6+1 protection for your data you can use SyncMirror. SyncMirror lies at the heart of MetroCluster, a well known product in Germany who love it to a fairly unnatural level for some reason.

Flex-what now?

2009-07-26T15:21:00.003-04:00

I did say that NetApp was all about Flex-This and Snap-That. It's pretty bewildering to those inside the organization, let alone those without. Allow me to bring clarity to Exchange professionals using NetApp.

FlexScale is the software licence you PAY for when you get the Performance Accelerator Module (PAM card) which is technically, kinda, FREE. Pay for licence, receive hardware free. This is the solution that allows disk metadata (amongst other scenarios) to be loaded onto memory on the controllers so that a read from disk is faster. Remember that all reads from disk are two operations; one to find the location of the data and one to get the data. The PAM puts the location of the data into memory so that the controller only needs to go and get the data. The PAM card is shiny because it allows you to use larger capacity, slower, SATA disks rather than FC disks. The card is 16GB today and 256/512GB modules are due out soon. You can have multiple cards in the controllers depending on the specific FAS model.

FlexShare is the FREE bit of software that allows you to assign priorities to various volumes. You would look at using this if you wanted the spindle count and had your logs and stores in a single aggregate and were still experiencing problems with log queues. Normally the only way out of this hole is to add yet more disks. You have already done a good thing by maximizing the number of disks you've given to the logs but now you want to tweak the write priority without adding disk and thereby unnecessary capacity to the solution. It's an excellent disk avoidance measure that you simply cannot do with DAS based solutions.

FlexCache is something you do not need to worry your pretty heads about because it's for NFS.

FlexClone is something else that you PAY for. You use it to make zero space clones of the volume(s) containing Exchange Server information (or anything else for that matter) so you can present it to alternative hardware and do proper, realistic, SP and patch testing. Without FlexClone you either have to do the dumb things (that we have all done) by "testing" a Service Pack against a server. How dumb is that? You've got a Windows server on your hardware, you put Exchange on it and you "test" the SP. What? Were you expecting it not to work? With FlexClone you create your entire production environment, present it to DR or lab hardware and you test real data. When you're done you tear it down. It takes minutes to set up and minutes to tear down.

Next will be a Snap* post, but not today.

Exchange and NetApp SnapMirror

2009-07-10T17:37:00.004-04:00

Here we go.
There was a thread today on one of our myriad internal DLs asking for some advice and background information after one of the esteemed members of Microsoft Consulting Services in a country (to remain anonymous) told a customer that Microsoft does not support SnapMirror.

The MCS guy was, not to put too fine a point on it, talking shite.

Here's why.

Microsoft does not, indeed, support replication of live data twixt arrays.
NetApp does not replicate live Exchange data.
NetApp replicates VSS captured information - i.e. out of use blocks of data.
To say that Microsoft do not support SnapMirror is EXACTLY the same as saying that Microsoft does not support Microsoft Data Protection Manager backups because the underlying technology of how the data was captured (VSS) is the same damn thing!

Clearly, the MCS individual has given the customer a bum-steer. Wasn't the first time, won't be the last time. Sadly.

Information @ Microsoft about SnapMirror.
1. http://www.microsoft.com/presspass/events/HHHlaunch/docs/netapp.doc
2. http://www.microsoft.com/casestudies/casestudy.aspx?casestudyid=4000000127
3. https://solutionfinder.microsoft.com/Solutions/SolutionDetailsView.aspx?solutionid=fd0f4e474c294eb78a548eac2130e279&Partnerid=eaebc712-b043-445f-a925-33249e9c4edf

All of which will attest to the fact that SnapMirror is a supported solution for Exchange and indeed other Microsoft applications. There's even a white paper (blogged here some time ago) on DPM and NetApp which specifically calls out the SnapMirroring of data as cool-and-shiny (my words)

Additional information.
http://blogs.technet.com/evand/archive/2005/06/03/405864.aspx (point 4 is where you want to be)

NetApp, Exchange and Fractional Reserve

2009-07-08T16:11:00.009-04:00

One of the more foot-meet-bullet things that NetApp have done in the past is to recommend "100% Fractional Reservation". In common with a lot of things we repeatedly get thrown this by potential customers who are in favour of inferior SAN solutions and don't understand NetApp.

We have not recommended 100% FR for Exchange for a very long time. If you have any NetApp employee or vendor/partner that talks about using 100% FR you need to 1) get your NetApp operating system (Data ONTAP) up to 7.3 and, 2) tell the individual that you don't want that FR and are not playing the game.

But, why did we even have FR?

Safety. We wanted to ensure that even if every single block in a volume changed we could take a snapshot of it. In the old days there was no feature in ONTAP called 'autogrow' so there was a chance that if every block change, come snapshot time, the system would run out of allocated space and take the LUNs offline. Not great with the whole Exchange availability story going on there.

Now that we have moved several versions on and have additional features such as thin provisioning and volume autogrow and snapshot autodelete we don't need to reserve that extra safety space which was, it has to be said, costing us deals to the piddly side competitors such as LH, EQ, Pilar and 3Par. Loss of revenue is a great motivator.

Never in a million years will every single block on an EDB change in the few hours between snapshots. if they do then you have Exchange problems before you have storage problems. You're going to see log growth explosion and you're probably going to take the store down before you fill up your log LUN. The possibility of taking the LUN housing the EDB offline is diminished if you think about it in a logical manner.

So then, what the heck is Fractional Reserve anyway?

Ah, well, that's one of the most complex things I've had to deal with and frankly I can't explain it without a) getting it wrong and b) using pictures. Essentially NetApp has a feature that will reserve space within the volume, which is available for snapshot data. There is a competition though. Data growth and snapshots vie for space within the confines of a nominally fixed size volume. FR is there to ensure that you cannot add data into a volume that would result in the complete filling of that volume should every block change between snapshots. Thought of another way, an initial 1GB of data actually needs 2.2GB of space - if you have Snapshot reserve of 20% (the default) - just in case. It's the just in case part that's a) pointless in Exchange and b) commercially damaging. You want to ensure that you can always take a snapshot of the current data and the only guaranteed way of achieving that is to reserve 100% of the space that is being taken up by data, not including that space already reserved for snapshots (that 20% again). This is why you hear "Two x plus delta" bandied around by NetApp old lags, and some newer people as well.

Previous versions of Data ONTAP wouldn't allow for either the automatic deletion of old snaps or the automatic growth of a volume to ensure the maintenance of equilibrium. Result? LUNs offline. Red faces. Pain. Stress.

It's easily countered by a number of things.
1. Competent monitoring - so that you know how much space snaps are taking and what the change rate is. We monitor and report that. All you have to do is read what you're being sent, plan and act accordingly.
2. The latest software. 7.3 is mature.
3. Not mixing data in a volume. Exchange best practices currently say: One EDB in One LUN in One Volume. Period. Volume breaches were often due to people shoving stuff where they shouldn't.
4. Implementing volume autogrow and snapshot autodelete as necessary.
5. Thin provisioning. If you are going to allow a store to grow to 200GB EDB in a 240GB LUN there's no need to guarantee the whole 240GB plus 48GB (20% snaps), i.e. 288GB of space, from day one and then saying another 100% FR on top - grand total 528GB. The EDB will grow steadily to 200GB and the snaps will be between 5 and 8% per day. Thin provision it and add physical disk when necessary. Competent monitoring will see you right, and enough customers use it to ensure that you are not alone.

Bottom line folks. FR = good. FR = safety. FR = unecessary. FR = commercial suicide. FR = 'the idiots safety net'. Whilst "You can't fix stupid", you can mitigate it. If you are in an environment where someone might do something incredibly dumb you need NetApp and you need 100% FR. If you're all competent then you just need NetApp

Love Fest Continues - Oh The Horror

2009-06-17T07:59:00.005-04:00

Earlier I brought news about EMC and NetApp putting a joint article together pointing out the shinyness of NFS and VMware. Not to be out-done Microsoft and NetApp have cosied up and are jointly billing and cooing regarding Hyper-V, here. Microsoft have long been the red headed stepchild in the attic with regard to virtualization and, to be honest, Hyper-V v1 remains a bit of a joke. But that's all about to change with the forthcoming release of Windows Server 2008 R2. Microsoft have worked out that doing things on a SAN can bring them benefits and shift a helluva lot more management licenses (Hyper-V itself being free) if they play nice with people who know what they're talking about, own the disks and want to give the customer a good, innovative and efficient service.

Exchange team, are you listening?

Well done to Steve Rackham in NetApp Blighty, Chafee McKenna in NetApp Bellevue / Redmond and Taylor Brown from the Microsoft Core OS division and a bunch of others not in Philadelphia.

Exchange on NetApp FAS - The new DAS

2009-06-16T12:14:00.007-04:00

First thing to do is to take a look at this from John Fullbright. What does it mean for you?

In between all the dry statistics there are some key facts:

1. The FAS2020, HP and Dell storage "cages" all came in within $1,000 of each other - servers not included since they're a constant.
2. The FAS gave you more spindles and capacity. This means that you got to drive more IOPS from the storage system with the same number of disks and shove more data onto it. It's NetApp Storage Efficiency baby.
3. You get the hardware VSS backups built into the storage platform and we bundle the necessary host based software with the 2020.
4. You don't have to worry about RAID0 and implementing multiple copies in the DAG. Sure, you're probably going to do some, but you've now got access to RAID-DP and can put aside your disk resiliency worries. You can still do DAG copies in place of E2K7 SCR scenarios.
5. And here's the kicker. If you find that your FASDAS is not working too hard you can expand it some more. The 2020 will give you 68 spindles or up to 68TB of storage, whichever comes first. This means that you can DAS attach a couple or three servers to the 2020. If you only have the one Exchange server you can get rid of your Windows file server and use the 2020 as NAS as well. Don't worry, Microsoft will still get their money for the CIFS licensing so it's not even a competetive situation.
6. You can connect the servers up with FC and the NAS up with Ethernet, or do both with Ethernet.

So there you go. Bottom line is that anyone who tells you that DAS is cheaper than SAN simply doesn't understand the key financials or commercials.

NetApp & EMC Virtualization Love Fest

2009-06-09T13:38:00.004-04:00

Sky not falling. Visigoths not coming over hill, End of Days not scheduled, Populist revolt not imminent, Wolves not at door, Earth not yet surrendering its dead in Zombie uprising, Rome not burning and Revolution (orange, velvet or glorious, televised or otherwise) not taking place:

EMC and NetApp with NFS at NetApp and the same article at EMC.

Exchange and NetApp ? RAID-what-now?

2009-04-17T11:32:00.003-04:00

A lot of people in the Exchange world don't understand RAID too well. Luckily the nice people at Microsoft have cleared that up for you and graciously they have included RAID-DP and a fairly flattering description of the same. http://technet.microsoft.com/en-us/library/bb738146.aspx

SharePoint, NetApp & Hyper-V Webcast

2009-04-02T14:56:00.002-04:00

There's a TechTalk Webcase coming up showing how users get to use Hyper-V, De-duplication, SnapManager for SharePoint and and NetApp storage together.
http://communications.netapp.com/p/Network_Appliance/20090409140000WL?REF_SOURCE=netappsite

Trust me, it'll blow your mind how we get 27 pints of Stella into one person. (UK in-joke)

Oh, the shame of it

2009-03-19T12:43:00.003-04:00

As part of the migration of my mobile phone bill into the corporate plan I have moved to a CrackBerry; an 8830 no less. Now starts the comedy to take something off the household bill and into the company.

I need to shower.

Well, ok. Admission time. It's 20 times better than my 5800 from Verizon which was dog slow, dog slow to reboot, dog dog slow once it had been running more than 15 seconds and generally pants. Whilst I suspect that, as someone in MS once said, it's more likely to be the vendors implementation that's crap rather than the OS, the fact that MS allows someone to make a crap implementation of their OS/Phone speaks volumes.

Announcing New Blog

2009-03-13T14:21:00.003-04:00

http://blogs.netapp.com/msenviro is live today. It's a way of letting anyone interested in Microsoft technologies and who wants to, or is already running, the NetApp platform learn about how things work together. It's also going to be a good platform to initiate some thought provoking debates as there are a lot of people out there who have an understanding of a lot of technologies but would like some pointers on why one working with the other might be a good idea.

The first post is by Robert Quimbey, the same name that appears as a contributor on the worlds most used spreadsheet; the E2K7 Storage Calculator. He's talking about a subject very dear to my heart; virtualization of Exchange mailbox servers. I'm one of a handful of MVPs (for however long that award lasts) who think it's a great idea when done appropriately. Obviously since I think it's a good idea they're all wrong and will eventually see the light, but I digress.

Robert looks about 12 years old, is an Exchange MCM and actually has to carry a special box around with him because his brain is too big to fit inside just one head. Ahh the young, with their hair and the clothes and the rock & roll.

Little Jimmy Rides again

2009-03-07T20:05:00.001-05:00

http://jimmyontour.blogspot.com/
This time, on a sled.

The SAN, Solid State Disks and Exchange

2009-02-12T19:34:00.004-05:00

Here goes.

I recently had cause to talk to a customer about solid state disks. Another vendor had been doing their dog and pony show about how wonderful SSD is and that they would get blazing fast performance on their Exchange databases. All of that is completely true, briefly.

I say briefly because putting Exchange on SSD will eat your entire IT budget for little tangible gain. Let's take a look at some good old common sense facts.

By far the larger part of an Exchange database is made up of stuff that users will never again access but want to retain it just in case. Sure, there are exceptions, caveats and a bunch of other factors but when you boil it down it's a fact that the EDB is largely full-o-junk.

SSD disks have a finite number of writes that you can make to it. Check it out.

SSD disks deliver blazing, insane IOPS but the user can't call messages that fast and the disks can't hold enough data to cater that I/O potential. What's the point of storing stuff on something that's expensive and can sometimes be described as "too fast"

Exchange has no capability to put the most recent parts of its EDB onto SSD and the rest of it onto a different tier. To get that (not the SSD part, but the tiering part) you will need Compellent. Heck, did I plug a competitor product? Eww. I must shower.

But if you did have SSD, or Flash as it's often described, do you put your stores or your logs on it? Neither. But you knew that. Logs are generated by the server at a (generally) steady rate. They just don't get generated quickly enough to warrant being put onto SSD. In addition, the finite write limits are going to wreck your SSD far too early. Financially, it'll kill ya. So then, put the stores on it. Well, still more with the no. I will not sell you shed loads of non-spinny simply because it's the new best thing on the market.

So what do you do to improve performance?

Well, it will come as no surprise to learn that NetApp have one answer. Whether it's the right answer isn't the point here, but it's an answer that is getting traction and seeing results.

NetApp have the Performance Accelerator Module. It's a card with a shed load of RAM on it that slots into the controller. It reads in 'all' the disk metadata so that whenever the box has to do a read from disk it only has to hit the disk once. The FAS reads the block location of the data from the information in the PAM and then does a single read to retrieve the necessary data. Stick a PAM in each node and save yourself (about) a shelf of the spinny stuff, which is great news on loads of levels. There's a 2nd generation PAM coming out relatively soon with all manner of shiny improvements which I'd like to talk about but best not because I don't know what's internal, partner only and public.*

*I suddenly have new respect for Microsoft people on that score. Whereas I often felt that the whole MVP-getting-early-information thing was relatively pointless because for the past eight years everything product-wise I learned as a result of being an MVP I found out days later through TAPs or other means. Now, having contact with the best brains in the Exchange world to learn the how and the stuff that white papers and KBs don't tell you is a whole different story!

Carry on.