Wednesday, December 17, 2008

You want to convert a machine to a vhd

So, for some reason you want to convert a machine to a vhd. Let me not limit your thinking by saying that this has to be a physcial machine, it could be a VM.

I am going to cover two items: XenConvert (free, converts running machine), and a Hyper-V feature (part of Hyper-V, converts any mountable disk).


Using Hyper-V:

With Hyper-V you can take a physical disk, attache it to a Hyper-V host and convert the disk (fully intact) to a VHD. A neat feature that no-one talks about.

This is great for a number of scenarios where you have a physical drive, a host, and the machine on the physical drive can be is is offline (we will assume that this is a disaster recovery exercise).

But those are the limitations. Still, a neat feature.

Here's how:
Mount the disk to a Hyper-V host. (SCSI, in an external enclosure, ATA, however)
Be sure that the disk is attached and recognized in Disk manager
Open Hyper-V manager
Select Actions -> New -> Hard Disk..
Step through the wizard to select your type of virtual disk, the location, then stop.
On the Configure Disk page, select the option "Copy the contents of the specific physical disk"
Then complete the wizard.

This then copies the contents of the desired physical disk (not just the files, but the actual blocks, formatting and all) to a VHD.

That is it.

Another option is to

Use XenConvert:
Download the XenConvert tool (You can download it from MyCitrix (@ Citrix.com) where you download XenServer)
Install it within the Operating System that you want to convert (again, physical or VM).
Then begin the conversion process choosing VHD as your output.
You can just convert an existing installation to VHD (it can even route the VHD to a network share using a mapped drive).

the benefit of XenConvert: You machine is running the entire time.
I have converted SQL, Operations Manager, and SCVMM servers to VHDs (yes, they boot and everything) then moved them to either Hyper-V or XenServer.

I hope you get creative with your options.
You can convert physical to virtual, virtual to virtual, etc. No need for expensive conversion tools as long as you have the time and the knowledge.

Tuesday, December 16, 2008

Should I put all my VHDs on one volume?

I recently responded to someone who was considering placing all of their VHDs onto one huge storage volume.

This might be easy for storage management and for backup management. But you have to consider the ancillary impacts that this might have on the overall environment.
The greatest impact will be the impact that each VM has upon another VM.

This is where we need to h ave a full understanding of all of the components involved in our virtualization infrastructure and how all of those components interact and affect each other.

Here is the statement that began the idea for putting this to blog:

"We were hoping to create a "big" LUN, store all our Hyper-V Guests on that LUN, assign the LUN to several HOSTS, and access the Guests that way."

Now, the response:

That is all fine, good, and noble - but you may want to reconsider.

Even VMware has had the recommendation (for a great number of years now) of no more than 10 VMs per LUN (they consider this a 'rule of thumb' that is taught in class).

Why? Disk IO.

You have to balance the disk IO demands of the VMs stored on the LUNs.

If you have VMs with a bunch of disk IO all happening at the same time they end up impacting each other.

For example - you always split out your SQL databases from the temp log, from the OS - for the same reason.

In this case you have different considerations - VM1 will impact VM2.

An easy test you can do at home:

With many VMs on a single LUN, boot one and wait for Integration Services to report some data from within the VM (Integration Services loads late in the stack so it is a good measure that the VM is nearly done booting).

Be sure to time it.

Now, repeat the test, this time with 4 VMs all at once.

Don't stop timing until all four are up, then get your average time to boot.

All that disk reading and page file building created massive disk IO.

Just something to think about as you plan your infrastructure.

Thursday, December 4, 2008

Dedicating a physical NIC for Management with HyperV

Here is an item that comes up in enterprises. Dedicating Hyper-V management traffic to a single physical NIC. This generally also includes disabling the Host access to an External Network Switch.

BTW - the Microsoft recommended configuration is a dedicated Host management NIC that is not associated with a virtual switch.

I have pulled this from a real example that I helped some with and have tried to protect the innocent.

My assumptions in this example:

Two NICs – ‘HP NC373i Multifunction Gigabit Server Adapter’ and ‘HP NC373i Multifunction Gigabit Server Adapter #2’

You are not local to the servers

In Network Connections you should see the following Device Names:

  • HP NC373i Multifunction Gigabit Server Adapter
  • HP NC373i Multifunction Gigabit Server Adapter #2

Microsoft Virtual Network Switch Adapter

An External virtual network exists that is named "InternalEthernetPort" and it is using NIC 1

What we need to do is:

1) make sure that 'HP NC373i Multifunction Gigabit Server Adapter #2' is not bound to a virtual switch

a. In the Hyper-V Virtual Network Manager there should only be one Virtual Network

i. It should be named “InternalEthernetPort”

ii. It should be of type External

iii. It should be bound to physical NIC ‘HP NC373i Multifunction Gigabit Server Adapter’

2) make sure that 'HP NC373i Multifunction Gigabit Server Adapter #2' is enabled and on the foo.bar.com network

a. This should show the Hyper-V host as multi-homed as it would have two NICs on the same network. And may produce an alert. It should have two IP addresses on the foo.bar.com network.

3) Connect to the Hyper-V host on the second NIC - 'HP NC373i Multifunction Gigabit Server Adapter #2'

4) Disable the Hyper-V virtual NIC that is attached to “InternalEthernetPort”.

a. Do this through Network Connections - Control Panel -> Network and Sharing -> Manage Network Connections

b. Find the NIC whose Device Name begins with “Microsoft Virtual Network Switch Adapter”

c. Select -> Right click -> disable

i. I am sure that you can predict what would happen if the order was wrong, or the wrong NIC was selected.


The end result should be:

  • VM network traffic is using switch “InternalEthernetPort” which is using physical NIC ‘HP NC373i Multifunction Gigabit Server Adapter’
  • Host management network traffic is using physical NIC ‘HP NC373i Multifunction Gigabit Server Adapter #2’
  • This should maximize the host management throughput.

    Oh, as a side note - Server 2008 R2 with Hyper-V makes this a lot easier and less complicated…

Tuesday, November 18, 2008

The feeling of success-VMware Server 1.0.6 on Ubuntu 8.04 x64

Until this moment I was starting to think to myself: "why do I put myself through this pain?" "Why do I attempt to install Linux and Linux applications every now and then?"

Then, suddenly, it works!

And you feel that you conquered the world.

Okay, I am writing this for myself as I know that I will have to do this again in the future.

The task at hand: Getting VMware Studio to work properly.

My personal opinion about the entire adventure: This application was obviously developed by a person that is hardcore Linux and considers Windows a scourge on the landscape.
I arrived at this opinion after noting in the readme that the OpenSSH engine is not known to work properly for the installation on any operating system other than WinXP.
(at that moment the software tester in me said: "duh" - security enforcement is next to nothing)

Anyway - back to the task at hand.

Installing VMware Server (1.0.6) on Ubuntu 8.04 (64-bit)

First of all - I would like to thank the Ubuntu community for all the documentation. Tons of it. My only problem, it gets really confusing and hard to follow. I ended up with instructions from three sources and even dribbled this down into its most basic steps (fewest possible).

Second, if you need the gory details - please reference the links that I note.

Let us begin:

Download and install Ubuntu.
During the install, enable the OpenSSH server.
Oh, and set a root password (I found it all goes better in the end - as VMware Server wants you to use root anyway)

sudo passwd root

Then go here: https://help.ubuntu.com/community/VMware/Server/AMD64
and do the following (you only need to use sudo if you are not logged in as root):

sudo apt-get update
sudo apt-get upgrade

sudo apt-get install build-essential linux-headers-`uname -r` xinetd
sudo apt-get install build-essential linux-headers-`uname -r` xinetd gcc-4.1 g++-4.1
sudo apt-get install ia32-libs libc6-i386

Then download VMware Server and put in under /tmp

tar xvzf VMware-server*.tar.gz
cd vmware-server-distrib
sudo ./vmware-install.pl

Please read the prompts.
There is a build error at one point, do not accept the default no, say 'yes' else you will be stuck and install fails.
Also, I freely declined VM NATting - the default is on. why? I ask.

Now, before installing the Server interface we need to fix something, else the MUI install will fail.
https://help.ubuntu.com/community/VMware/Server

mv /usr/lib/vmware/lib/libgcc_s.so.1/libgcc_s.so.1 /usr/lib/vmware/lib/libgcc_s.so.1/libgcc_s.so.1-old

Now, we can download the VMware Server MUI and put that under /tmp

tar xvzf VMware-mui*.tar.gz
cd vmware-mui-distrib
sudo ./vmware-install.pl

After install completes the service will not start properly, guess what? Another fix, because the startup script is broken.
http://www.go2linux.org/vmware-server-on-debian-etch

obtain the patch from here: http://users.piuha.net/martti/comp/ubuntu/en/httpd.vmware.diff
and place it under /tmp

cd /
patch -b -p0 < /tmp/httpd.vmware.diff

Then start and query the Management service

sudo /etc/init.d/httpd.vmware start
sudo /etc/init.d/httpd.vmware status


Now, on your desired remote client machine be sure to download the VMware Server Client package and install it.
Only the latest VMware Server 2 has a built in web management console.
(and its installation is supposedly far less painful)

Wednesday, October 15, 2008

Virtual Machine Interoperability with Hyper-V and XenServer using OVF

Okay, I know this will sound a bit like marketing hype, but I hope to keep that to a dull roar.

For the past few months I have been involved with a virtual machine interoperability project called: "Project Kensho"

Project Kensho uses virtualization interoperability standards that are currently being developed by the DMTF (Distributed Management Task Force - the people who are bringing us lots of things such as CIM and DMI) to provide a framework for all of the virtualization vendors to use to talk the same language.

You can check the whole thing out here:
http://community.citrix.com/display/xs/Kensho

And..you won't see a bunch of screen shots, because I already did those (they are at the link above)

What does this mean to the administrator that is in the grind all day long?

You can take a VM and move it from one virtualization host to another.

You can export an entire enterprise application environment (some complex application that needs SQL, IIS, Active Directory - and those components must run on individual machines).

You can import items like these from other sources.

Here are some real world examples of what I use this for today:

Exporting a complex or large test environment.
I have an environment of 30 client virtual machines. I can export that to a package, sit on it, rebuild my hypervisor, import it back. I can also import those virtual machines to either XenServer or Hyper-V.
I also have an environment of 4 servers (the virtual disks are 36 Gb each) - but I can export and store and then import and use.

It actually ends up making my life a bit easier.

Go ahead, check it out.

Monday, September 29, 2008

Securing Hyper-V WinRM with a Self Signed certificate

It is funny. When you are working on things you just take some things for granted.

After all, you would expect that generating a self signed certificate for use with testing communication with Server 2008 would be some sort of built-in function right?

I mean if you Google around you can find all kinds of instructions, but they all have one big general assumption – IIS is installed. I don’t know about you, but I am a purist on my servers, a service does not get installed that is not being used. I refuse to install IIS just to generate a self signed certificate (oh, Exchange does this for OWA too).

Considering that this article is about securing WinRM (Windows Remote Management) traffic with SSL on Hyper-V – why do I want to install IIS on my Hyper-V host just to set this up?? This was my challenge.

I have checked out the Microsoft instructions for securing WinRM: http://technet.microsoft.com/en-us/library/cc782312.aspx

I have two comments – difficult to read and tailored to Server 2003. They are not ‘no brainer’ type of instructions and there are easier ways to do this. So, for my notes I decided to document this, and did it here to share.

    The assumptions:

    • I want to use a self signed certificate (No need to mess around with a Certificate Server or IIS when all you want to do is some quick testing over SSL).

    • I want this to be simple.

    Okay, how to generate a self signed certificate on Hyper-V (Server 2008).

    The tools:

    Not a simple right click anywhere. Days of searching turned up little. The easiest way to do this is: Install IIS – didn’t I say no to that already?

    I discovered two ways to generate a self signed certificate that met my needs: MakeCert.exe and SelfSSL.exe

    MakeCert.exe is for testing only and comes in Visual Studio – any developer would jump on this in a heartbeat.

    SelfSSL.exe is a part of the IIS 6 Resource Kit. – Resource kit = administrator friendly.

    I settled on SelfSSL because I found better documentation and it did more work for me.

    Click on the link above, download the resource kit, and run the installer to extract just SelfSSL.exe, copy it somewhere convenient.

    I have SelfSSL, now what?

    (You may notice that I have more instruction than is necessary – that is for the curious among us)

    Open a command prompt (don’t leave the prompt, we will be here for a bit).

    Check your WinRM listener status winrm enumerate winrm/config/listener (this will list any listeners that are already configured – if you ran winrm quickconfigure at any time you will see an HTTP listener)

    Switch to the folder where you placed SelfSSL.exe

    Go ahead, type: SelfSSL /? (you know that you want to)

    1.SelfSSL

    Note the default behavior. If you only typed SelfSSL you would get a certificate with your local machine name (“ITProctology” in my case), good for 7 days, with a 1024 bit key, bound to web site 1 (but I don’t have one..) and port 443. All fine.

    I want to generate a certificate using my local machine name, good for 60 days, and have it automatically registered into the Trusted Root Certificate Authority of my server.

    To do this I type: SelfSSL /V:60 /T (I am going to accept all the other defaults)

    I am prompted to replace the SSL settings for site 1 – say yes. (if you say no, the certificate is not created nor registered with the Trusted Root CA).

    2.SelfSSL

    Oh no, I have an error..that I can freely ignore, IIS isn’t here (no metabase) so no problem.

    Now, open up the MMC, add the Certificates snap-in for the Computer account.

    You will notice that you have a Personal certificate issued to and by your machine name.

    3.SelfSSL

    If you open the Trusted Root CA, you will see that it is registered there as well.

    4.SelfSSL

    Easy enough.

    Create the WinRM HTTPS listener

    Now we need to create the Windows Remote Management listener and bind it to the certificate.

    Stay in the Certificates MMC.

    Select the self signed certificate from the Personal store. Double click it to open it.

    Select the Details tab. Scroll down to the Thumbprint field and select it.

    Select and copy the Thumbprint in the details window so you can insert it in the next step.

    5.SelfSSL

    Return to the command prompt

    Type: winrm create winrm/config/listener?Address=*+Transport=HTTPS @{Hostname=”<the name of your server>”;CertificateThumbprint=”<Paste from the previous step and remove the spaces>”}

    *Be sure to remove the spaces from the thumbprint string

    This command creates a listener on the HTTPS port (443) using any / all network address of the server, and my SelfSSL generated certificate.

    Mine looks like this:

    6.SelfSSL

    After hitting return I get a confirmation message that the listener created.

    7.SelfSSL

    Modifying the Firewall rules (if necessary)

    Now, we are working with Server 2008 here. If I had run winrm quickconfig I would end up with the default HTTP listener and the built-in, inbound, firewall rule is enabled. Also, my server is in a workgroup so it is pretty shut down by the firewall.

    There isn’t a built-in rule for using WinRM with SSL so we will have to create our own.

    Open Windows Firewall with Advanced Security from the Administrative Tools menu.

    Right click the Inbound Rules and select New Rule – the new rule wizard opens.

    I want to enable a port (443 to be exact – remember no IIS, so this has not been opened for any previous purpose or by any other widget)

    Select the port radio button.

    On the Protocol and Ports screen select TCP and enter the specific port 443.

    On the Action screen select Allow the connection.

    For Profile I am going to select all three, since my server is in a workgroup.

    Give the rule a name and you are done.

    You should see your new Rule, and it should be enabled (green check box).

    Test your configuration

    This is always one of those steps that no one does. Actually, I totally forgot about the firewall rules until I did this.

    At your server type and connection and a command, such as:

    8.SelfSSL

    If I used HTTP I would have an error, since I only enabled an HTTPS listener.

    To check your WinRM service configuration:

    9.SelfSSL

    Accepting the certificate from the remote machine

    From my remote machine I need to accept the self signed certificate so that I can use it.

    Ay my remote machine I open Internet Explorer and type HTTPS://<my other server name> and after I moment I am greeted with a certificate warning.

    10.SelfSSL

    Select View Certificate.

    11.SelfSSL

    Select Install Certificate.

    Accept and import the certificate and key pair and complete the wizard.

    Now select Yes to proceed.

    And I get a 404 page not found error – perfect, exactly what I expect.

    Testing the connection

    From the client, open a command prompt and type the same command that we typed at the Server (above).

    12.SelfSSL

    It works. Now you have a working WinRM listener using HTTPS, nice and secure.

    Troubleshooting

    The biggest items that you can run into are:

    The firewall – we talked about that.

    WinRM security – I didn’t touch that.

    WinRM security might need to be modified to allow communication to properly flow; it totally depends on the situation.

    Basic security might need to be enabled. Hosts might need to be trusted. In the case of HTTPS encryption might need to be turned off.

    This could be an entire post all to itself. I will save it for later.

    .

    Thursday, August 7, 2008

    Hyper-V and Failover Clustering patch

    For all of you that have been grumbling about a few things regarding Hyper-V and Failover Clustering there is a little bit of relief.

    Jode Baretto (MSFT storage guy) just announced a new patch that fixes the following:

    • Changes to the virtual machine view
    • Changes to virtual machine actions
    • Allow for more than one virtual machine in a "Services or Applications" group
    • Add support for mount points or volumes without a drive letter
    • Changes to the virtual machine refresh action
    • Behavior changes if any node of the failover cluster has a disconnected virtual machine
    • Behavior change when you add a pass-through disk to a virtual machine
    • Behavior change when the parent differencing disk is not on shared storage
    • Volume path copy
    I don't know about you, but these are welcome fixes to me.

    You can find the patch here:
    http://support.microsoft.com/?id=951308

    Thursday, July 17, 2008

    Managing Hyper-V snapshots - the basics

    This came out of a blog comment that I thought needed some attention. Really basic; “what happens” type of stuff.

    (Although, I truely feel that a little time exploring the user interface and anyone could figure this out on their own.)

    I have written about snapshots quite a bit as Hyper-V snapshots are a new thing for most folks.

    Not that a snapshot is new (or a checkpoint for those of you using SCVMM), but the way that Hyper-V does it is definitely different.

    You can find those previous posts here, here, and here.

    Anytime you think Hyper-V snapshots, always remember that the integrity of the timeline must be maintained. That is key to grokking the concept.

    Now, let’s begin.

    First of all, you can create a snapshot ( wow, amazing stuff ). This creates a differencing disk that reverences your base VHD, thus you always have that base VHD to return to, and saves your current memory status (if your VM was running).


    Second, you can Apply a snapshot. This takes me back to the point in time that is represented by the snapshot I selected. It does this by the same process as above, it creates a differencing disk and attaches it to the VHD that is referenced by the snapshot.


    Okay, if I am losing you – you need to back up and read the posts I referenced above.


    For the next couple you need to think about trees and the branching of a tree structure. The simplest way to show this is a single picture of an expanded tree (visual aids are always good).



    Third, I can delete a single snapshot. This deletes the moment in time that a snapshot represents, its differencing disk and any saved memory state. If a snapshot is ‘below’ this one on the subtree, then its reference pointers are modified to reference the proper snapshot ‘above’ it.
    In this example I deleted Snapshot B.1

    There are other things happening here as well. To maintain integrity of the snapshot timeline the differencing disks need to be merged. This happens in the background when a VM is powered off and I will save the gritty details for another post.

    Fourth, I can delete a snapshot subtree. This deletes the moment in time that a snapshot represents and any other snapshots that are ‘below’ it in the tree. Thus it does what a delete does, but without selecting individual snapshots, it takes an entire branch. Yes, we get a merge, not a revert as I mistakenly put in my graphic.


    Now, to get started on the really complicated post..what happens under the hood.

    I have alluded to this before with the instructions about how to manually merge VHDs. Maybe I need to back up even more and talk about differencing disks...hmm..

    Friday, July 11, 2008

    Unable to template Server 2008 virtual machine with SCVMM or deployment fails unable to customize

    I struggled with this myself, and it is an issue that an administrator rarely has to deal with anymore.

    Any administrator that has had to set domain password secirity levels and enforcements, knows what I am talking about. Password security policy.

    Let's back up and talk about VMM first.
    My references are going to be specific to SCVMM 2008 (since that is what is new and improved - this can apply to VMM 2007 in some cases).

    In VMM there are two ways to create VM templates.

    One is to select the Library view, Click the New Template action to launch the wizard. Select the button to use an existing virtual machine.
    After you finish the wizard, the VM is prepared to be a template. One of these steps involves using the local administrator credentials that you provided and actually modifying the VM with a blank local administrator password and running sysprep. This way, on deployment of the template, there is a blank administrator password that allows the injection of the OS Profile (sysprep answer file - sysprep.inf or unattend.xml) to finish the deployment.

    The other way to create a template is from an existing VHD that is already stored in the Library. Following the process above - VMM expects that the VHD has been prepared by the administrator with a blank local administrator password and sysprep has been run.

    (This is all in the documentation BTW - no secrets are being revealed).

    Now, why does the tempolating process fail? Especially with Server 2008? - this is where the machine local security policy comes in.

    Start -> Administration Tools -> Local Security Policy (this is within the VM - just to make sure that you were following along)
    Account Policies -> Password Policy

    Minimum Password Length must be 0 (this is a blank password that we are trying to accept)
    Password must meet complexity requirements must be disabled.

    Wait a minute...I just tried changing these settings in my VM and I can't...what's up?

    Most likely, your VM is joined to a Domain and the Domain security policy is overriding the local security policy - you remember, strictest and/or last applied wins.

    Unjoin your VM from the Domain, set the local security password policy, shutdown the VM - return to the VMM New Template wizard and prepare your VM as a template.

    This will get you past the security bug when you create templates.

    In the case of VHDs that have been used to create templates - If you did not set the local administrator password to blank then the deployment of the VM will fail, not the template creation.

    It is a different step in the process, and a different failure, but the same cause.

    BTW - just to clarify. When I said the local administrator password needed to be blank I didn't mean the letters b.l.a.n.k or the word 'blank' - I meant (in programmer terms) empty string - null - nothing.

    Wednesday, July 9, 2008

    Remoting devices into a Hyper-V child (virtual machine)

    I have answered many questions about this in the forums, but I keep forgetting to document it. It took my wife asking me how to remote her Smart Card into a Hyper-V VM this morning that caused me to think about it again.

    If you are using Hyper-V, you have already discovered that there is no support for remoting local devices into a Hyper-V VM. This is by design.

    Hyper-V is an enterprise level hypervisor, like ESX, like XenServer, like VirtualIron, etc. None of those platforms support remoting devices either.

    To guarantee a division between the parent and the children (host and its VMs) this separation must be maintained.

    You may have also figured out that the Hyper-V console uses remote desktop protocol (RDP) to present the VM desktop. It does this by the host actually proxying the desktop from the VMs virtual video card. No different than ESX or XenServer using VNC under the covers. However, this limits you to network based resources and keyboard and mouse.

    You, of course, want more.

    It is really pretty simple, you simply use the Remote Desktop Client (RDC) to remote directly into your VM (don’t use the Hyper-V VM console client).


    First, you must tweak your RDC connection.

    1. Open the Remote Desktop Client (this must be version 6.1 or higher).

    2. Select the Options button (the one with the arrows showing you that there is More to it).

    3. Select the Local Resources TAB

    4. Select More



    5. Now, choose those devices that you want to remote into your VM.



    In my case I have Smart Cards and PnP devices selected.

    But I could also choose drives, serial ports, etc. Expand the trees, look around.

    Now, this will only work with the Remote Desktop Client, not the Remote Desktops Client (did you notice the ‘s’ ?).


    And - this is very important - you must successfully log in to the remote VM after adding these settings, or else the settings changes are lost.

    Okay, back to the previous thought..

    Remote Desktops is for management, Remote Desktop is for remoting the user experience, they are different animals.

    So, for all you testers, developers, people with Hyper-V on your laptops, etc. Here you go. You can have your Hyper-V and your weird devices too.

    Oh, another caveat - if you RDP into a Windows Server 2008 box (machine or VM) using RDP and you want to take all your neat-o cool devices in with you - you need to enable the Desktop Experience feature on the Server 2008 box (machine or VM). This is what installs all the support for all the neat-o cool things that we all enjoy on our desktops.

    Friday, June 27, 2008

    Plenty of Hyper-V RAM but I can't start my VM

    I have been toying with writing about this. And I say toying becuase I have been digging and digging in an attempt to find some good documentation about RAM management, usage, and allocation when running Hyper-V.

    Guess what? Very little has been documented so far.

    The title comes out of the TechNet Hyper-V forum and has finally been dealt with on Tony Voellm's MSFT blog. However, it isn't the entire picture - there is more going on here than a simple NUMA bug.

    What I have been able to uncover is this:
    • The parent partition reserves 256 MB RAM for itself that can never be consumed by a VM.
    • Each VM that is running carves a physical slice of RAM out of the hardware.
    • Each VM causes an additional 32 + 8 Mb of RAM to be used by the paraent partition for over head and management (not unique to Hyper-V).

    Now, all that being said, this is what I have been able to uncover and verify in some way.

    Server sizing recommendations follow that of Server 2008, plan on at least 512 Mb of RAM for the parent and then add on for the children (the actual VM RAM plus overhead).

    Now - to the issue.

    The NUMA bug:
    Tony Voellm talks about a NUMA bug here:
    http://blogs.msdn.com/tvoellm/archive/2008/06/11/can-t-start-my-vm-when-there-is-plenty-of-memory.aspx

    This article keeps pointing to caching..
    There is also a few threads in the TechNet forum that have resolved this issue in another way.

    the root thread is here:
    http://forums.technet.microsoft.com/en-US/winserverhyperv/thread/908d304c-0746-4272-ac7f-85cf4350f2f2/

    The solution on this thread? - disable automatic pagefile management in the parent partition.

    Again, we go back to paging (or caching..)

    So, something is going on under the hood that I am sure will be fixed but for now think ahead and take a look at the articles.

    Personally, I would try changing the page file setting or relying on Remote Management before applying a patch, but that is just me.. :-)

    Thursday, June 26, 2008

    Hyper-V went RTM (early, late,does anyone remember?)

    For any of you following Hyper-V this is all over the blog space - I am not breaking any news here.

    You can find the update here:

    http://support.microsoft.com/kb/950050

    Upgrading from Beta or RC0 or RC1:

    If you have any VMs with a saved state be sure to remove that saved state prior to upgrading the Host.

    • This is any VM with a saved memory state
    • A VM currently in “Saved”
    • A VM that had an online snapshot taken – while the VM was running
    • Delete the online snapshot (from the Hyper-V Manager), power down the VM, wait for the snapshot merge process to complete.

    Cleanly shutdown all VMs prior to upgrade of the Host

    • Not cleanly shutting down a VM will cause the Host to place the VM into a saved state by default, thus creating the risky condition above.

    You can patch the WS08 guests prior to upgrading the Host

    • Just power them down instead of re-booting

    Be sure to upgrade the Integration Services in all children

    • Simply open a console connection to the VM, login, choose Actions -> insert Integration Services disk and allow the upgrade to happen.
    • If the new device wizard appears in the VM, it will prevent the Integration Services from being upgraded
    • Just cancel the new device wizard within the VM


    Using SCVMM 2008 Beta:

    To this moment the VMM team has reported no issues with the Hyper-V RTM when the SCVMM Hyper-V RC1 patch has been installed. (your mileage my vary)

    To obtain the SCVMM Beta Hyper-V RC1 Compatibility patch:
    https://connect.microsoft.com/Downloads/Downloads.aspx?SiteID=580

    When installing the patch, have some patience – it will cause strange things to happen until you push new agents onto each managed host (after the host is raised to Hyper-V RC1) and the Hyper-V RC1 patch must be applied to the SCVMM Server (in addition to the SCVMM patch).

    SCVMM 2008 Operations Manager integration - I made it work

    There has been a lot of consternation regarding getting the Operations Manager integration working with SCVMM 2008 Beta.



    Unfortunately, this took me four tries and three rebuilds of my SCVMM server and my Operations Manager server.



    Why the pain? The fancy new PRO Tips feature, thats why. So I have spelled out all that I had to do.

    My personal opinion of this feature? It works, it does what it does. It has a ton of potential.


    If you want to get this working on your environment, give the following a try:



    Prerequisites (note: install them in this order):
    1. OpsMgr 2007 SP1 installed on the OpsMgr server
    2. All of the dependant Management Packs are installed:
    · Microsoft.Windows.InternetInformationServices.CommonLibrary.MP
    · Microsoft.Windows.InternetInformationServices.2003.MP
    · Microsoft.SQLServer.Library.mp
    · Microsoft.SQLServer.2005.Monitoring.mp
    · Microsoft.SQLServer.2005.Discovery.mp
    · Microsoft.SystemCenter.VirtualMachineManager.2008.mp
    · Microsoft.SystemCenter.VirtualMachineManager.Pro.2008.Library.mp
    · Microsoft.SystemCenter.VirtualMachineManager.Pro.2008.HyperV.HostPerformance.mp
    · Microsoft.SystemCenter.VirtualMachineManager.Pro.2008.VMRightSize.mp
    · Microsoft.SystemCenter.VirtualMachineManager.Pro.2008.VMWare.HostPerformance.mp
    · Microsoft.Virtualization.Reports.2008.mp
    3. Install an OpsMgr console on the SCVMM 2008 server (update to SP1)
    4. Add your SCVMM 2008 server computer account to the local Administrators group of the OpsMgr server.
    5. Add your host clusters to SCVMM 2008 – the hosts must be grouped as clusters (this applies to both Hyper-V and ESX hosts (ESX Hosts must be clustered in VirtualCenter)).
    6. Install OpsMgr agent on all hosts and virtual machines.
    7. For a VM to generate PROTips, the OpsMgr Agent must be installed on the VM.
    8. Enable remotesigning ( set-executionpolicy remotesigned ) on the SCVMM 2008 and OpsMgr servers. – This step may not be necessary
    9. Install the SCVMM administration console on the Operations Manager server. (it is currently unclear if this is a bug or by design)

    Configure PRO tips for each host cluster.
    1. Go to PRO tab in the Host Cluster Properties dialog box
    2. Select "Enable Host PRO on this host cluster" check box
    3. Select the severity level "Critical Only" or "Warning and Critical"
    4. Decide if you want to let SCVMM 2008 automatically take actions based on the recommendation by checking "Automatically implement PRO tips on this host cluster", and you can also further scope the auto-implement method only applies to "Critical Only" or "Warning and Critical" PRO tips.
    Microsoft recommends: "Bake time" (this is quoted from a SCVMM 2008 PM blog)
    · It's NOT recommended for users to enable PRO immediately after finishing install of SCVMM. Users are encouraged to get a stable environment first.
    · For PROtips to successfully migrate a VM, SCVMM 2008 requires some performance data to be collected.
    · Typically, two days of VM operation "bake time" should be sufficient.

    This is most likely because some actions require history to establish a baseline in Operations Manager, and to make sure that integration is functioning correctly.

    Turn on PRO for the cluster within SCVMM 2008

    On SCVMM 2008 console, go to Administration view, select "System Center" node on the Administration list, double-click on the "Operations Manager Server" line on the middle pane, and specify the OpsMgr 2007 root management server.

    Verification of PRO configuration

    The way to verify if the configuration was successful, is to go to OpsMgr console, open Discovered Inventory in Monitoring Space, Change Target Type to PRO Enabled Managed VM, PRO Enabled Managed Host and PRO Enabled ESX Server.

    Tuesday, June 24, 2008

    This is excellent! - Gizmodo goes to Lego

    For most geeks (young and old) Lego holds a near and dear place in our hearts.

    I hold great kudos to Gizmodo for sapping my attention span today..

    http://gizmodo.com/tag/legotrip

    This is too cool..
    http://gizmodo.com/5018247/lego-employees-have-minifigs-as-business-cards-and-a-great-sense-of-humor

    All I can say is: wow. I would cry for hours if my ladder fell over while working on this..
    http://gizmodo.com/5018606/750000+brick-kennedy-space-center-is-the-mother-of-all-lego-models

    DMZ isolation of VMs with Hyper-V

    I recently posted this in the TechNet forums and thoguht that it needed a bit longer life.

    Mostly, this is about understanding virtual machine networking under a hypervisor (ESX, XenSERver, Hyper-V, etc.) and that they all basically work the same.

    The simplest way to accomplish isolating VM traffic into a DMZ is to do a form of physical network isolation.

    Your Host has two physical NICs.

    Connect NIC 1 to the management network. Connect NIC 2 to the DMZ network.

    Create two External Virtual Network Switches - one per physical NIC and name them appropriately.

    Connect your VMs only to the DMZ virtual network switch (using the VM settings).

    Login to your Hyper-V Host console - check the network connections. You most likely have a Virtual Network Adapter (possibly two).
    You may end up with one for each virtual network switch. If this is the case, open the properties of the Virtual Network Adapter that is attached to the DMZ virtual Network switch and remove (uncheck) all protocols - this will prevent any traffic from the Host into the DMZ and vice versa.

    The alternative is VLAN IDs.

    Now..backing up.

    Looking at network connections within the Hyper-V console can be confusing.

    When you first install the Hyper-V role you have Physical NICs and properties just like you know and love from any Windows server. As soon as you create an external virtual network switch the physical NIC only has the virtual network protocol bound to it (it is no longer 'owned' by the Host, but instead by the Hypervisor) - leave this NIC alone.

    If your Host was using this physical NIC it is now given a new virtual network adapter that is attached to the virtual network switch which is in turn using the physical NIC - therefore keeping the host on the same physical network.

    The part that makes this confusing is that we CAN look at the network connections, and we CAN play with things just like we have with Windows Server for years. However, now we have to realize that the architecture is different - a mix of what we knew, a dash of what we saw with Virtual Server on Server 2003, and parts that are totally new becuase we have a hypervisor and a virtualization stack.

    Saturday, June 7, 2008

    How to manually merge Hyper-V snapshots into a single VHD

    Okay, so you have to recreate your VM configuration and you absolutely know that your VM had a snapshot at some time.

    You also realize that if you just link to the base VHD that you will lose the current state of your VM - what do you do?


    You manually merge your snapshots into your base VHD before you boot your VM. (I am assuming that you know how to connect to an existing VHD using the new VM wizard).
    Merging of snapshots can be performed manually. This is achieved by:

    1. On your Hyper-V host.Power off the Virtual Machine.
    2. Make a copy of the VHD and its corresponding AVHD files.
    3. Rename the AVHD extension to VHD.
    4. Write down the order of the disks from youngest to oldest (the oldest should be the root VHD). You can do this by looking at the last modified time stamp on the origional AVHD files, find the one that last changed. And find the last one that changed before it.
    5. In the Hyper-V manager, open the Edit Disk wizardBrowse to the youngest VHD in the chain, then choose 'reconnect' to point to the next youngest (the one that came before).
    6. Open the Edit Disk wizard a second time and merge.
    7. Then repeat the process until you have only a single VHD.

    In a disaster case, you need to recover a copy of the root VHD prior to attaching it to a new VM and booting it (the act of booting it, modifies it)

    Usually the most difficult part of this process is finding the last AVHD (differencing disk) in the chain.

    The easiest way to do this is to find the configuration file for the VM.






    Then open up that configuration file and locate the information for the virtual hard disk. In the screen shot below is the location of the current running state of the VM. The snapshot is a point in time to return to, the current running state is the "Now" and is contained in a differencing disk (AVHD) after a snapshot has been taken.


    Now, find that AVHD file within the file system and rename it to VHD.

    Now, go back tot he Hyper-V manager and open the "Edit Disk" wizard - Select the disk that you renamed above, and merge this disk into the one before it.

    This process can be continued until all of the snapshots are merged back into a single VHD (the base VHD).

    Hyper-V lost my vm and I connected to the vhd and lost everything..now I am panicking

    I have read this story on the TechNet forums a few times. Something happened in Hyper-V, either a patch, or a temporary burp, or something else happened that caused a VM to be 'lost' from the Hyper-V console.

    That might not be the worst thing.

    Then a well meaning and informed administrator creates a new VM configuration attaching tot he existing VHD only to discover that a great deal of time on the life of the VM is gone.

    What happened??

    Most likely what happened was that your VM had a snapshot(s). If you claim that it didn't - think back, did you ever create one, then delete it? (and, you have not powered off the VM for an extended period of time to allow that deleted snapshot to be merged back into the base vhd..)

    First of all the state of your VM may have been lost for good. The reason? - snapshots

    I have blogged a bit about snapshots (here and here, and here) and they continue to cause confusion. I am guessing that the confusion comes from a lack of understanding of the underlying system.

    The state of the VM is lost because the differencing disk integrity is broken. A differencing disk is a vhd that is chained onto an existing vhd. The first vhd (the base) stops receiving writes and the new vhd (the differencing disk) receives all of the writes to disk.the other item to know is that vhd is block based - therefore the differencing disk depends on the fact that the base disk does not change. If at any time you connect tot he base disk and modify it (even slightly) - the differencing disk chain is broken and there is not a tool (yet) that can put it back together.

    Okay, you are now the better informed administrator - you made this mistake once and you tell yourself - never again!

    This is a new post..

    Monday, June 2, 2008

    Remotely managing Server Core using compmgmt.msc

    We all have those days when we want a nice graphical view of a Windows server so we can quickly compare installations, we can't remember the command line syntax properly, or (as some CLI zealots would put it) we just want a nice lazy interface.

    Well, if you try to connect to Server Core - we all have run into the standard firewall rule of enabling Remote Management on Server Core. Well...this is just a portion of the picture.

    What if you want to bring a disk volume on-line (this is what brought me to this in the first place), and you don't want to type and then take it back off-line. (darn it, I want my right click - I want to do this relatively quickly, and I don't want to fish around for the CLI commands).

    My Google-fu was working well this morning and I found an excellent posting by Sander Berkouwer about getting all the rules and settings right to remotely administer a Server Core WS08 install using the Computer Management MMC snap-in.

    Here it is:
    http://blogs.dirteam.com/blogs/sanderberkouwer/archive/2008/04/03/remotely-managing-your-server-core-using-compmgmt-msc.aspx

    Thursday, May 15, 2008

    Hyper-V + TCPOffloading = poor network performance?

    * Update * - Here I use the term TCP Offload.  The actual feature is properly termed TCP Task Offload.  TCP Offload is better referred to as TCP Chimney and that should never be turned off.


    Here are some tips that I have passed on regarding Hyper-V and TCPOffloading.
    http://forums.microsoft.com/TechNet/ShowPost.aspx?PostID=3107983&SiteID=17

    In theory TCPOffloading should always make things better. It takes work away from your server OS, thus everything should work better. Right?

    As a former System Administrator, I have been disabling TCPOffloading on Windows Servers for years. Always to fix network performance issues. This works for SQL servers, Terminal Servers, and now Hyper-V servers.

    Why? I honestly have not gotten deep enough into the issue to fully understand why, I just know that if your network throughput is pathetic on a Windows Server that is running some application or role, try this trick.

    (as always take caution, this might work, it might have no effect - I take no risk from you)

    In this case, focus on the NIC driver on the base network connection. (not the virtual one that your parent partition uses, but the Physical NIC properties (the NIC with only the Microsoft Virtual Network Swith Protocol enabled))

    Make sure it is either the WS08 provided driver, or the latest from the manufacturer (it is usually always safer (read this as: overall more stable) to use the one that ships with Windows - unless you need a feature provided by a different driver)

    Try turning off TCP offloading - as this can cause strange behavior on some networks.

    Also, with Hyper-V if you are using a manufacturer provided driver and you want to do fancy teaming and stuff like that at the driver level - please make sure the driver has been updated for Hyper-V, or else you will get weird results here as well.

    Don't use NIC teaming, or NIC management software unless the manufacturer says it is Hyper-V role happy (not just WS08 happy).

    Some NIC teaming management software needs special control of the hardware and it might not work after Hyper-V is installed, since the physical hardware is now shared.

    Monday, May 12, 2008

    Using a single base VHD to create many individual VMs

    First of all, please note - this is a test and development trick. The use of differencing disks does cause a reduction in disk performance of a VM (a snapshot is also a differencing disk).


    Setting Read only on the parent VHD is not critical in Hyper-V as it was in Virtual Server.

    Here are the basic steps:
    1) create your Base VM - base.vhd
    2) prepare him with sysprep (or your tool of choice)
    *it is at this point (or after) that you can tag base.vhd 'read-only'
    3) create differencing disks using base.vhd as the parent.
    * Use New -> Hard disk -> Differencing from the Hyper-V manager
    4) create a new VM 'using an existing VHD' for each differencing disk

    You are done.

    In the end you get:
    base.vhd
    _ differencing disk 1.vhd
    _ differencing disk 2.vhd
    _ differencing disk 3.vhd

    Base.vhd can never be used for a VM - so the VM that you created when you created him must be deleted.

    The only reason for making the base.vhd read-only is that if it is modified at all, all the rest of your children differencing disks will be 'broken' - broken as in they will stop booting and will crash, just as if a hard drive failed.

    VHD is block based and differencing disks contain a 'map' that references the parent - so parts of your VM disk reside within both VHD files (the parent and the child).

    The versatility of Hyper-V Export / Import - fixing Hyper-V VM failover and backup problems.

    The Export and Import functions of Hyper-V are not well understood. Are they a simple tool just to move virtual machines about?


    I challenge that thinking, the functions can be used for much more, such as 'fixing' some VM behavior in Hyper-V.


    First of all - lets explore what these two function do:


    Export collects the pieces of a Hyper-V virtual machine and collects them into a single folder (with the same name as the VM) on a volume or at a local path.


    Import simply 'attaches' to the exported virtual machine, reads in the configuration, and allows you to run the VM from the location that you imported it from.


    Cases where these functions are useful:


    The obvious - export, move, import


    The not so obvious:


    Fixing Failover Clustering of a VM. - When a VM is created the default settings place the configuration and snapshots under the path %programdata% and the virtual disk files under \users\public. If you want your VM to be highly available, you need all of these bits in a shared storage location. Using Export to do this is a really simple solution to collect the VM together and to change its configuration, snapshot, and VHD location to that shared volume. *Poof* a bit of administration magic, you just fixed your VM for failover clustering.


    Enabling a VM for easy backup and recovery - All administrators know that backup and restore is an exercise that involves chicken sacrifices, finger crossing, and sweaty palms. You quickly learn how good your backup plan is when you work through the exercise of a performing a staged disaster recovery of a particular VM or application within your enterprise.


    Anyway, just like for failover clustering, having your VM neatly bundled in one location makes it easier to comb through volumes and folders within your backup catalog (not everyone uses DFS).


    You can also Export a copy of your running VM, just for safe keeping, until that application is upgraded again, then you have a quick machine to use for testing and recovery.

    Sunday, May 11, 2008

    How do I stop Failover Clustering long enough to perform some maintenance that involves a reboot without chasing my VM between my clustered hosts?

    I have actually been going around with support about this issue myself. Maintenance mode does not exist for the VM workload.

    How do I stop Failover Clustering long enough to perform some maintenance that involves a reboot without chasing my VM between my clustered hosts?

    In failover cluster manager I can select the Virtual Machine and take it offline - this causes my HA VM to be powered down - and also removes it from the Hyper-V manager.

    If I select Manage Virtual Machine in failover cluster manager I get directed to the Hyper-V manager.

    If I take my Virtual Machine configuration offline - then my VM is shut down but it is not removed from Hyper-V manager.

    If I take a snapshot and try to revert, as soon as my VM gets stopped to perform the revert, it gets migrated to another host.


    Here is a solution that works ONLY IF you need your VM off: (this works for changing hardware, etc.)
    Open the properties of the Virtual Machine resource in Failover clustering.
    Set the off-line action to Shut down (this is supposed to stop the VM from failing over and restarting on another host).
    Take the resource off-line.
    return to Hyper-V and make your changes.

    Here is a solution if you want failover clustering to leave your VM alone for a while (so you can reboot it without if failing over):
    Open the Virtual Machine properties in Failover Clustering
    Select the policies tab
    Select "if resource fails, do not restart"
    Save that setting.
    Return to the Hyper-V Manager and you can open your Virtual Machine Connection console and do things within your VM that may involve a shutdown or reboot to your hearts content.

    Just remember that when you want Failover Clustering to be back in charge, you need to return to Failover Clustering and undo what we did above.

    Thursday, May 8, 2008

    Hyper-V plus failover clustering, an interesting marriage

    Hyper-V is a really cool Windows add-on as by itself it is “just another hypervisor” but with the addition of a bunch of other Windows Roles and Features it quickly becomes much more.
    Take High Availability for example. Hyper-V, plus a VM workload, plus Failover Clustering.


    For those of you not already familiar with Failover Clustering I am going to talk a bit about Windows clustering in general. First of all, I am speaking of Failover Clustering, not Network Load Balancing clustering, that is totally different.


    In the generic sense, Failover Clustering is a way of taking a workload that runs on a clustered node and keeping that workload available. With Hyper-V it involves keeping a VM powered on.
    The only big requirement is shared storage. This can be old fashioned SCSI shared storage, fiber SAN, or iSCSI. If Windows can see it as storage and you can present it to more than one server then you have shared storage.


    The Failover Clustering setup and validation wizards in Windows Server 2008 make clustering really super simple (makes me cringe when I recall my first NT 4 cluster). You run the wizard, and if you listen to it, you have a fully MSFT supported cluster – you even get a recommended quorum configuration.


    One limitation to consider is NTFS. By default only one node (clustering term for a member server in a cluster) can own a LUN at any one time. To be a bit more granular, only one server can write to an NTFS partition at any one time. It is possible to share a LUN with two Windows servers, but even having one reading and the other only looking your volume will begin to degrade very quickly.


    This sets up a one Highly Available VM to one LUN model for Hyper-V.


    A highly available (HA) guest is made up of three parts. Part one – a configuration file. Part two – the workload. Part three – the LUN (that contains the VHD).


    When a HA guest is failed over from one node to another all three parts must be moved between the nodes. The configuration is passed, the volume is passed, and the workload is passed.


    The logistics behind this is that your HA guest is saved, its LUN is passed (assuming that all the bits of the VM reside in one folder), and then the guest is started (resumed).


    The passing of the LUN prevents having more than one VM workload on a shared volume as the other VMs end up being ignored.


    Why? You might ask. In a previous post I had mentioned about struggling with failover clustering for an hour or so, and above I mention that Hyper-V is not making the guest highly available but it is Failover Clustering.


    Failover Clustering is acting upon that HA vm workload and doing whatever it takes to keep that VM up and running. It is what is controlling the VM, not Hyper-V.


    Hyper-V is still involved, but from the standpoint that the VM guest heartbeat is lost for a moment, then failover clustering is right there, ready to move that VM in a snap and keep that darn thing running. IF there is collateral damage, that is not the fault of failover clustering, but the admin.


    Will this behavior change? Who knows. Windows clustering has worked this way for a long time now, and so has NTFS. I guess that if you could get past the NTFS limitation, then you could do it.


    That is enough for now. More later.

    Tuesday, May 6, 2008

    Hyper-V Snapshots, more about how they work

    Previously, I wrote about snapshots with Hyper-V and the architecture behind them.
    There still seems to be some confusion among administrators regarding them, so let’s take another look.

    The big first hurdle to understanding snapshots is that a snapshot is a moment in time.
    (Start thinking like Lieutenant Daniels in Star Trek Enterprise and forget the Vulcan Science Directorate opinion that time travel is impossible).

    The second concept is how snapshots work.

    I have a virtual machine that I will call “TestVM.” I just created TestVM using the new virtual machine wizard and finished installing Windows Server 2008. My intent with TestVM is to test various WS08 Roles and Features.

    My root run state is my fresh and clean install of WS08.

    To make an offline snapshot I shutdown the VM and take a snapshot – this snapshot is (by default) labeled with the moment in time of ‘now’ (5/6/08 1:06 pm).

    In the background Hyper-V copied my hardware configuration and attached a differencing disk to my TestVH.VHD thus when I power on the VM anything that I do while the VM is running will be written to the differencing disk.

    If I power on the VM and do something then decide that what I did was ‘bad’ for my VM I perform a Revert. What does the revert do?

    The revert takes the differencing disk that my current running state is using and throws it away and creates a new one, thus returning me back to the moment in time that the snapshot was taken.

    After performing the revert I added IIS and ASP.net. This is good. But I want to go back to my clean snapshot moment (saving my current moment) and do something different. In this case I choose apply. And I select the option to “take a snapshot and apply”.

    Now I have a new Online snapshot of ‘now’ (5/6/08 1:12pm) and I return back to my previous Offline snapshot.

    In the background Hyper-V did something a bit different. Hyper-V copied my hardware configuration, but it also saved my running memory state, and created a differencing disk for when I return to that snapshot.

    Now that I have gone back in time I am running within (yet another) new differencing disk attached to TestVM.VHD

    Wow, this is getting deep. And it can. You can build an entire tree with many branches.
    What I need to remember is what each moment in time represents (I can rename snapshots), that they return me to a moment in time that includes the hardware configuration (in case I change that), the disk state of the VM (whatever was written) and the running state of the VM at that moment in time.

    You can see how this is pretty powerful in a test and development type of scenario. But production is a totally different issue, and smells like a new post.

    Wednesday, April 30, 2008

    Hyper-V + SCVMM + XenDesktop

    A lot of people don't realize that Citrix XenDesktop supports more than XenServer as a backend.

    In fact, it will support any backend infrastructure (or infrastructure-less) environment that has a plug-in. XenDesktop ships with plug-ins for XenServer (of course), SCVMM, and Virtual Center.

    Since most folks don't know that SCVMM support is available I took some time to put a little something together. I even have my VMs running on Hyper-V.



    One caveot, your XenDesktop Desktop Delivery Controller Server must be 32-bit.

    Friday, April 25, 2008

    Hyper-V + Failover Clustering .. the fight for the managed workload

    Wow, I just had quite an experience with my Highly Available Hyper-V virtual machine.

    First of all, the HA feature with Hyper-V using Failover Clustering works REALLY well.
    Second, I have to spend a bit more time thinking about what the heck I am doing before I go nuts again!

    Okay, here I am, playing with snapshotting - taking snapshots, reverting, etc. Trying to document how things work and change.
    Taking snapshots was a no brainer, everything worked fine.

    When I deleted a snapshot I diligently shut down the VM and started monitoring the volume waiting for the merge to happen..I started at the screen for 20 minutes, watching, switching back to the Hyper-V console, watching the volume.

    Finally, I tried to refresh the volume - boom! The volume is gone. Oh, cr**! iSCSI must be having problems - poke, troubleshoot, poke some more.

    Suddenly, in a fit of frustration - duh, I have failover clustering set up. Check failover clustering - my VM is now running on Host 2. GAAA!!!

    Okay, move the workload back. Get the merge to happen properly.

    Now, revert.. Do a revert - boom, communication to the VM is lost (since the Host is serving up the console over RDP). I check Failover Clustering again - there is my VM on the other Host again - not properly reverted either.

    Wow. The things to think about now.

    Microsoft has done a brilliant job at using other Windows Server 2008 features WITH Hyper-V (think about it, Hyper-V is not much at all without all of the WS08 and System Center add-ons.) but the complexities of interplay between these components is not for the admin with a weak constitution.

    Keep those skills up to par and be the admin that thinks out of the box.

    Thursday, April 17, 2008

    Determining versions under Hyper-V

    [Update:  Just ot let everyone know, the method below still works, however what has happened over time is that the various drivers have drifted and the release builds are no longer consistent among the various components on one side.  By that I mean that all the Server side components might not be at the same build number and all the client side drivers may not be at the same build number.  What should be consistent is tha the build of hte server service and its corresponding client driver should align.  So, just think.]


    Every now and then we need to determine the release level of a component that we are running.

    In the case of Hyper-V, if you have the role installed you can check the version of vmms.exe

    There is the GUI way:
    Browse to \windows\systm32\vmms.exe and check the properties -> details tab

    There is the command line way:
    wmic datafile where name="c:\\windows\\system32\\vmms.exe" get version

    you should see a build version either way - here is the breakdown:
    Beta Version of Hyper-V installed == 6.0.6001.17101
    RC0 version of Hyper-V installed == 6.0.6001.18004

    Now, you don't have the Hyper-V role installed and you want to check the release level of the Integration Components. (this also works within a VM to check the version installed within a VM)
    We just look at different files.

    Open Device Manager and find one of the vmBus devices (vmBus Network Adapter, VMBus HID Miniport, vmBus Video Device), open its properties, on the driver tab, check the driver version.

    The version will end in 18000 if you are running Beta Integration Components
    The version will end in 18004 if you are running the RC0 Integration Components
    ...
    The version will end in 18016 if you are running the RTM Integration Components

    If you need to update Hyper-V or its integration components under Windows Server 2008 then install patch KB949219 to move to the RC0 release. To update any other supported guests insert the Intgration Services Setup Disk from the Connect client window and let autorun do its work (assuming a Windows guest).

    Thursday, April 10, 2008

    Hyper-V RC0 Patch is on Windows Update

    Yea, the Hyper-V RC0 patch is available on Windows update!!

    Okay, stop now, relax, and think.

    What does this mean to me as an admin?

    If you have built a nice fresh and clean install of Windows Server 2008 AND have not put any virtual machines on it - go ahead, patch freely and with reckless abandon.

    If you are running Hyper-V as the WS08 RTM and have running virtual machines. Stop!
    Read this most excellent post from John Howard (all of it).

    In a nutshell..if you have snapshots - delete them now (this merges the snapshots into the base VHD), shutdown the VM and wait for the snapshots to be merged in to the base VHD.
    This can take a while, soe you have to monitor the AVHD files of the VM - when they are all gone (poof, they disappear when they get mergd in), then you have a VHD you can walk away with.

    The snapshots are not 100% compatable between the RTM and RC0.

    Document any non-default network settings you have in your VMs (host too)
    The netwoking is upgraded in this release AND you will end up creating new NICs within your VMs which means hardware detection kicks in and gives you a new NIC with default settings.
    (And if you have an application installed in a VM that binds itself to the MAC address - make sure you know that MAC address so you can set it again)

    After the VMs are ready and shutdown you can patch the Host.

    After reboot you may need to recreate a VM configuration file, you may need to recreate virtual switches, you may need to attach a VM Network Adapter to a new virtual switch..

    As the hypervisor and the virtualization stack is modified the GUIDs assigned to devices changes. In the Windows world we know that if you move a NIC from slot A to slot B we end up with a new device in device manager (a new device GUID).
    The same thing is happening under the hood in this case. Only in this case you are left pullling your hair out.

    Oh, and if your VMs won't boot after the upgrade you might want to explore this TechNote:
    http://support.microsoft.com/kb/949222/

    Wednesday, April 2, 2008

    Networking under Hyper-V (who moved my network settings?)

    There is a lot of discussion around networking and network changes on a Windows Server 2008 server after the Hyper-V role is installed.

    Part of this issue goes back to what happens (what is done) to your WS08 server when the Hyper-V role is installed.

    As I had mentioned in a previous post when the Hyper-V role is installed the WS08 server is fundamentally changed. As an administrator you login to a console and you see WS08 - it looks like nothing changed, however it has.

    Ben Armstrong has a quick posting here that gives some basics.

    Lets take a more architectural look at the server and what has happened / is happening..

    During the process of adding the Hyper-V role the WS08 installation was turned into a virtual machine itself and it is running on top of the hypervisor as the parent partition. Unlike other hypervisors where you see a Linux based console, in this case you see your WS08 server - a nice, friendly GUI interface that really does not look any different than it did before.

    Now, what does this have to do with the networking? It has to do with modifications that were made to the network interfaces of the WS08 server when the Hyper-V role was installed.

    Quite honestly, the modifications that are made are no different than what happens when modifying any other Windows server in a similar way.

    Most likely, if you set a manual IP address (any manual settings) they were lost.

    If you look at Network Connections in the WS08 server you notice new Virtual Network Adapters and your original network connections were changed.

    Noting back, your WS08 server was turned into a VM (its hardware was changed) - the original network connection (which you can still see) was turned into a virtual switch (the WS08 server no longer owns that NIC).

    Your WS08 server (WS08 parent partition - that is what it is now) was given new virtual network card(s). And as with adding any new NIC to any Windows server it gets the default settings (DHCP for example).

    Now, there is also talk about performance (the parent partition performance is terrible, but a VM runs great - or the other way around).

    Now we have to begin looking at the NIC driver and driver configuration.

    First of all, good old TCP Offloading, long a performance issue in many Windows environments might need to be turned off. Mind you, this issue seems to be environment specific.

    The other is the NIC driver itself. You are pretty safe using the included WS08 drivers.

    Some troubleshooting questions:
    Did you install a non-Windows delivered driver?
    Did you install a teaming driver?
    Did you configure teaming?
    Was there management software installed with the teaming driver or was only the driver installed? (some experience has shown that the driver itself might be fine but the management software causes problems - as it is trying to monitor a NIC that the parent partition no longer owns)

    My goal with this post was to help an administrator understand what is going on, and with that where to look to solve his/her problems.

    Saturday, March 8, 2008

    Hyper-V, not your mother's Windows server anymore..is it really appropriate to run Exchange in the Hyper-V parent partition?

    MSFT has a very good reason for the recommendation of not installing other applications or roles into the Parent partition of a server on which you desire to run Hyper-V. Or taking another application server and converting it to a Hyper-V server.

    Yes, it is technically possible to install any application into the Parent partition of your Hyper-V server, just as it is technically possible to turn any WS08 server with any applications installed into it into a Hyper-V server.


    But the rub comes because adding the Hyper-V role is not the same as installing Virtual Server. It isn't just another Windows application that is installed onto a Windows server. The behavior of the operating system is modified and its overall behavior is changed.


    Until Hyper-V is installed you have "just another Windows server" that you can do anything with and pile applications onto just as you have done for years. Once the Hyper-V role is added, the OS is fundamentally modified and the architecture is changed.


    What was a normal install of WS08 is now transformed into more of a VM itself (we call it the Parent). And like Parents everywhere it has a role in life - to protect, care for, and sacrifice for its children (the VMs in this case).


    This is best described through a graphic. In the picture below the left shows a 'standard' Windows server, the arrow represents the process of installing the Hyper-V role, and the right represents the new architecture that the server is converted into.



    After this transformation your Windows Server 2008 server is now architected totally differently - it is fundamentally a VM itself. If you are familiar with other hypervisors (ESX server, XenSource, VirtualIron, Xen, etc) it is a similar model. In many of those cases you would not consider installing an email server or an LDAP service into that Parent partition - those servers are seen to have a definite and dedicated role in life.


    This perception is further complicated by the interface that Hyper-V has. It is a nice, friendly Windows server GUI - just like we know and love. We can perform functions using the Hyper-V manager, but we can also work with the underlying OS, manually manipulating the network connections, disk management, etc.


    Now, Hyper-V capitalizes on this by using other Windows services to its advantage.


    Do you want to set up two Hyper-V servers for high availability? Set them up in a traditional Windows cluster configuration and make each VM that you want highly available the clustered "application" (it is here that I begin to talk about a VM as a workload - in this configuration the VM is no different than an application).


    Do you want to set up role based permissions? Set up Authorization Manager.


    These two things are not specific or unique to Hyper-V but they are used as a component of your environment / deployment.


    I hope that I was at least successful in starting you thinking and asking questions regarding this new way of looking at a Windows server.

    Friday, March 7, 2008

    Snapshotting in Hyper-V does not equal VMware checkpointing.

    The snapshotting concept under Hyper-V

    For all of you familiar with ESX server and Virtual Center (or using a 3rd party product to do backups using checkpoints) you will have to think about things a little differently.

    First of all.
    Is using Hyper-V snapshots a good method to backup your VMs?

    Answer: No. It isn't a backup method at all. It provides the ability to return to a defined point in time in the life of your VM. It is a very useful tool for testing application upgrades and service packs.

    What are snapshots then (if they aren’t the same as a checkpoint)?

    The best way that I have been able to describe snapshots is by using a timeline or traveling in time.
    A snapshot is a moment in time that you can return to. And going right along with that - you can freely move forward in time, but moving backward in time will alter your future.

    "Don't alter the timeline" - we have heard it hundreds of times (especially if you watch Star Trek Enterprise). Well, then you also hear the Vulcans state that time travel is impossible..

    Snapshots in Hyper-V are all linked together, a single snapshot cannot stand alone (at least not without doing some tricks) as it is linked to its parent and so on.

    The concept looks like this:

    You begin with a base VHD. When a snapshot is taken an AVHD disk is created and the configuration of the VM is updated to use the AVHD as the current virtual hard disk.
    (An AVHD is a snapshot specific differencing disk that is being used as the running point of a VM)

    The AVHD disk is now the point where all system changes are written from this time forward - the base VHD is no longer being modified. And the AVHD is linked to (dependent upon) its parent disk. If you were to move one of these two files the VM is broken.

    You can continue to spawn off additional snapshots. Each snapshot is linked to its parent in a linear (timeline) arrangement. They cannot link in a branched tree arrangement because that would create dead branches.

    When you go back to a previous point in time (return to a snapshot) everything to the right of that point in time is destroyed (rendered unusable) because you altered the VM at a previous point.


    Managing snapshots
    Okay, what can I do with snapshots besides take them?

    You can select a shapshot for a VM and “apply” it or “revert” to it.
    This takes you to that moment in time and begins running the VM at that point in time. (You begin using the AVHD file that was created at that point in time).
    “Revert” takes you back one step while “apply” can take you back many snapshots in one step.

    Deleting a snapshot removes it from the tree.

    You might wonder “if all of the snapshots are linked together how can I delete a snapshot out of the middle?”

    This is because in the background Hyper-V is maintaining the integrity of your VM by combining the various states of your VM (or flattening the structure of your differencing disk timeline). If this didn’t happen your entire VM would be broken – that would be a serious bummer.

    The merge process happens quietly in the background when a VM is powered off.
    That is an important thing to remember – the VM must be powered off for all of the disks to be merged together.
    This counts for anytime that snapshots are modified or merged into the Parent.

    Delete a snapshot and its subtree.
    This process deletes a snapshot and any other snapshots that are to the right of it in the timeline. The result is just that any snapshots taken to the right of the point in your timeline that you elected to perform this operation are all committed and merged into your current snapshot.

    What if you want to “flatten” your tree and have everything written back to the base VHD?
    Delete each of your snapshots one by one, even the current running one, and wait for the changes to be merged back into your base VHD. This can take a long time, and the VM needs to be powered down.

    Deleting snapshots does not move you back in time, only applying a snapshot does that. Deleting snapshots merges your changes into another disk within the timeline.

    Oh, and why don’t I have a bunch of screenshots of the GUI?
    Because I figure that most Server Admins are smart enough to handle a GUI ;-)

    Hyper-V terminology

    For anyone that really wants to get a handle on the terminology behind Hyper-V you need to read this post by Ben Armstrong. It really goes through the MSFT terminology and how they refer to things in Hyper-V.

    It also covers some basic concepts and is a good overall primer worth everyone's review, especially if you are thinking of going whole hog with Hyper-V.

    The Xen of Server 2008

    Server 2008 with Hyper-V... Hmm.. A hypervisor.. but it looks like Windows.

    Actually, it is a totally different way to look at and think about what we already know of Windows Server and what a hypervisor is.

    Until now a hypervisor was a Linux experience with a management console that brings it into the Windows realm. Ironically enough most administrators of virtual environments have turned out to be server administrators that have grown up in the Windows world.

    After the Hyper-V role is added to Server 2008 the traditional way of thinking about a Windows server changes, as does what it is and how it behaves.

    It is really easy to look at a server with Hyper-V and forget that it is now a hypervisor. That it has a job of running virtual machines. And for those of us that have been using VMware, that this is a totally different animal - not because it is Windows but because it is not VMware.

    I am going to be expanding on these ideas as Hyper-V matures and we continue to explore it.

    I have been spending a great deal of time exploring snapshots and plan on looking into backup and disaster issues.

    Oh, and many have lamented that Hyper-V uses a Xen 'like' engine (this has been mentioned all over the place). The hard core open source community seems to be worried about what is happening to Xen since the Microsoft design inclusion and the acquisition of XenSource by Citrix. That discussion is strictly speculative and not one I intend on getting involved in.

    What Hyper-V means to IT is that there is a new platform, one that seems strangely familiar, yet isn't. But that is well worth touching, kicking, trying, and exploring.

    Is it foolproof? No, just like the other virtualization engines.

    Does it require thought and planning? Yes, just like the other virtualization engines.

    We can't be too comfortable with it just because it feels like Windows - "feels like" is the thing to remember.

    If you think about it - the Microsoft developers have achieved something rather amazing. Marrying Windows and what we all know of as a Linux concept and presenting a truly Windows like management experience (with some unique behavior).

    But, all administrators have to remember that it isn't just a Windows server anymore. It is more complex than that. And for those virtualization veterans from the VMware world and Virtual Server world..some of the things you know and have relied on - don't apply here.

    That is the stuff I plan on exploring.

    For now, ponder the future of a datacenter with virtualization Parents / Hosts and their Children / Guests much as a pool of resources - and we no longer care of the back-end engine - we care about the management interface, experience, and controls.