Live (VSM) migration fails with mirror operation failed and access is denied error

When doing a Live Migration from SCVMM (System Center Virtual Machine Manager) with VSM, moving a Virtual Machine from one Cluster to another Cluster and at the same time also to a new Storage Location, you are getting an error message similar to this:

The strange thing is that there is a destination folder in the new location, it’s just does not copy content to that folder and aborts with the Access Denied error. But If you shutdown the VM first, so it’ s just a migration over the Network, it works!

The solution is to give the SOURCE Cluster Write Access on the DESTINATION Storage. When you do a VSM Migration, the destination Hyper-V host, creates the Directory on the SOFS Node, but it’s the Hyper-V Host that owns the VM that copies the VHD’s files to the destination storage. And as the current owner, by default does not have access to write there, it will fail. One could think that VMM should grant permissions to a host when VMM knows that the host needs to write in the location?

Maybe it’s fixed in the next version, but until then, there are two ways to do this.
Solution 1) In VMM add the Destination SOFS Shares as Storage on the Source VM Hosts like this. That will make VMM add the VM Hosts with Modify Permissions in the SOFS Shares so it can write there.

sofs2

This works quite fine, if the Hyper-V Clusters and all Storage is located in roughly the same location. But if you have one compute cluster with storage in one location, and another compute cluster with storage in another location. There is then a risk that you may be running VM’s cross the WAN link.

Solution 2) This is the one we used. By not using VMM to grant permissions to the shares, but rather do it manually we achieve the same solution as above but with the added benefit that a new VM will always be provisioned on the local storage and there is no (or a lot less) risk of running a VM cross the WAN link. Yes, it’s still technically possible to do it, but no one will by accident provision a VM that uses storage in the other datacenter.

You can either add each node manually, so we have created a “Domain Servers Hyper-V Hosts” security Group in AD where we add ALL Hyper-V hosts to during deployment. And then added that group to the Share and NTFS Permissions. All Hyper-V hosts will then automatically have write access to all locations they may need.

I wrote these two short scripts to query the VMM Database for the available SOFS Nodes and use powershell to grant permissions to the share, and to NTFS.

As all our SOFS Shares were called vDiskXX or CSVXX (where XX is a number) I just used a vDisk* and CSV* to do the change on all those shares. You might have to modify it a little to suit your name standard.

Updated Script (2016-02-04):
I got a report that the script was getting an error on some servers, which I managed to reproduce. Here is an alternative version where it will connect to the server and execute the ACL change locally via invoke-command. It’s also only changing permissions on Continuously Available (SOFS) shares.

 

 

List of Private Cloud related Hotfixes – 2016-02-03

I’ve posted my list of resources for finding Hotfixes previously here.  And this is a list of hotfixes we’ve imported in our WSUS server for our and our customers Private Clouds.
I usually want to install a hotfix to avoid getting a known problem, than try to find a solution to a problem after it has already happened and affected the users and customers.

wsus3

I’ve used the script I posted here to make the list. I’m sorry for the format below, but there is no good way to extract the info from WSUS and as I don’t really know if anyone is interested in this besides myself, I won’t spend hours on fixing a nice output right now or I would never get this blogpost published.  Sorry!

I’m using the MUUri to paste into the WSUS IE to search and locate the hotfixes fast, instead of manually search for each. There is unfortunately no way to script the import according to Microsoft PM’s, so it has to be done manually. Sigh…

Continue reading “List of Private Cloud related Hotfixes – 2016-02-03”

Export information from WSUS about Hotfixes or Updates

I want to export information about all our Hotfixes in our WSUS Server, to share with the community as it’s sometimes hard to find up to date info of which hotfixes to apply in an environment.

Here is a quick and dirty script. No, scratch that, it’s not quick in any way but very dirty. The problem is that the WSUS Database does not contain the Description or Title of a Hotfix, so it’s not possible to export that info. Thus, I’ve to use a scripted Internet Explorer, to navigate to the URL of each Hotfix and grab the Title. Which makes the process veeeeeeery slow but I’ve been unable to come up with any better solution than that.

This is how the output looks like:

wsus2

And here is the full script, I will be really grateful if you share any changes you do as I’m sure there is a lot of ways to improve this script.

 

Geek Week – Building Datacenter and Private Cloud

Yours truly and @mikael_nystrom  are teaching a 5 day training called “Geek Week – Building Datacenter and Private Cloud“.
It’s a quite cool concept, where we are always at least 2 teachers (Mike and me, and sometimes members from our team) with a lot of real-life experience, teaching students how to build a Software Defined Private Cloud/Datacenter on Microsoft Technology in just 5 days. We have been doing this training about 4 times so far and each time got top score !

The training is covering things like; Software Defined Storage, Networking Compute, Bare Metal Deployment, SCVMM, SCOM, SCOR, Powershell(!), Azure Pack, Backup/Restore, Operational Insight, AD Design for your Fabric, GPO’s, Patch Management and a lot more!

So instead of just learning one product, you will learn the full stack, the concept, how to design, build and of course manage it.

We’ve made a short promo video with details here:

https://youtu.be/_PjRHv84VgM

You will of course also keep the Lab Server and all the scripts!
We are both staying at the same hotel as the students, so you will have – almost 24h access to our knowledge, experience and brains during breakfast, daytime and in the bar in the evenings, Sunday-Friday.

We have delivered the training only in Sweden so far, due to the huge hardware requirements,  but have had participants from several other countries, including students from US (and doing the training in English than).

Unfortunately, for you but great for us, the next training in October, is SOLD OUT!
The next opportunity will probably be in February or March in Sweden, so talk to your manager and sign up while there are seats left.
http://www.labcenter.se/Labs#!lab=Geek_Week_-_Building_Datacenter_and_Private_Cloud  (Unfortunately only in Swedish)

Send me a mail if you have any questions: markus . lassfolk at truesec . se

Script to change from Dynamic to Static MAC Address on all VMs

A customer had a lot of VM’s with Dynamic MAC address, rather than the preferred method of using Static MAC addresses.
Here is a small powershell script that will shutdown each of the VM’s with a Dynamic MAC Address, change to a Static MAC Address and then start the VM.
I’m running the script on the System Center Virtual Machine Manager (SCVMM) Server and to make sure VMM does not shutdown itself, I’ve added an exclude for the SCVMM Server.

The MACAddress 00:00:00:00:00:00 will automatically be transformed into a real static address from VMM’s mac address pool.

URL Rewrite (redirect) of HTTP to HTTPS with Powershell script

When deploying Web Application Proxy as a frontend to for example ADFS and Windows Azure Pack, or other services, the current version of Web AppProxy only supports HTTPS urls. It’s possible to use the “URL Rewrite” module for IIS to redirect users from HTTP to HTTPS. There are plenty of guides on internet on how to do that.
But I wanted to add that configuration to my WebApplication Proxy configuration script, and couldn’t find any powershell examples, so here is the script I’ve made.

It will use Web Platform installer to install the URL Rewrite module, then add the IIS Web Management tools, and in the end create a Global Rule redirecting all HTTP requests to HTTPS without the user noticing it.

 

 

Uninstall old SCDPM Agents with Powershell on all Servers

When you want to add a server to Microsoft Data Protection manager to be managed (backup), it might already contain an old DPM Agent, which prevents the installation from going through. SCDPM will tell you to manually use Add/Remove Programs on each server and uninstall the old agent.

To find all servers in the environment with an old SCDPM Version run this script;

Of course, update the Version number to the latest current version of SCDPM Agent if it’s newer than 4.2.1338.0.
The get-adcomputer command will find all servers that are not disabled or Cluster Objects. So it should only give you the “real” servernames.
And here is a script that will uninstall all old versions of SCDPM Agent on all servers in the environment, so you can add the server through the SCDPM Management Console and get the new agent installed automatically.

Once again, change Version (4.2.1338.0) to the correct version you want to keep. It will uninstall all older versions.

Yay! You can now add the servers to SCDPM Successfully.

Update SCVMM Agent with Powershell

I just updated our System Center Virtual Machine Manager 2012 R2 Environment to Update Release 7. SCVMM would then report that the SCVMM Managed Computers has an Out-of-Date Agent which need to be upgraded.

It’s possible to do it manually by right clicking on each server and choose “Update Agent” or use this short powershell script to do it on all machines at the same time.

It will Update the Agent on all Managed Servers for you (no restart needed in my case).

 

 

Microsoft Fabric (datacenter and private cloud) related Hotfixes

Here is the list of Hotfixes I’m deploying in our production environment and that I deploy regularly at customers. Those production environments are a Fabric (Private Cloud) running Hyper-V, Storage Spaces, SOFS, ADFS, Domain Controllers, Azure Pack, System Center, SQL Servers, and more, yes everything you need in a Fabric. Though not Exchange, Lync or Sharepoint etc. So this list might not be complete for your system.
And as always, use your own judgement which hotfixes you would like to deploy in your environment or not. Hotfixes are not tested as much as ServicePacks used to be, and Update Rollups are, so it’s possible there are problems with them.

My philosophy is that I like to have everything updated and reduce the risk of having a problem. The number of times I have had issues with a hotfix are, as far as I can remember one (1), including the several years I worked at Microsoft Premier support and were assisting customers with problems and now and then provided a hotfix for an issue. So I rather install hotfixes I know of and are relevant to reduce the risk of hitting a real problem than wait for that issue to actually happen and then find a hotfix or open a case with Microsoft.

A hotfix included all previous fixes for that module too, so when troubleshooting a problem, it’s common that Microsoft Support asks you to install hotfix X, Y and Z to get the components involved in the problem to the latest revision. Thus, it might look like some of the KB Articles and hotfixes below does not apply to you, or you don’t have that problem in your environment. But if it’s related to Cluster, Hyper-V or any other component that you do use, it might be wise to install it anyway as it could fix 10 other problems that you are not aware of.

I’ll always import the updates directly into WSUS and deploy them, so I can use approval rules and see reporting of which updates has been installed where. Here is a good guide for how to do it; http://www.thirdtier.net/2013/03/how-to-manually-add-a-hotfix-to-wsus/

There is as far as I know (and I’ve also asked Premier Support) no way to script the import of updates into WSUS directly from Windows Catalog. You will have to manually use a Web Browser to import them. Click, Click, Click, wait, Click, Click….

The list is ordered by release date so the latest hotfixes are at the top. And looking at a fresh Fabric deployment, it looks like most hotfixes older than 10/14/2014 has been superseded, except for KB2965733 which was still needed by a couple of servers in this new fresh environment. But things might be different for you. wsus1

I’ve also written a Powershell Script for SCVMM to create Baselines and import all Updates and Hotfixes there. So it’s easy to use compliance scans and use remediation from SCVMM to keep the Fabric updated.
You can find it here; https://gallery.technet.microsoft.com/scriptcenter/SCVMM-Automatic-Baseline-8779597b

It’s not that easy to find new hotfixes or to know which ones are mandatory. Luckily, there is a blogpost to help you out. I’ve collected all sources from Microsoft product teams, where they list the hotfixes they recommend.
You can find the lists here: http://www.isolation.se/list-of-resources-to-find-hotfixes-and-updates-for-windows-server-2012-r2/ 

 

Anyway, here is the long list of fixes for possible problems in your environments. Updated: 7/22/2015

Hyper-V cluster unnecessarily recovers the virtual machine resources in Windows Server 2012 R2

http://support.microsoft.com/kb/3072380   Released: 7/14/2015

Virtual machines that host on Windows Server 2012 R2 may crash or restart unexpectedly
http://support.microsoft.com/kb/3068445   Released: 7/14/2015

Added 07/22/2015    “0xc0000017” error when you restart a UEFI-based computer in Windows
https://support.microsoft.com/kb/3072381   Released: 7/13/2015

Interrupts to the Intelligent Platform Management Interface driver are missed in Windows Server 2012 R2
http://support.microsoft.com/kb/3061460   Released: 6/9/2015

Unexpected ASP.Net application shutdown after many App_Data file changes occur on a server that is running Windows Server 2012 R2
http://support.microsoft.com/kb/3052480   Released: 6/9/2015

Update adds support for compound ID claims in AD FS tokens in Windows Server 2012 R2
http://support.microsoft.com/kb/3052122   Released: 6/9/2015

Update to improve the backup of Hyper-V Integrated components in Hyper-V Server 2012 R2
http://support.microsoft.com/kb/3063283   Released: 6/9/2015

Stop error code 0xD1, 0x139, or 0x3B and random crashes in Windows Server 2012 R2
http://support.microsoft.com/kb/3055343   Released: 5/12/2015

Backup application that calls the VSS service becomes unresponsive when the DFSR service is running in Windows
http://support.microsoft.com/kb/3054249   Released: 5/12/2015

Resolution of external DNS records on a Windows Server 2012 R2 Hyper-V guest cluster fails through a Hyper-V Network Virtualization Gateway
http://support.microsoft.com/kb/3049448   Released: 5/12/2015

Shared Hyper-V virtual disk is inaccessible when it’s located in Storage Spaces on a Windows Server 2012 R2-based computer
http://support.microsoft.com/kb/3025091   Released: 5/12/2015

“The URL cannot be resolved” error in DirectAccess and routing failure on HNV gateway cluster in Windows Server 2012 R2
http://support.microsoft.com/kb/3047280   Released: 5/12/2015

Hyper-V host crashes and has errors when you perform a VM live migration in Windows 8.1 and Windows Server 2012 R2
http://support.microsoft.com/kb/3031598   Released: 4/14/2015

Hotfix enables AD FS token replay protection for Web Application Proxy authentication tokens in Windows Server 2012 R2
http://support.microsoft.com/kb/3042121   Released: 4/14/2015

“HTTP 400 – Bad Request” error when you open a shared mailbox through WAP in Windows Server 2012 R2
http://support.microsoft.com/kb/3042127   Released: 4/14/2015

Files cannot be copied when drive redirection is enabled in Windows 8.1 or Windows Server 2012 R2
http://support.microsoft.com/kb/3042841   Released: 4/14/2015

“STATUS_PURGE_FAILED” error when you perform VM replications by using SCVMM in Windows Server 2012 R2
http://support.microsoft.com/kb/3044457   Released: 4/14/2015

You cannot upgrade Hyper-V integration components or back up Windows virtual machines
http://support.microsoft.com/kb/3046826   Released: 4/14/2015

RDP session becomes unresponsive when you connect to a Windows Server 2012 R2-based computer
http://support.microsoft.com/kb/3047296   Released: 4/14/2015

“Your computer can’t connect to the remote computer” error because RD Gateway service freezes in Windows Server 2012 R2
http://support.microsoft.com/kb/3042843   Released: 4/14/2015

A SQL Server that is running in a Hyper-V virtual machine takes a long time to restore a database to a dynamic VHD
http://support.microsoft.com/kb/2970653   Released: 3/10/2015

DNS server does not try the second forwarder and other DNS improvements in Windows Server 2012 R2
http://support.microsoft.com/kb/3038024   Released: 3/10/2015

“0x000000D1” Stop error when you fail over a cluster group in Windows Server 2012 or Windows Server 2012 R2
http://support.microsoft.com/kb/3036614   Released: 3/10/2015

Hotfix for update password feature so that users are not required to use registered device in Windows Server 2012 R2
http://support.microsoft.com/kb/3035025   Released: 3/10/2015

AD FS cannot process SAML response in Windows Server 2012 R2
http://support.microsoft.com/kb/3033917   Released: 3/10/2015

Added 7/18/2015    “0x0000003B” or “0x0000007E” Stop error on a Windows-based computer that has 4K sector disks
https://support.microsoft.com/kb/3027108  Released: 2/10/2015

Custom values for various MPIO timers in Windows Server 2012 R2 may not be honored
http://support.microsoft.com/kb/3027115   Released: 2/10/2015

System may freeze if a reserved disk is mounted accidentally in Windows 8.1 or Windows Server 2012 R2
http://support.microsoft.com/kb/3027110   Released: 2/10/2015

RemoteApp window is too large or too small when you use RDP to run a RemoteApp application in Windows Server 2012 R2
http://support.microsoft.com/kb/3026738   Released: 2/10/2015

Operation fails when you try to save an Office file through Web Application Proxy in Windows Server 2012 R2
http://support.microsoft.com/kb/3025080   Released: 2/10/2015

You are not prompted for username again when you use an incorrect username to log on to Windows Server 2012 R2
http://support.microsoft.com/kb/3025078   Released: 2/10/2015

Hotfix to avoid a deadlock situation on a CSV file system volume on Windows Server 2012 R2
http://support.microsoft.com/kb/3022333   Released: 2/10/2015

You are prompted for authentication when you run a web application in Windows Server 2012 R2 AD FS
http://support.microsoft.com/kb/3020813   Released: 2/10/2015

Time-out failures after initial deployment of Device Registration service in Windows Server 2012 R2
http://support.microsoft.com/kb/3020773   Released: 2/10/2015

You are prompted for a username and password two times when you access Windows Server 2012 R2 AD FS server from intranet
http://support.microsoft.com/kb/3018886   Released: 2/10/2015

Cluster fixes for deadlock and resource time-out issues in Windows Server 2012 R2 Update 1
http://support.microsoft.com/kb/3023894   Released: 2/10/2015

RDS License Manager shows no issued free or temporary client access licenses in Windows Server 2012 R2
http://support.microsoft.com/kb/3013108   Released: 12/9/2014

iSCSI SAN server that’s running Windows Server 2012 R2 restarts unexpectedly on a high-speed network
http://support.microsoft.com/kb/3000123   Released: 11/11/2014

TRIM and UNMAP activities for thin provisioning on one volume block all activities on other volumes
http://support.microsoft.com/kb/2996802   Released: 11/11/2014

SMBv1 named pipe requests do not time out when the remote server hangs in Windows 7, Windows Server 2008, Windows 8.1, and Windows Server 2012 R2
http://support.microsoft.com/kb/2995054   Released: 10/14/2014

SMB 3.0 Transparent Failover feature does not work after you disconnect a drive cable in Windows
http://support.microsoft.com/kb/2991247   Released: 10/14/2014

WTSQuerySessionInformation API function always returns zero bytes for WTSIncomingBytes and WTSOutgoingBytes
http://support.microsoft.com/kb/2981330   Released: 10/14/2014

A network printer is deleted unexpectedly in Windows
http://support.microsoft.com/kb/2967077   Released: 8/12/2014

“0x00000018” Stop error when volumes are mounted in Windows Server 2012 R2 or Windows Server 2012
http://support.microsoft.com/kb/2973052   Released: 8/12/2014

Updates to improve the compatibility of Azure RemoteApp in Windows 8.1 or Windows Server 2012 R2
http://support.microsoft.com/kb/2977219   Released: 8/12/2014

Error 58 when an application calls BackupRead function to back up files that are shared by using SMB in Windows
http://support.microsoft.com/kb/2973055   Released: 7/8/2014

2965733 The guest cluster is not available to service users after failover in a Hyper-V Network Virtualization environment
https://support.microsoft.com/kb/2965733   Released: 6/10/2014

NFS version 4.1 and version 3 work unexpectedly in Windows Server 2012 R2 or Windows Server 2012
http://support.microsoft.com/kb/2934249   Released: 4/8/2014

CSV snapshot file is corrupted when you create some files on the live volume in Windows
http://support.microsoft.com/kb/2929869   Released: 4/8/2014

On-demand virus scan freezes in Windows
http://support.microsoft.com/kb/2904100   Released: 3/11/2014

Windows Server 2012 R2 or Windows 8.1 crashes when virtual volumes are exposed to hyper-v virtual machines
http://support.microsoft.com/kb/2925766   Released: 2/11/2014

iSCSI Target stops responding to requests in Windows Server 2012 R2
http://support.microsoft.com/kb/2919740   Released: 2/11/2014

Memory and deadlock issues for the RD Virtualization Host and RD Connection Broker role services in Windows 8.1
http://support.microsoft.com/kb/2908810   Released: 2/11/2014

Hotfix improves storage enclosure management for Storage Spaces in Windows 8.1 and Windows Server 2012 R2
http://support.microsoft.com/kb/2913766   Released: 1/14/2014

OffloadWrite is doing PrepareForCriticalIo for the whole VHD in a Windows Server 2012 or Windows Server 2012 R2 Hyper-V host
http://support.microsoft.com/kb/2913695   Released: 1/14/2014