Tuesday, November 14, 2017

Veeam Backup & Replication: "Failed to execute script in guest OS" (Linux Guest VM on Hyper-V)

Problem:

Veeam Pre-Freeze / Post-Thaw .sh Scripts Fail on Linux Guest VMs (e.g. Ubuntu) with "Failed to execute script in guest OS" although the scripts run fine. I use Hyper-V but that should not matter.

Failing scripts are configured to "Require successful script execution" under "Application-Aware Processing Options" under "Guest Processing" in the Veeam backup job.

My Solution:

Some commands executed inside the scripts seem to return error output which is passed back to Veeam through the script and confuse Veeam so it reports that the script was not successful.
So we must redirect error output from such commands to 2>/dev/null or some file, otherwise error status is passed back to this script and Veeam reports a failure. For information about discarding error output, see https://bash.cyberciti.biz/guide//dev/null_discards_unwanted_output


Additional information:

Also make sure that Veeam scripts (*.sh) are located on the Veeam Backup & Replication Server, NOT on the Linux guest. To me this did not become clear from Veeam docs! Veeam will copy the scripts every time to the Linux Guest /tmp/ directory, execute them and then remove them again. This should work if you have created a user that is able to elevate to root (add it to sudo group and modify the sudoers file).

Format script files as UNIX in Notepad++ when writing them on Windows.

Monday, October 6, 2014

SOLVED: Network connection problems (shares, domain) after Windows 8.x wakes up from standby (sleep)

The Windows 8.x Standby / Wakeup from Sleep Problem:


Some time ago I set up multiple Dell Optiplex 9020 mini towers with Windows 8.1. After testing standby / sleep mode I noticed a very strange and annoying problem: Whenever the PCs are waken up from sleep (standby) all connections to open network shares and open Office 2010 documents are lost. All shares are on a Windows 2012 R2 ESSENTIALS server + domain controller which runs as a Hyper-V VM. So after waking up a Dell 9020 from standby I get my Win 8.1 login prompt, log in, but for any open shares that were open when the PC went to sleep in explorer, now I randomly get either a domain authentication error message or a new authentication prompt that asks me for domain username and password. The connection to open Word 2010 docs is always lost and Word tells me that the computer came back from standby but a connection problem occured and the document is now read-only, i.e. it must be saved as a new document. Very annoying!

 

What I tried (without success):

I googled around a lot and found some hints that might be related to the problem:

1. Many people recommended to go to network adapter settings in device manager and completely disable standby for the network adapter, i.e. uncheck "Allow the computer to turn off this device to save power". I found out that similar suggestions date back to Windows 7 or even XP. Unfortunately, this did not help. I also tried tweaking some more advanced power settings of the drivers but without success. The problem remained. BTW, I don't see the "computer can turn off this device to save power" in Intel's latest drivers anyway. There are some other power management options but not this one.

2. I tried installing the latest original Intel network drivers instead of Dell's. No success.

3. On Dell's forums people reported that there are problems with the Dell Optiplex 9020 I217-LM NICs in standby, flooding the network with packets. The problems were related to Standby, IPv6, or even AMT. But I already had the latest drivers and BIOS and no AMT enabled. So this was not the problem.

4. I tried tweaking some advanced power management settings like hybrid sleep, etc. Again, no success.

5. Maybe the problem was related to IPv6? I tried turning it off but again no success.

6. Maybe it was the virtual Server 2012 R2 domain controller? Turned out this is not the case (see below).

7. Some people reported that a similar problem occurs with intelligent switches. I have a HP ProCurve 1810G. Dell ProSupport could not help me to solve the problem but they confirmed similar problems that could be solved with latest switch firmware updates. Indeed, attaching a simple Netgear gigabit switch seemed to solve the problem. So I spent much time updating the switch firmware and tweaking the switch options and finally decided to buy a brand new Netgear intelligent switch, just to make sure that the problem was not caused by the HP switch. Unfortunately, the same behavior appeared with the new switch.

8. There was a Microsoft Office bug that caused problems when waking up a PC from standby but that should have been fixed meanwhile. A Microsoft fix did not help either.

... I probably tested even more "solutions" but neither did help...

 

Solution to the Standby problem (cause: VirtualBox Bridged Network Driver):

Right before giving up and turning off standby in Windows completely I suddenly found a thread in Microsoft's forums (don't remember the url) where someone suggested that Oracle's VirtualBox may be causing similar problems. And in fact I had installed VirtualBox on each PC because I had virtualized the old XP computers to have them as VMs and to be able to run the old software.

Indeed, the issue was caused by the Oracle VirtualBox Bridged Network Driver. So I disabled (unchecked) it on the network connection (where the IPv4, IPv6, etc. protocol settings are done) and no more problems. The computers now come out of standby and all network connections (shares, open Word docs) are still open. Everything seems to work properly.

BTW, I am using "NAT" with VirtualBox, so I don't even need "Bridged".

Hope this helps!

Anguel

Tuesday, August 12, 2014

Changing the Reserved IP Address in a Static DHCP Reservation in Windows Server

 

Static IPs preferred

DHCP is a good thing if it works - set and forget. But what happens if the DHCP server is down for some reason? Exactly - the whole network stops working. Another problem of DHCP are those random IPs a client gets each time. This is very bad for printers that use DHCP. Therefore I prefer static IPs over dynamic.

 

Windows Server 2012 R2 Essentials: You cannot stop Windows LAN Configuration Service anymore


Unfortunately, there is some strange behavior on Server 2012 R2 Essentials if you just enter static IPs on your Windows clients - your Windows OS will show exclamation marks on your network connection from time to time, etc. There is actually a Windows Server LAN Configuration service which keeps overwriting your manual settings and the bad thing is that you cannot stop that service in Server 2012 R2 any more for some reason...So, instead, I decided to use DHCP on the server and add an Alternative IP Configuration with all static data as a fallback if my server is down. This "alternative configuration" becomes available in the client's IP Settings as a new tab as soon as you enable automatic DHCP configuration on the main tab.

 

Creating Static DHCP Entries in Windows Server

So DHCP is running but I still need static IPs that don't change. To achieve this in Windows Server DHCP configuration I need to find the leases, right click and convert them to static reservations. Then under reservations if I look at the properties I see that now my client will always get a static IP. But wait! This IP is set by my DHCP server and not by me. What if I want to change that IP as I like? E.g. I want it to be the same as the static IP I have set in the alternate IP configuration on my clients as described above. But it is greyed out for some reason - only Microsoft knows why. Sometimes I wonder how stupid programmers can be. Oh sure, it is more important to add new stupid features to a server OS than making the everyday features we use work more efficiently.

I googled around and even found this Microsoft Technet Howto from 2005 (!) regarding Windows Server 2003 (!):

http://technet.microsoft.com/en-us/library/cc758459%28v=ws.10%29.aspx

It states:

The IP address in an existing client reservation cannot be modified directly. To achieve similar results, the existing reservation must be deleted and a new reservation added.
How stupid is that?!?!?!

 

Changing the Reserved IP Address


So, to make my desired static IPs work with DHCP I need to go through these "simple" steps (thank you Microsoft!):
  1. Open the DHCP configuration on my Windows Server
  2. Find the address lease for the particular client - this is some random IP the DHCP server gave to my client
  3. Create a reservation form that lease (right click + add to reservations)
  4. Find the new reservation under reservations
  5. Open reservation properties by right clicking - PROBLEM: IP is still "random", defined by the DHCP server and it is grayed out - can't change it!
  6. Create and open a text file...
  7. Manually copy the host name and the MAC address form the fields to my text file
  8. Delete this reservation - isn't that stupid?
  9. Create a new reservation by right clicking on reservations - once again...
  10. Copy and paste the hostname and the MAC address from the text file to the fields of the new reservation
  11. Finally: Enter your desired IP for the new reservation - that's all we wanted to achieve
Again: How complex is that Microsoft? Every modern router is easier to configure.



Hope this helps!

Anguel

Wednesday, April 9, 2014

Backup to USB disks: Do-it-yourself setup for USB drive rotation / swapping

I was looking for a simple but effective solution to use external USB disk drives attached to my server for backup with backup applications that do not directly support USB drive rotation / swapping.

Unfortunately, Windows does not seem to support assigning the same drive letter to different USB drives, e.g. you cannot assign the letter U: to two different USB drives. This would be nice to be able to rotate the drives but it does not work. So my idea was to use a network share name that automatically mounts different USB drives to the same share name, depending on which USB drives are connected, that creates some sort of priority as we will see. An additional advantage is that drive can be also accessed directly from other computers through the network share.

The first step is to assign different drive letters to each USB disk through Windows disk management, e.g. U: and V:. Then create the same directory to be shared on each drive, e.g. "USB_SHARE". Backups will be stored there.

Then a simple batch file (usb_share.bat) can be created. First the share name USB_SHARE is auto-deleted (if it should already exist). Then we attempt to connect that share name to the USB drives. This will of course only succeed if the corresponding drive is attached to the PC. I.e. if U: is attached, it will be mounted as the share, if it is not attached, V: will be attempted to be mounted as the share. With "grant" we can give the appropriate network permissions, make sure that NTFS permissions also apply. So we can enter the following in the batch file:

net share USB_SHARE /delete
net share USB_SHARE=U:\USB_SHARE /grant:Everyone,FULL
net share USB_SHARE=V:\USB_SHARE /grant:Everyone,FULL


This batch file can be executed manually after connecting a USB drive and it can also be setup to be run automatically on computer startup, e.g. through the task scheduler on a Windows Server OS (make sure you tell Windows to execute it with admin permissions).

Hope this helps!

Anguel

Wednesday, March 26, 2014

SOLVED: Making Wake-on-Lan (WOL) work in Windows 10 / 8.x

WINDOWS 10 UPDATE:
THIS FIX ALSO APPLIES TO WINDOWS 10, IT IS EVEN MORE IMPORTANT,  BECAUSE WINDOWS 10 TURNS "FAST STARTUP" (read below) BACK ON AFTER UPDATES (yes, Microsoft does not stop creating nonsense features / bugs).
TO DISABLE FAST STARTUP ON WIN 10 THROUGH GROUP POLICY PREFERENCES CREATE THE FOLLOWING KEY:
HKLM\System\CurrentControlSet\Control\Session Manager\Power\HiberbootEnabled
and set its value to 0 !

I had very serious problems getting Wake-On-Lan (WOL) to work on my new Dell Optiplex 9020 MT (MiniTower) on Windows 8.1 Pro. I finally got this to work and would like to share my experience here.

Note: At the time of this writing the current Dell BIOS for Optiplex 9020 MT was A05.
UPDATE: The same behavior is also observed with DELL BIOS A07. Maybe Dell needs to comply with some power saving requirements and therefore enables Deep Sleep instead of Wake-On-LAN? Whatever the reason is, it is extremely annoying that this is the default and it is not clearly documented.
Note: I am using UEFI and Secure Boot.

UPDATE February 2017:  Besides the fix described here it might happen that you experience strange problems with your Dells like NOT waking up on WOL or the other way round - "MAGICALLY" waking up randomly for no reason (you see things in Windows log like "firmware" as start reason) - in both cases I recommend updating to latest Intel network drivers (the Dell drivers are often outdated). I don't know what is gouing on there with Dell hardware vs Microsoft vs Intel issues but it has become a nightmare to get simple things to work reliably these days...

WOL vs Dell Optiplex 9020

In Dells forums people recommended to downgrade the Dell BIOS or even to get their mainboard changed to get WOL to work:

http://en.community.dell.com/support-forums/desktop/f/3514/t/19539946.aspx?pi21953=1

I don't like doing such things because newer BIOS also has other bugs fixed and downgrading may even break something.

So I decided to look for a better solution. Actually nowadays things don't get easier but more and more complex as we will see...

How to test WOL?

First I downloaded a nice little utility and installed it on another PC to be able to send WOL wake-up packets to my Dell PC. It is called WinWOL is free and I found it here:
http://www.telnetport25.com/2013/01/winwol-windows-wake-on-lan-version-1-0/

Of course you can also use another WOL packed sending utility.

1. Enable Wake on Lan in the BIOS

So the first step is to make sure your BIOS is set correctly:

1. The obvious WOL setting in BIOS is under Power Management > Wake on LAN. You have to enable it by setting it to LAN Only for example.

2. The not so obvious setting is under Power Management > Deep Sleep Control. You must set this to Disabled, otherwise WOL will just not work.

2. Enable Wake on Lan in the Network Adapter Settings

Now in Windows go to the Device Manager > Network Adapter > Intel Ethernet Connection I217-LM, in my case.
Open the adapter settings and go to the Power Management tab
Make sure that all Wake on LAN options are enabled. I actually did not change any other energy settings on this tab, only activated the WOL options.


These settings should already make it possible to use wake on LAN in sleep and hibernate modes (BTW: hibernate mode must be explicitly enabled in Windows 8 (some of Microsoft's "brilliant" developers decided to do so).

What about WOL after shutdown?

The problem that actually caused all my headaches was the fact that I tested WOL after shutting down the PC from Windows. My PC just did not wake up after a normal Windows shutdown.

Normally, in a corporate environment you can't expect your users to put their PCs in sleep or hibernate only - they will also shut them down if they like. And shutdown definitely did not work with the WOL settings above alone.

Then I noticed the following: To have at least a chance that Wake-on-Lan will work you must make sure that the some lights are still on on your Dell's network adapter while attempting to wake up the PC via LAN.

But there was a problem: After a Windows shutdown of the PC all network card lights were OFF! Then I found out one more thing: If I switch on my Dell through the power button and then turn it off immediately (before Windows boots up) the network adapter lights are still ON - this means that WOL is possible! But after a Windows shutdown they stay OFF. So WOL is not possible.

So to me the situation was clear: Something in Windows shutdown makes my Dell turn off the network adapter completely, although WOL is enabled in  BIOS and in the Windows network adapter settings.

Microsoft did it again! Shutdown in Window 8 is "hybrid" and explicitly does not allow WOL!

After googling a lot I finally found an official Microsoft article describing that WOL behavior has changed in Windows 8 / 8.1:

http://support.microsoft.com/kb/2776718

Quote:
In Windows 8 and Windows 8.1, the default shutdown behavior puts the system into the hybrid shutdown state (S4), and all devices are put into D3. WOL from S4 or S5 is unsupported. Network adapters are explicitly not armed for WOL in either S5 or S4 cases because users expect zero power consumption and battery drain in the shutdown state. This behavior removes the possibility of invalid wake-ups when an explicit shutdown is requested. Therefore, WOL is supported only from sleep (S3) or hibernation (S4) states in Windows 8 and 8.1.

So WOL is explicitly disabled in shutdown in Windows 8 and 8.1. It is nice to hear that this is expected by Microsofts "brilliant" developers (they seem to expect a lot in Windows 8.x ...) but what I expect is WOL to work in a corporate environment and not to turn it off just because someone decided to save battery, especially on a desktop workstation (!) in this case.


WOL and Hybrid Shutdown in Windows 8.x

Hybrid shutdown for faster startups

Another strange thing is that this Windows 8.x WOL behavior seems to be related to hybrid shutdown which MS describes as follows:

In Windows 8 and 8.1, hybrid shutdown (S4) stops user sessions but lets the contents of kernel sessions be written to the hard disk. This enables faster startups.

Why are faster startups related to Wake-on-LAN?



I don't know - ask Microsoft and their "brilliant" engineers!

Disable fast startups and get WOL to work!


So according to Microsofts article you can disable fast startups = hybrid sleep as follows (of course not recommended, bla bla bla ...):

To disable the S4 state in Windows 8 and 8.1, follow these steps.

Note We do not recommend that you disable the hybrid shutdown (S4) state.

  1. In Control Panel, open the Power Options item.
  2. Click the Choose what the power buttons do link.
  3. Clear the Turn on fast startup (recommended) check box.
  4. Click Save Settings.
Another important annoying fact: If it seems that you cannot just change the power button settings but you need to first click on the text saying something like: "Some settings are currently not available..."  Again, very "user friendly", Microsoft...

It works!

The most amazing thing is that disabling the fast startup setting above really makes WOL work after Windows shutdown. Now if I shutdown my Dell from Windows 8.1 it leaves the network adapter powered, i.e. the green LED is lit and the orange LED is blinking. Sending a magic packet to the Dell wakes it up.

So it looks like this stupid WOL behavior is expected and wanted by Microsoft! So I am not sure if Dell will ever give us a BIOS update that will change it.

IMHO one thing is certain:

It has never been harder to get Wake-on-LAN to work!


I hope this helps! Again, thank Microsoft for wasting our time and making our life as hard as possible. In Windows 8.x everything illogical is possible! Expect the unexpected! Expect surprises everywhere!

Anguel

Tuesday, March 18, 2014

Domain Controller on top of Hyper-V - Firewall problem

The problem:


I have:

1. A single physical server which is a Hyper-V host running Windows Server 2012 R2.
2. A single Windows Server 2012 R2 Essentials Guest VM running on top and acting as a domain controller.

Now I have researched a lot whether I should join the host to the domain running on top of it or leave it in a separate workgroup. There is no best practice but many sources confirm that this is possible and joining the domain offers many management benefits, e.g. here see Option #4:
http://blogs.msdn.com/b/virtual_pc_guy/archive/2008/11/24/the-domain-controller-dilemma.aspx

Now the problem is that after having joined the host to the domain sometimes everything works fine and the host manages to join the domain on startup but sometimes this seems to fail - I think that the join process times out as the DC in the Guest VM does not start up in time. In this case the firewall tells me that the host is connected to a "public network" instead of the "domain network" and I cannot connect to the server via remote desktop as this is not allowed by default. If I disable and reenable the network adapter the "domain network" is recognized fine but this has to be done manually.

The solution:

In the meantime I am pretty sure that I solved the problem:

On my host I had entered two DNS servers: The first was the IP of the Server Essentials Guest VM and the second was Google's DNS server (8.8.8.8); my idea was to have access to the internet even if my Guest VM DNS does not run.

Now I am pretty sure that this second DNS server confused the firewall configuration on startup - it could not detect the primary DNS because the Guest still has not started up, but then saw this second DNS server and fell back to "public network" instead of the desired "domain network" mode. After removing Google's DNS server from the IP v4 configuration now the host seems to start up fine and always ends up in the "domain network".

Delayed domain join?

I really like the idea to delay domain joining on start up but I have not found a way to do this. I only found how to add longer timeouts but this is not really solving all problems:

http://superuser.com/questions/328739/how-to-delay-windows-7-autologon-so-that-the-domain-will-be-available

Any hints on completely delaying domain join on startup are welcome.

Anguel

Friday, March 14, 2014

How do client PC backups work in Windows Server 2012 R2 Essentials?

The problem:

I was trying to get my Server 2012 R2 Essentials (Hyper-V VM) to do nightly backups of my Win 8.1 client PCs. As far as I had understood from the Microsoft presentations the server should be able to wake up the clients automatically from hibernate (I selected this option during setup of the connector). Unfortunately, Microsoft does not care about providing serious docs for their Windows / Server products. What is really annoying is that I don't even find any documentation how the wake up procedure actually works. Does it use Wake On Lan? Are there any magic packets involved? You pay a lot and get something that either does not work or does not tell you how it works.

What I had tested so far: I had setup the server to start client backups at lets say 10:00. So at 9:50 I put my client computer in hibernate and wait and wait ... but nothing happens. It is very sad to have to guess and wait multiple hours or days to see if backup actually works on a server system. I still don't know if the server uses wake on lan or sets some timer on the client computer to wake it up. If I know it uses WoL I will also check my BIOS and network adapters. But this is a nightmare how a server OS is sold without any serious docs.

The solution:

Finally I found a very nice YouTube video that explains in details how the whole process works:

According to the video, the server simply schedules a task on the client PCs at some time in the range defined on the server: E.g. if I enter 11:00 - 12:00 this task seems to set a random wake-up time at e.g. 11:25 on the client PC. This task is then scheduled to wake up the PC - didn't know that this is possible.

It seems to work now. As discussed above the clients are woken up by the task set by the server in the task scheduler. This works in hibernate mode, not in shutdown. Still wondering how the server exactly schedules backups as it does not know how long a client backup will take and therefore does not know when to schedule the next PC...

In each case I am still very very disappointed and annoyed because of the lack of docs for the latest Windows Client/Server products.
 
IMHO Windows Server should have offered the possibility to use Wake On Lan too. This is a much nicer way to do backups. It can't be that hard for Microsoft to send out a magic packet which would also be able to wake up a client when the user has shut it down.

Here is the link to the video I talk about: http://www.youtube.com/watch?v=c6ZOWjaZpIk

BTW: Another possible problem may be some special BIOS power settings, for example on my brand new Dell PCs there is a default Deep Sleep option in the BIOS concerning states S4 and S5. When this option was turned on (default) it prevented Wake On Lan from working at all, although it was enabled in the BIOS, see e.g. here:
http://en.community.dell.com/support-forums/desktop/f/3514/t/19434669.aspx

I first thought that Deep Sleep may be causing my wake-up problem but now according to the video above backup seems to rely on task scheduler only and not on WOL, let's see...

Hope this helps! Thank Microsoft for wasting our time so successfully!

Anguel