Donnerstag, 20. Februar 2014

Hyper-V High Ping Latency


Had an interesting issue today, an insanely slow, brand new server….. or so I thought. First a bit of background on the client, they are fairly large, with over 400 client access devices to maintain, not including server, network equipment etc. to support this the client has 3 servers, one purchased a year and the oldest one thrown out, keeping all devices in warranty and with modern powerful equipment to keep things running, in addition to this there are two other servers that are replaced every three years, these are treated differently as these are speciality servers, and only do one task.

So, building a new server for a client, in this case a HP ML350p gen8 with Windows Server 2012 R2 STD running as a Hypervisor, nothing special in that, I do this exceedingly regularly so it has become more of a routine build for me. What got me with this one though was during testing I was getting insanely high ping latency, not only to the virtual machines from the network and vice versa, but also from the hypervisor to the machines and vice versa. Pings to other virtual machines on others server, on different LAN segments were all responding normally in <1MS

My first thought was there was something wrong with the virtual machines and that I had butchered something in the migration, but as they worked on other hypervisors without delays, that knocked that one on the head. Then I thought network location issue, but that does not make any sense due the fact that pinging from a hypervisor to guest does not go across a physical network, so it has to be the brand new server.

Ok, so what’s new about this server, well its got newer Processors, greater memory, faster HDD’s with larger capacity’s, basically it was more a case of what wasn’t different to the last server. Not going to go through the whole process of troubleshooting, but basically it was to do with the NIC’s, fine now what about them is it. After trial and error, and of course every techs most important tool, Google I came across the issue what is it…



THE ISSUE IS VMQ or Virtual Machine Queuing inside the Broadcom NIC drivers as shown below, disable this and the issue clears instantly





Pings and other indicators are now back down to <1ms which is what I expected to see in the first place.

Hardware effected by this was as follows

HP ML350p gen8
Server 2012 R2 Standard
onboard Broadcom Quad Port NIC

Sonntag, 2. Februar 2014

High Ping Latency in Hyper-V Virtual Machines



While working on a Hyper-V 3.0 deployment recently, I noticed that all of the virtual machines running on all newly built hypervisors were showing an unhealthy high ping latency trying to ping anything on the local network. Pinging the same destinations from the hypervisors resulted in perfectly reasonable 1ms ping times, but pings from inside the virtual machines varied from 1ms all the way past 200ms.

I started digging, thinking initially that it was just a clock synchronization issue that we saw on AMD as well as Intel chips, specifically on HP DL servers, in Windows Server 2003 days. The issue is not very well documented but one of the few Microsoft articles that attempts to explain it here. Solution to that issue is to add /USEPMTIMER switch to the boot.ini file. It used to fix negative ping times on AMD chips as well as lengthy ping times on Intel chips.

Windows Server 2012 does not have a boot.ini file obviously, so the way to add this switch is to issue the following command in elevated command line or PowerShell:bcdedit.exe /set USEPLATFORMCLOCK on


(This explains the right side of the screenshot above – boot configuration is shown “before” and “after” running this command).

Unfortunately, applying this configuration at the hypervisor level and rebooting the server made no difference to ping times from VM (guest partitions).
Virtual Machine Queues

Virtual machine queues were introduced in Windows Server 2008. The purpose of this feature was to improve network performance of virtual machines receiving a lot of inbound traffic, by providing a more direct access to the hardware NIC. To quote directly from Microsoft:

“When VMQ is enabled, a dedicated queue is established on the physical network adapter for each virtual network adapter that has requested a queue. As packets arrive for a virtual network adapter, the physical network adapter places them in that network adapter’s queue. When packets are indicated up, all the packet data in the queue is delivered directly to the virtual network adapter. Packets arriving for virtual network adapters that don’t have a dedicated queue, as well as all multicast and broadcast packets, are delivered to the virtual network in the default queue. The virtual network handles routing of these packets to the appropriate virtual network adapters as it normally would.”
Resolution

When the hypervisors were built, VMQ features of the network card on the physical host were enabled to achieve better VM network performance. Through disabling performance features such as VMQ, TCP Chimney, and Receive Side Scaling, it turned out that VMQ was the root cause for high ping latency. As soon as VMQ was disabled at the parent partition’s NIC driver level, VM ping times got down to a steady <1ms reading.

The hardware/software affected by this issue:
HP DL 360p Gen8 server chassis
4-port 1Gbps Broadcom NIC (HP Ethernet 1Gb 4-port 331FLR Adapter)
Windows Server 2012 Datacenter Edition
Broadcom’s NIC driver dated October 26 2012, version 15.4.0.17 (link to HP site)

Bottom line, always perform at least basic QA of your builds before putting them into production. Some hardware (and more likely drivers) may cause performance issues instead of providing a performance boost - especially in early releases.

Samstag, 21. Dezember 2013

Reset Administrator Password on ILO

ESXI


Thanks to HP drivers, ILO can be configured directly inside ESXi without the need for a server reboot: going into the command line (locally or via ssh) you can move to /opt/hp/tools, and there you will find the commandhponcfg. This tool can be used to configure ILO:
first, write down the actual configuration of ILO. The easiest way to do it is by saving in an XML file:
1/opt/hp/tools # ./hponcfg -w ilo.xml
2HP Lights-Out Online Configuration utility
3Version 4.0-10 (c) Hewlett-Packard Company, 2011
4Firmware Revision = 1.81 Device type = iLO 2 Driver name = hpilo
5Management Processor configuration is successfully written to file "ilo.xml"
then, edit the XML file and configure your desired parameters:
01<!-- HPONCFG VERSION = "4.0-10.0" -->                                                                                                           
02<!-- Generated 8/6/2012 11:11:4 -->                                                                                                             
03<RIBCL VERSION="2.1">                                                                                                                           
04 <LOGIN USER_LOGIN="Administrator" PASSWORD="password">                                                                                                                                                                                                                            
05  <RIB_INFO MODE="write">                                                                                                                       
06  <MOD_NETWORK_SETTINGS>                                                                                        
07    <SPEED_AUTOSELECT VALUE = "Y"/>                                                                                                                                                                                                                               
08    <IP_ADDRESS VALUE = "10.123.183.197"/>                                                                                                        
09    <SUBNET_MASK VALUE = "255.255.255.0"/>                                                                                                                                                                                                      
10    <DNS_NAME VALUE = "esx-ilo"/>                                                                                                                                                                                                           
11    <DHCP_ENABLE VALUE = "N"/>                                                                                                                  
12    <DOMAIN_NAME VALUE = "domain.local"/>                                                                                                                                                                                                                                                                                                 
13  </MOD_NETWORK_SETTINGS>                                                                                                                       
14  </RIB_INFO>                                                                                                                                                                                                                                                                    
15 </LOGIN>                                                                                                                                       
16</RIBCL>


Note: you will not need to edit everytime all the parameters, it will be enough to add the lines you want to change. But, you will always have to add the username and password line to authenticate the changes you are doing.load the new configuration into the ILO, it will take some time to complete:
1/opt/hp/tools # ./hponcfg -f ilo.xml
2HP Lights-Out Online Configuration utility
3Version 4.0-10 (c) Hewlett-Packard Company, 2011
4Firmware Revision = 1.81 Device type = iLO 2 Driver name = hpilo
5<INFORM>Integrated Lights-Out will reset at the end of the script.</INFORM>
6
7Please wait while the firmware is reset. This might take a minute   
8Script succeeded
Finally, ILO will automatically reset to load the new parameters, and you will be able to reach the ILO via web interface and login into it. Also, ESXi will show you the new ILO configuration:

WINDOWS 

1. I installed SNMP because it was a preprequesite for HP Insight Management Agents.
2. I dowloaded and installed the HP Insight Management Agents.
3. I then downloaded HP Lights-Out Online Configuration Utility.
4. I ran into NTVDM errors trying to run the file so I just used WinRAR to extract the contents into C:\hp\ilo. I also extracted the zip file contained within the initial archive.
5. I then downloaded the HP Lights-Out XML Scripting Sample for Windows (Linux users can download the files in tgz format here or here.) extracted it and found the file I was looking for --



<ribcl VERSION="2.0">
<login USER_LOGIN="Administrator" PASSWORD="boguspassword">
<user_INFO MODE="write">
<mod_USER USER_LOGIN="Administrator">
<password value="newpass"/>
</mod_USER>
</user_INFO>
</login>
</ribcl>



6. Using notepad, I opened up the sample file and modified it slightly. Initially, I just removed the LOGIN and ran the file but HPONCFG gave me a syntax error. I then added it back and gave the Administrator a bogus password. Apparently, the LOGIN line is required for syntax reasons but it is not actually processed.
7. Next, I opened a command line and changed directories to C:\hp\ilo and typed the following:
HPONCFG.exe /f Administrator_reset_pw.xml /l log.txt > output.txt

8. I opened up Firefox, navigated to my iLO machine and voila! I was able to login as Administrator.

If changing Administrator's password seems too scary, you can also add another user with administrator privileges. You can then login as that user and change the Administrator password via the web console. Use the following code, suited to your liking:



<ribcl version="2.0">
<login USER_LOGIN="Administrator" PASSWORD="boguspass">
<user_INFO MODE="write" >
<add_USER
USER_NAME="Chrissy"
USER_LOGIN="Chrissy"
PASSWORD="mynewpass">
<reset_SERVER_PRIV value = "Y" />
<admin_PRIV value = "Y" />
</add_USER>
</user_INFO>
</login>
</ribcl>

Rename Onedrive Business root folder

Rename Onedrive Business root folder Here is what I remember: In the Office 365 web admin pages, change the organization name to a shorte...