Posts

Showing posts with the label VMware troubleshooting

Virtual machine is inaccessible after vMotion | VMware

Issue :
VM fails to power on, power off or modify. If tried to migrate to another ESXi host the VM gets disconnected.
Error from vCenter :
VM will be shown in vCenter as inaccessible. 
Snippets from the logs from ESXi host:
[root@ESXi01:/vmfs/volumes/570248ff-86524429-7f05-848f691451f9/VMname] cat VMname.vmx | grep vmdk cat: can't open 'VMname.vmx': Device or resource busy
Troubleshooting:
The logs shows that the VM is busy or locked due to some reason. Check the lock status using vmx
[root@ESXi01:/vmfs/volumes/570248ff-86524429-7f05-848f691451f9/VMname] vmkfstools -D VMname.vmx Lock [type 10c00001 offset 151009280 v 326, hb offset 3198976 gen 61, mode 1, owner 58109c0d-10ae7368-fd7b-848f69156ba8 mtime 15841905 num 0 gblnum 0 gblgen 0 gblbrk 0] Addr <4, 326, 119>, gen 129, links 1, type reg, flags 0, uid 0, gid 0, mode 100755 len 5230, nb 1 tbz 0, cow 0, newSinceEpoch 1, zla 2, bs 8192
The highlighted number in the log output 848f69156ba8 refers to the MAC address of the…

VMware ESXi host disconnects from vCenter

Issue :
ESXi hosts disconnects from vCenter and may not even connect directly using vSphere client. The VMs will continue to run. SSH to host will work. Execution of esxcfg-scsidevs -m command will hang. LUN disappears in one or more hosts in an ESXi cluster. 
Errors from vCenter : A general system error occurred: Invalid response code: 503 Service Unavailable.
Unable to communicate with the remote host, since it is disconnected.
Cannot contact the specified host. The host may not be available on the network, a network configuration problem may exist, or the management services on this host may not be responding.
Snippets from the vmkernal logs from ESXi host:
Check for non-responsive luns
[root@ESXi01:~] cat /var/log/vmkernel.log  | grep -i responsive
cpuxx:yyyyyyy ALERT: hostd detected to be non-responsive
Check vmkernel.log where device status is 0x18 which corresponds to Reservation conflict
[root@ESXi01:~] cat /var/log/vmkernel.log | grep 0x18 | head
cpuxx:yyyyy)NMP: nmp_ResetDeviceLogThrot…

VMware ESXi host disconnects and do not connect back to vCenter

Issue:
ESXi host disconnects from vCenter. And if tried to reconnect, the activity fails at 89%
Error from vCenter :
A general system error occurred: internal error Processing data from vCenter agent on ESXi01
Snippets from the vpxd logs from vCenter:
=============================================================== 2016-03-29T12:28:37.644+05:30 error vpxd[14460] [Originator@6876 sub=HttpConnectionPool-000001] [ConnectComplete] Connect failed to <cs p:000000001099ab00, TCP:IP Address:443>; cnx: (null), error: class Vmacore::Ssl::SSLVerifyException(SSL Exception: Verification parameters: --> PeerThumbprint: Thumbprint --> ExpectedThumbprint:  --> ExpectedPeerName: IP/Hostname --> The remote host certificate has these problems: -->  --> * The host certificate chain is incomplete. -->  --> * Host name does not match the subject name(s) in certificate. -->  --> * unable to get local issuer certificate) ============================================================…

VMware vSphere Update Manager 5 remediation fails

Issue

When VMware vSphere Update Manager 5 tries to remediate a host, the remediation fails at 25% with the below error:

fault.com.vmware.vcIntegrity.VcIntegrityFault.summary

VM options greyed out | VMware

Issue

We faced an issue today morning with one of our VMs hosted in VMware. The VM related options were greyed out.
Root cause
There was a snapshot job running in the background (not visible from vCenter), which prevented any administration task in the VM. This task was stuck at 0%. This activity cannot be cancelled from vCenter or from console as it was initiated by a system user called vpxuser.
Workaround
Login to the SSH console of the ESXi host holding the VM using putty.
Identify the vmid of the affected VM (In our case the vmid was 391) using the command
vim-cmd vmsvc/getallvms
Check the tasks running in background for this particular VM using the command
vim-cmd vmsvc/get.tasklist 391
See if you can cancel the task using the command
vim-cmd vimsvc/task_cancel <taskname> [Task name will be something like hatask-391-vim.virtualmachine.createsnapshot-1234567] .

In our case this was not working as the task was initiated by a system user. But in scenarios were the snapshot or any VM…

VMware vMotion error at 14% - Scene 2

Issue

While performing vMotion, the operation fails at 14% with the below error :

vMotion migration [-1062731490:1419235061251156] failed to create a connection with remote host <Destination vMotion IP>: The ESX hosts failed to connect over the VMotion network
Migration [-1062731490:1419235061251156] failed to connect to remote host <Destination vMotion IP> from host <source IP>: Network unreachable
The vMotion failed because the destination host did not receive data from the source host on the vMotion network. Please check your vMotion network settings and physical network configuration and ensure they are correct.

Resolution

I've already penned a post on VMware vMotion failure at 14%. This blog is an extended version of that post. If none of the steps mentioned in my previous post helped you, then you are in the right page.

Check whether vMotion is selected for multiple vmkernel NICs in ESXi host. !!!!

If yes, make only one NIC available for vMotion.




Interpreting VMware CPU performance metrics - RUN, WAIT, RDY, CSTP

Quick Reference guide from VMwareRun, %RUN: This value represents the percentage of absolute time the virtual machine was running on the system. If the virtual machine is unresponsive, %RUN may indicate that the guest operating system is busy conducting an operation.

VMware and Windows Interview Questions: Part 3

What is vSAN?It is a hypervisor-converged storage solution built by aggregating the local storage attached to the ESXi hosts managed by a vCenter. Recommended iSCSI configuration?A separate vSwitch, and a separate network other than VMtraffic network for iSCSI traffic. Dedicated physical NICs should be connected to vSwitch configured for iSCSI traffic.What is iSCSI port binding ?Port binding is used in iSCSI when multiple VMkernel ports for iSCSI reside in the same broadcast domain and IP subnet, to allow multiple paths to an iSCSI array that broadcasts a single IP address.

VMware vMotion error at 14%

Issue

While performing vMotion, the operation fails at 14% with the below error :

A general system error occurred: Migrtion to host <Destination ESXi IP> failed with erro Connection closed by remote host, possibly due to timeout (0xbad003f).

Migrate virtual machine:A general system error occurred: Migration to host <Destination ESXi IP> failed with error Connection closed by remote host, possibly due to timeout (0xbad003f).

vMotion migration [-1062729272:1406020861428172] (19-71629048648008) failed to receive... 

Scenarios

Scenario 1: Your management network and vmotion network are in the same subnet using the same physical NIC.

How to extend a logical volume in Windows 2003

Image
Scenario

You have a single vmdk file of 200 GB and it has two logical volumes C & D with 100 GB each. Suppose you need to add another 100 GB to the D drive making it 200 GB. What would you do ?

The steps are simple and straight forward.

Centos network adapter not working after cloning

Symptom: 

Eth0 interface will not be present for a Centos VM after cloning. Only the loopback networking interface will be available. If you try to turn up the interface manually (using the command ifup eth0 or ifup-eth0), you will receive the below error.

Device eth0 does not seem to be present, delaying initialisation 

Root Cause:

When you clone a Centos VM from a template, a new NIC card will be created for the cloned VM. In other terms, a new MAC address will be generated for the NIC of the cloned machine. This change happens only in VMware perspective and no modification is made in Centos. Therefore the kernel will be still searching for the NIC with old MAC address and hence fails.

Clone a VM without VMware vcenter

Steps for cloning VM using SSH:

SSH to the ESXi hostIdentify the path of the source (say, /vmfs/volumes/datastore1/SourceVM/ where sourceVM is the name of the source VM)Create a new folder in the desired datastore mkdir /vmfs/volumes/datastore1/DesintationVM

where DestinationVM is the name of the new VM
Clone the sourceVM vmdk to the newly created folder DestinationVM
vmkfstools -i /vmfs/volumes/datastore1/SourceVM/sourceVM.vmdk /vmfs/volumes/datastore1/DesintationVM/DesintationVM.vmdk

Once cloning is completed, proceed with the creation of the new VM using vSphere clientIn the option where you need to provision the harddisk for the new VM, choose 'Use an existing virtual disk'Browse and point the newly cloned vmdk file


VMware ESXi host shows disconnected

Symptom:

The host and VMs in a vcenter server appears to be disconnected.

Issue:

The host and its corresponding vms could appear as disconnected due to issue with any of the following services.

Virtual Machine is unresponsive at times

                 I don't know if this is a subject for a blog. But since the resolution of this issue appeared to be so silly and simple, thought of sharing that with you.
Issue
VM in an ESXi appears to be unresponsive at times. We are receiving ping response but not able to connect via RDP or console. The ultimate method of resolving this issue was a VM reset until I found the root cause.
Resolution
After many hours of troubleshooting, the issue got resolved.

Software iscsi adapter is missing in VMware ESXi ?

Issue

You will find software iscsi adapter missing in ESXi

Resolution
Log in to the vSphere ClientSelect the configuration tabSelect Storage AdaptersClick on the Add optionYou will be given the option to add Software iSCSI adapter

Failed to start a VMware virtual machine

Image
Issue

When you Power On a VM, you will be greeted with the below error message:
'Failed to start the virtual machine.
Module DevicePowerOn power on failed. Unable to create virtual SCSI device for scsi0:0, '/vmfs/volumes/...........vmdk' Failed to open disk scsi0:0: Unsupported or invalid disk type 7. Ensure that the disk has been imported.'

Cause

This will occur when you

How to install Vmware ESXi 5.x on Dell Optiplex 990

Image
ESXi is a hypervisor which is intended to work on a server platform. But that doesn't mean that we cannot install hypervisor on a workstation. Here, I'm trying to install VMware ESXi 5.5 on Dell Optiplex 990 machine.

I will not be discussing about the installation steps as I believe it is pretty much straight forward and most of you are very much expert in it. Will be just focusing on the tips that would help us install ESXi on a workstaion.

I've booted my DELL Optiplex 990 using the VMware ESXi 5.5 bootable disk and everything

VM appears as inaccessible in VMware vCenter

Image
Symptoms
VMs will be shown as inaccessible in the vCenter

Reason
A VM can become inaccessible due to any of the below reason:

VMware Virtual Machine Optimization tips

Focus : How can we optimize the performance of a virtual machine ?

Below are a few points you can consider during your virtual machine deployment time:

Failed to start the virtual machine

Image
When you try to Power On a virtual machine, you will receive the below error:

Failed to start the virtual machine. Module DevicePowerOn power on failed. Unable to create virtual SCSI device for scsi…