VMware Transport Modes: Best practices and troubleshooting

 

Encountered issues restoring via “SAN” failed during the boot up, error symptom is “operating system not found”, tried again with “NBD”, it works.

http://www.symantec.com/business/support/index?page=content&id=tech183072

 

Issue

A VMware Backup Host can access Virtual Machine data from datastores using four different methods – SAN, LAN(NBD), HotAdd, NBDSSL. These methods are referred to as VMware Transport modes. This article talks about these transport modes, the best practices around them and troubleshooting tips for some commonly seen errors related to transport modes in NetBackup and Backup Exec.

Solution

For both Backup and Restore operations, NetBackup and Backup Exec allow choosing any of the four transport modes or a combination of these. If a combination of the transport modes is given, NetBackup and Backup Exec will try all of them one by one until gaining successful access to the data of the Virtual Machine.

Details on each of the transport modes

1. SAN:  The SAN transport mode requires the VMware Backup Host to reside on a physical machine with access to Fibre Channel or iSCSI SAN containing the virtual disks to be accessed. This is an efficient data path because no data needs to be transferred through the production ESX/ESXi host.

In this mode, vStorage APIs obtain information from the vCenter server or ESX/ESXi host about the layout of VMFS LUNs and, using this information, reads data directly from the SAN or iSCSI LUN where the VMDK resides.

Best practices around SAN:

• For using SAN, make sure that datastore LUNs are accessible to the VMware Backup Host.
• SAN transport is usually the best choice for backups when running on a physical VMware Backup Host. However, it is disabled inside virtual machines, so use HotAdd instead on a virtual VMware Backup Host.
• SAN transport is not always the best choice for restores. It offers the best performance on thick disks, but the worst performance on thin disks, because of the way vStorage APIs work. For thin disk restore, LAN(NBD) is faster.
• For SAN restores, disk size should be a multiple of the underlying VMFS block size, otherwise the write to the last fraction of a disk will fail. For example, if virtual disk has a 1MB block size and the datastore is 16.3MB large, the last 0.3MB will not get written. The workaround in this case would be to use NBD for restores of such Virtual Machines.
• When using SAN transport or hot-add mode on a Windows Server 2008/2008 R2 VMware Backup Host, make sure to set:
  • SAN policy to onlineAll
  • SAN disk as read-only, except during restores

2. LAN (NBD): In this mode, the ESX/ESXi host reads data from storage and sends it across a network to the VMware Backup Host. As its name implies, this transport mode is not LAN‐free, unlike SAN transport.

LAN transport offers the following advantages:

• The ESX/ESXi host can use any storage device, including local storage or NAS.
• The VMware Backup server could be a virtual machine, so you can use a resource pool and scheduling capabilities of VMware vSphere to minimize the performance impact of backup. For example, you can put the VMware Backup Host in a different resource pool than the production ESX/ESXi hosts, with lower priority for backup.
• If the ESX/ESXi host and VMware Backup Host are on a private network, you can use unencrypted data transfer,which is faster and consumes fewer resources than NBDSSL. If you need to protect sensitive information, you have the option of transferring virtual machine data in an encrypted form using NBDSSL.

Best Practices when using LAN:

• Since the data in this case is read by ESX/ESXi server from storage and then sent to VMware Backup Host, It is must to have network connectivity between ESX/ESXi server and VMware Backup Host. If the VMware Backup Host has connectivity to vCenter server but not the ESX/ESXi server- snapshots will succeed but vmdk read/write operations will fail.
• The VMware Backup Host will need the ability to connect to TCP port 902 on ESX/ESXi hosts while using NBD/NBDSSL for backup/restores.
• VMware uses Network File Copy (NFC) protocol to read VMDK using NBD transport mode. You need one NFC connection for each VMDK file being backed up. There is a limit on the number of NFC connections that can be made per ESX/vCenter server. These limits differ in different versions of vSphere - please refer to the NetBackup for VMware Admin Guide(linked below) for these limits. Backup/Restore operations using NBD might hang if this limit is reached.

3. HotAdd: When running VMware Backup Host on a Virtual Machine, vStorage APIs can take advantage of the SCSI Hot-add capability of the ESX/ESXi server to attach the VMDKs of a Virtual Machine being backed up to the VMware Backup Host. This is referred to as HotAdd transport mode.

Running the VMware Backup server on a virtual machine has two advantages: it is easy to move a virtual machine around and it can also back up local storage without using the LAN, although this incurs more overhead on the physical ESX/ESXi host than when using SAN transport mode.

Best practices when using HotAdd:

• HotAdd works only with virtual machines with SCSI disks and is not supported for backing up virtual machines with IDE disks.
• A single SCSI controller can have a maximum of 15 disks attached. To run multiple concurrent jobs totally more than 15 disks it is necessary to add more SCSI controllers to the HotAdd host.  The maximum number of 4 SCSI controllers can be added to a HotAdd host, so a total of 60 devices are supported at the maximum.
• HotAdd requires the VMware Backup Host to have access to datastores where the Virtual Machine being backed up resides. This essentially means:
  • ESX where the VMware backup host is running should have access to datastores where the Virtual Machine being backed up resides. 
  • Both the VMware backup host and Virtual Machine being backed up should be under the same datacenter.
• HotAdd cannot be used if the VMFS block size of the datastore containing the virtual machine folder for the target virtual machine does not match the VMFS block size of the datastore containing the VMware Backup Host virtual machine. For example, if you back up virtual disk on a datastore with 1MB blocks, the VMware Backup Host must also be on a datastore with 1MB blocks.
• Restores using HotAdd on a Windows Server 2008 proxy require setting the SAN policy toonlineAll
• If you are converting a physical machine to a virtual machine with the intention of using hottadd to back up the virtual machine, do not use IDE controllers for any disks that are used during the conversion process.
• The VMware Backup Host will need the ability to connect to TCP port 902 on ESX/ESXi hosts while using HotAdd for backup/restores.

4. NBDSSL:  NBDSSL is the same as NBD, except that NBDSSL uses SSL to encrypt all data passed over the TCP/IP connection.

Troubleshooting for some common transport mode related failures

Backups/Restores failing with status 6 or status 13 or status 11 with following indication in Activity monitor might indicate that there is some issue with transport modes:-

• ERR - Error opening the snapshot disks using given transport mode: Status 23 indicates that there was some problem in accessing the vmdk using given transport mode.
  Here are some tips on handling this kind of error:
  • If you are using NBD, make sure the VMware Backup Host has connectivity to ESX server hosting the virtual machine.
  • If you are using SAN, please make sure that the datastore LUNs are accessible to VMware Backup Host.
  • If you are using HotAdd, please make sure that your backup host is Virtual Machine and following conditions are satisfied:
    • The VM should not contain IDE disks.
    • Ensure that there are sufficient SCSI controllers attached on the Backup Host VM.
    • The Backup Host VM has access to datastores where VM being backed up resides.
    • The Backup Host VM and VM being backed up should be under the same datacenter.
    • If the previous backup failed, it might have left some disks of the backup VM attached to Backup Host. These disks need to be manually removed before attempting the next backup.
  • If a non-default port for vCenter is in use, then that port needs to be defined while adding vCenter credentials to NetBackup or Backup Exec.
  • If using NBD/NBDSSL/HotAdd, please make sure the VMware Backup Host is able to communicate to port 902 of ESX server hosting the VM.
• file read failed indicates that there might be problem in reading the VMDK using the given transport mode. 
• file write failed indicates that there might be some problem in writing to the VMDK using the given transport mode.
  • If using SAN for restores, please make sure datastore LUNs are accessible to the VMware Backup Host and in an online state.
  • If using HotAdd for restore, please make sure that SAN policy on the Backup Host is set to OnlineAll.
  • If using SAN for restore, make sure that the size of VMDK is multiple of datastore block size.  Otherwise, the write of the last block will fail.  In this case, a workaround would be to use NBD for restore.
  • Please make sure that the you assign necessary privileges to the user configured in NetBackup or Backup Exec to log on to vSphere.