IO Filters marked offline in a vSAN cluster

13.11.19 - Reading time: 2 minutes
Symptoms

IO Filters marked offline in a vSAN cluster. One of the reasons can be the missing Self Signed SSL certificate in /etc/vmware/ssl/castore.pem Before making any changes in the system , please validate if the customer is using the third party certificates Configuring Custom Certificates on ESXi hosts to authenticate vSAN hosts https://kb.vmware.com/s/article/56441

Cause
  • The vSAN GUI ( vCenter > Configure > Storage Providers ) may show all the hosts IOfilter storage providers as “offline”
  • The Re-scan or re-synchronize of the VASA providers does not make any change to the state of IOfilter.
  • The upgrade of ESXi does not resolve the issue.
  • Reboot of effected Hosts does not resolve the issue.
  • Remove and re-import Host to vCenter does not resolve the issue.

You may find the below instances showing that SSL certs are not being verified for the hosts in the IOfiltervpd logs (ESXi : /var/log/iofiltervpd.log )

iofiltervpd[2099744]: IOFVPSSL_VerifySSLCertificate:239:Client certificate can’t be verified

The newly added Host may show the IOfilters providers as “online” You may find that the certificates located at /etc/vmware/ssl/castore.pem, have missing the “Self Signed Certificate” for the host with IOfilters offline Following is Self signed certificate which must be part of the /etc/vmware/ssl/castore.pem file

Resolution

You may follow the below steps :

  • Put the host in Maintenance mode with Ensure Accessibility one at a time
  • Take backup of the current of cert file /etc/vmware/ssl/castore.pem
  • Copy the file /etc/vmware/ssl/castore.pem to the Hosts from the working host.
  • Run command to replace the older file with newer one : cp /tmp/castore.pem /etc/vmware/ssl/castore.pem
  • Reboot the hosts one at a time.

The providers should show as “online” in vCenter > Configure > Storage Providers

Kudos to Shrikant (https://virtuallyvtrue.com )

About

This is the place to be for Virtualization and Datacenter stuff :-)
#sharingiscaring