VSAN Part 15 – Multicast Requirement for networking – Misconfiguration detected
This is an issue which has caught a number of customers out during the Virtual SAN beta, so will probably catch some folks out when the product goes live too. One of the requirements for Virtual SAN (VSAN) is to allow multicast traffic on the VSAN network between the ESXi host participating in the VSAN Cluster. However, as per our engineering lead on VSAN, multicast is only used for relatively infrequent metadata operations. For example, object creation, change in object status after a failure and publication of statistics such as a significant change of free disk space (the publication of statistics is throttled so that only significant changes will cause an update, so these are also very infrequent events).
So how does this lack of multicast support on the network manifest itself? Well, what you will see after enabling VSAN on the cluster is that the network status is shown in a misconfigured state (Misconfiguration detected), even though you can ping/vmkping all the VSAN interfaces on all the hosts:
How do you resolve it? Well, a number of our VSAN beta customers discussed some options on the community, and these were the recommendations:
- Option 1 – Disable IGMP Snooping. Now this will allow all multicast traffic through, but if the only traffic is VSAN, then this should be a negligible amount of traffic and should be safe to use.
- Option 2 – Configure IGMP snooping querier. If there is other multicast traffic and you are concerned that disabling IGMP snooping might open the network up to a flood of multicast traffic, then this is a preferred option. Cisco detail how to do this here.
Customers who ran into this situation stated that both methods worked for them.
Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @CormacJHogan
Cormac,
I have a 4 node vSAN cluster. When I look at the multicast information on the switches I see all 4 nodes with the agent multicast address, but I only see 2 nodes with the master multicast address. Is this normal behavior?
Thanks
I just checked my 4 node cluster Jim.
All 4 report Agent Group Multicast Address: 224.2.3.4 & Master Group Multicast Address: 224.1.2.3 using “esxcli vsan network list”.
When I run “esxcli vsan network list” on each host, they all show the appropriate information.
~ # esxcli vsan network list
Interface
VmkNic Name: vmk2
IP Protocol: IPv4
Interface UUID: dc683c53-5331-e374-5b18-c81f66d9cc23
Agent Group Multicast Address: 224.2.3.4
Agent Group Multicast Port: 23451
Master Group Multicast Address: 224.1.2.3
Master Group Multicast Port: 12345
Multicast TTL: 5
~ #
but when I look on my Cisco 5K switches I’m seeing all 4 with the agent address, but only 2 with the master address.
2032 224.1.2.3 v2 D Po214 Po10
2032 224.2.3.4 v2 D Po214 Po10 Po8 Po9
I’m seeing a similar result in another vSAN cluster and I just want to confirm my results are expected.
Spoke to one of the engineers Jim. This seems normal as only master and backup actively use these addresses.
Good news! Multicast can be tricky sometimes, so thank you for being able to confirm this.