25GbE networking issues

TL;DR Update your firmware and use name brand parts!

Recently I’ve been involved in an 8 node VxRail deployment that involved a Compellent 5020 and a couple SC420 shelves.  Both sets of equipment are using 25GbE networking for the first time in this .EDU data center.  We’ve run into a couple issues and I’ll outline the fixes here.

First, the Compellent 5020 wouldn’t even *see* the HBA cards.  It was shipped from the factory with cards that the controllers weren’t even aware of.  The root cause was out of date firmware.  The reason it was old was the firmware was released the day after the physical install!  Co Pilot dropped the ball on this one and we spent about 12 hours chasing license files thinking that was the fix.  At that time, 23/24 August 2018, you had to opt into the firmware release and gain Co Pilot approval by serial number.  Once we shook enough trees, we find out Storage Center 7.3.2 was required to unlock the 25 or 100 GbE networking. This also requires Storage Manager 18R1 or newer. Bottom line, update your firmware!  Here’s the relevant release notes. 680-021-021 Storage Center 7.3 Release Notes (003)

Next up, the VxRail!  The architect involved in the original Bill of Materials was pushed to use customer supplied SFP28 modules instead of the Dell branded parts.  The customer was adamant to use their supplier as they had been using the off brand 10GbE SFP+ for years and they would save (in theory) several hundred per SFP.  We relented,  and on deployment day not only were the customer supplied SFP28 modules not recognized, the Corning fiber jumpers wouldn’t even seat properly.  The jumpers did seat perfectly in the Dell branded SFP28s that we ordered as backups. Unfortunately, due to shortages we’re now pushing the install a week or more.  Additionally, we ran into an issue with the ports not lighting up.  We found the answer in this Tech Note. Auto negotiation may or may not be enabled depending on switch type and media type.

25GbE_Auto-Negotiation_TechNote

 

Stay frosty!