I am making a huge upgrade to my home lab and faced a trouble when dealing with second-hand NIC. I am. going to share my experience on NIC for this upgrade project.

1. Choosing NIC

First of all, we have to know what we need

• Support ESXI 8.0 Update 2
• Support True NAS 23.x.x
• 100 Gigabit Ethernet (as we have 20+ 4TB SSD on the NAS and high IO performance for applications we hosted in VMs)
• Cheap (My fault, I am not rich enough XD)

Finally, I chose 4 x Mellanox ConnectX-4 MCX455A ECAT and 1 x Mellanox ConnectX-4 MCX456A ECAT, then purchase second-hand from Taobao.

Those second NICs in Taobao are disused from data centre.  They are all in good condition, it just reached the time to replace new hardware in data centre to ensure stability of their servers. Usually we can get hardware 2-3 generation ago. For this time, the latest Mellanox NIC is ConnectX-7, we go ConnectX-4 for my home lab.

2. Additional Cables and Optical Transceiver

Apart from NIC, we also need cables and optical transceiver to make our 100G ESXI to TrueNAS connection work as expected.

• Cable – MPO to MPO 12 OM4
• Optical Transceiver – MMA1B00-C100D 100G MPO

3. Switch from InfiniBand to Ethernet in Mellanox NIC

All second-hand Mellanox ConnectX-4 NICs I got are in Infiniband mode, which is not supported by both ESXI host (supported inside VM) and TrueNAS. We have to switch all of them to Ethernet mode.

For this action, I would suggest you to plug all of them to an ubuntu server to change the configuration of NIC and update the firmware in next section. Although we can do it in ESXI, but trust me, doing this in Ubuntu server will save you a lot of time.

Install MST to the Ubuntu Server first

4. Upgrade Mellanox ConnectX-4 firmware

Old firmware is not supported by ESXI 8.0 Update 2, we have to upgrade to the latest firmware.

There is a big trap when I am upgrading the firmware. Even the model is MCX455A and I found the latest firmware in Mellanox site, I cannot flash official firmware to OEM NIC. We have to check the board_id for the NIC, then find the OEM partner and download OEM version firmware from their site.

For my case, all NIC I purchased are came from Fujitsu server. I have to find firmware in their site.

https://support.ts.fujitsu.com/IndexDownload.asp?lng=PT

Finally, I hope that I will have time to share more details on my home lab upgrade as it was really fun and challenging.