I am making a huge upgrade to my home lab and faced a trouble when dealing with second-hand NIC. I am. going to share my experience on NIC for this upgrade project.

1. Choosing NIC

First of all, we have to know what we need

• Support ESXI 8.0 Update 2
• Support True NAS 23.x.x
• 100 Gigabit Ethernet (as we have 20+ 4TB SSD on the NAS and high IO performance for applications we hosted in VMs)
• Cheap (My fault, I am not rich enough XD)

Finally, I chose 4 x Mellanox ConnectX-4 MCX455A ECAT and 1 x Mellanox ConnectX-4 MCX456A ECAT, then purchase second-hand from Taobao.

Those second NICs in Taobao are disused from data centre.  They are all in good condition, it just reached the time to replace new hardware in data centre to ensure stability of their servers. Usually we can get hardware 2-3 generation ago. For this time, the latest Mellanox NIC is ConnectX-7, we go ConnectX-4 for my home lab.

2. Additional Cables and Optical Transceiver

Apart from NIC, we also need cables and optical transceiver to make our 100G ESXI to TrueNAS connection work as expected.

• Cable – MPO to MPO 12 OM4
• Optical Transceiver – MMA1B00-C100D 100G MPO

3. Switch from InfiniBand to Ethernet in Mellanox NIC

All second-hand Mellanox ConnectX-4 NICs I got are in Infiniband mode, which is not supported by both ESXI host (supported inside VM) and TrueNAS. We have to switch all of them to Ethernet mode.

For this action, I would suggest you to plug all of them to an ubuntu server to change the configuration of NIC and update the firmware in next section. Although we can do it in ESXI, but trust me, doing this in Ubuntu server will save you a lot of time.

Install MST to the Ubuntu Server first

4. Upgrade Mellanox ConnectX-4 firmware

Old firmware is not supported by ESXI 8.0 Update 2, we have to upgrade to the latest firmware.

There is a big trap when I am upgrading the firmware. Even the model is MCX455A and I found the latest firmware in Mellanox site, I cannot flash official firmware to OEM NIC. We have to check the board_id for the NIC, then find the OEM partner and download OEM version firmware from their site.

For my case, all NIC I purchased are came from Fujitsu server. I have to find firmware in their site.

https://support.ts.fujitsu.com/IndexDownload.asp?lng=PT

Finally, I hope that I will have time to share more details on my home lab upgrade as it was really fun and challenging.

This tutorial is going to show you how to use crontab to backup your Mincraft server data to Dropbox automatically.

If you would like to see how to setup a minecraft server on Raspberry Pi 4, you can go to part 1 here.

Create a Dropbox account first.

Then create a Dropbox app from the link below

https://www.dropbox.com/developers/apps/create

You need to get the access token and update permission settings after you created the app.

Select “No expiration” for Access toke expiration and click “Generate” to generate the access token for you app.

Save your access token to somewhere, we need it later

Switch to permission tab

Then check “files.metadata.write” and “files.content.write”.

Click submit to save all changes.

Log on to Portainer we created in part 1.

Click Volume —> Console icon —> Connect

You can access your Minecraft server’s container now.

Install zip tool

Open text editor and create file “auto_minecraft_backup.sh”.

Copy these commands to the text editor and you have to replace the access token to yours.

Please also note that you need to keep “Bearer “ in Authorization

Press Control + x, then enter “Y” to save the script file.

You can test the script with the command below:

If you can see the backup in your Dropbox, you can create a cron job to schedule run this script.

Open crontab config file

Insert this line to config file

It means run the script every day at 5 a.m.

The link below can help you to setup your own schedule for cron job.

https://crontab.guru/

Press Control + x, then enter “Y” to save your setting.

Done!

This series of tutorials are going to show you how to build a low cost (~100USD ) Minecraft server on Raspberry pi 4 with web-based management tool and auto backup.

Before we get started, we have to prepare some hardwares:

Download Ubuntu for Raspberry 4

https://ubuntu.com/download/raspberry-pi

Download and install Etcher

https://www.balena.io/etcher/

Flash Ubuntu image to your micro SD card by Etcher

Insert the micro SD card to your Pi, then connect your lan cable and power supply to it

Find IP address of your Raspberry Pi

Method 1: Goto your router dashboard and find the device with name “ubuntu”

Method 2: Run nmap command to scan your network

You may need to download nmap first

Mac: https://nmap.org/book/inst-macosx.html

Windows: https://nmap.org/book/inst-windows.html

The ip range depends on your subnet setting for your router
e.g. If your router dashboard IP address is 192.168.0.1, you may need to change the IP scanning range to 192.168.0.0-255

Find the circled host and that is the IP address for your Raspberry Pi

Connect to your Raspberry Pi via ssh(Mac/Linux), Putty(Windows)

Connection information: 

For Windows users:

You can download Putty below:

https://www.putty.org/

For Mac and Linux users:

Open terminal and run the following command:

You will be asked to change your password, just change and remember your new password.

After you changed your password, you need to logon with your new password.

Your Raspberry is ready to use and we are going to install

• Docker
• Portainer
• Minecraft Server

Update package

Add required dependences

Add the security key

Add the docker repository for ARM64 + Ubuntu Linux.

Update the repository again.

Set higher priority for docker official repository, instead of using default Ubuntu Repository

Install Docker

Add current user to docker group

Now we have installed docker, then we can install Minecraft Server with single command now.

You can change the parameters to fit your actual requirement, here are some common parameters you may want to change:

After installed Minecraft server, we will install Portainer for managing the Minecraft server container with web-based UI.

Create volume for Portainer

Create Portainer container

You can access Portainer by entering the following URL to your broswer:

http://:9000

Choose Docker and connect to it

Congrat! You can manage your Raspberry Pi Minecraft server by Portainer now. By using Portainer, you can:

• Start/Stop the Minecraft server
• View the Minecraft server log
• Manage your dockers in web-based UI

Congrat! You can connect to your Minecraft server now.

I hope this tutorial can help you to setup your own Minecraft server on Raspberry Pi.

Part 2 “Auto backup your Minecraft Server”  here.

Python web scrapping is a important skill for data sciencists. Most websites are written in Javascript framework or making use of Javascript to generate their content. If we would like to scrap data from modern website, we have to know how to scrap data from Javascript-based websites.

There is no difficulty for the code. We just need to aware that it is a Javascript-based website and choose appropriate tools to scrap this efficiently.

In this article, I would use Beautiful Soup + Selenium to scrap a simple javascript website and hope this can help you.

Install required libraries

You also have to install web drive for parsing Javascript in selenium.

https://pypi.org/project/selenium/

Download the web driver you want and place that driver to your PATH (i.e. /usr/bin, /usr/local/bin for linux and MacOS, $PATH for Windows)

Import required libraries

If we use the “requests” library, we only get the source code from the website, but not the generated content.

Output from codes above

If we use selenium with Chrome web driver, we can get the Javascript generated content.

Output from codes above

Then we can pass the rendered content to Beautiful Soup object and scrap data we want

Output from codes above

Hope this guide can help you to scrap web content from Javascript-based websites.

If you would like to learn Beautiful Soup and Selenium, you can find their documentations below or leave a comment here.

Both of them are powerful libraries for web scrapping and browser automation projects.

Beautiful Soup: https://www.crummy.com/software/BeautifulSoup/bs4/doc/

Selenium: https://www.selenium.dev/documentation/en/

Having a home lab like me is a great idea if you require a lot of storage and need a virtual environment to build or test something frequently. I would like to share the concept and setup of my home lab in this article, but not the step boy step guide to replicate what I have done before.

If you want a step by step guide, you can check these links:

• FreeNAS 11.2 and ESXi 6.7 iSCSI Tutorial
  • https://johnkeen.tech/freenas-11-2-and-esxi-6-7-iscsi-tutorial/
• How to passthrough SATA drives directly on VMWare EXSI 6.5 as RDMs
  • https://gist.github.com/Hengjie/1520114890bebe8f805d337af4b3a064

ESXi is installed as the base OS and FreeNAS is installed on top of ESXi. I passed all hard disks to ZFS (File system used in FreeNAS) as RDM (Raw Drive Mapping) to let FreeNAS gets closer to the hard disk and gain the benefits of ZFS and virtualization.

The storage is managed by FreeNAS VM. We just need to connect to the ZFS volume via NFS (Network File System) in ESXi, then we can store all of our virtual machines to ZFS volume.

“ZFS combines the roles of RAID controller, Volume Manager, and file system, and since it’s all three in one, it wants direct access to your disks in order to work properly. The closer you can get ZFS to your storage hardware, the happier ZFS is, and the better it can do its job of keeping your data safe.”

https://www.ixsystems.com/blog/yes-you-can-virtualize-freenas/

All of my virtual machines are stored into the ZFS volume, except the FreeNAS VM. Each virtual machine has an isolated environment to serve its own purpose.

List of my VMs:

• Gitlab
• Minecraft server
• MySQL server
• Apache web server

I started to run this 3 years ago and this server saved me a lot of money for renting cloud server from VPS. I can deploy any applications I want and the performance is acceptable.

TOSHIBA DT01ACA100 for ESXi installation and FreeNAS VM.

Seagate BarraCuda 3TB * 4 passed to ZFS as RDM (Raw Device Mapping).

ESXi does not support all hardwares, please check the capability guild carefully.

https://www.vmware.com/resources/compatibility/search.php

• Allocate at lease 8GB to FreeNAS VM and reserve some CPU resources to make sure that your NFS volume works as expected
• Do not run this setup in production environment
• Don’t use anything cheap

https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.storage.doc/GUID-9E206B41-4B2D-48F0-85A3-B8715D78E846.html

https://www.ixsystems.com/community/threads/please-do-not-run-freenas-in-production-as-a-virtual-machine.12484/#post58364