There are plenty of resources on setting up a lab for reverse engineering malware but most of them fail to address the issue of hardware. While it is possible to reverse engineer on a single machine, if you are confined to one machine, you tend to find out fairly quickly that you are going to run out of resources whether it is disk space, RAM, or processor cores. This is especially true when you start running multiple virtual machines simultaneously. In my lab, I typically have several virtual machines running at once such as OpenVAS, Elastic Stack, Kali Linux, a couple of Domain Controllers, Metasploitable 3, etc. Running all of these or even a couple of them at the same time can eat up a lot of the resources on a single machine.
The solution to the hardware problem is to get a dedicated server to host the virtual machines. For my lab, I have 3 x Dell PowerEdge R610 rack servers. Each of these servers contain 2 quad core processors and can be purchased, stripped down, for around $150 on ebay. These machines usually come with minimal amounts of RAM and hard drive space. In my lab, I also needed to replace the RAID controller as the SAS RAID controller that came with these servers was only capable of supporting RAID 1 and 0. In each one of my machines, I was able to upgrade the RAID controller to a Dell PowerEdge PERC 6/i RAID controller for around $15 a piece. The RAM was a little more expensive as it needed to be registered server memory such as PC3-10600R. The most expensive part of the servers was the hard drive space. The important thing to remember when building up these servers is the fact that each virtual machine will be accessing the same hard drive at the same time so speed is important. Each server comes with 6 hotswappable harddrive slots. In my configuration, I setup two solid state hard drives in a RAID 1 configuration for the server operating system with the remaining 4 slots being used for 500 gig solid state hybrid drives in a RAID 5 configuration.
The operating system used on the servers really depends on which Hypervisor software you choose to host your virtual machines. The good news is that the host operating system from the two major hypervisors are both free. If you use VMWare to host your virtual machines then you will want to install VMWare's ESXi, but if you use Windows Hyper-V then you will want to install Windows Hyper-V Server 2016.
For my lab, I installed the Hyper-V server because most of the virtual machines I had built were already setup for Hyper-V on my desktop. It was just a matter of copying the virtual machines over and importing them into the new Hyper-V server. The initial reason I went with Hyper-V over VMWare was due to being able to run virtual machines without having to be logged in which was something VMWare wasn't able to do out of the box. Once the Hyper-V operation system is installed, it's just a matter of right mouse clicking on the Hyper-V Manager and connecting to the server through the Hyper-V manager of any host computer on the same network.