Memory Configuration
This section explains how to leverage 1GB Huge Pages for addressing your memory, which is necessary for efficient virtualization on systems with large amounts of RAM. This section is only applicable for VM Mode.
When to Use Huge Pages
Based on your system's RAM capacity:
- < 128 GB RAM: Not recommended. Proceed to Next Steps.
- 128-512 GB RAM: Optional but can provide performance benefits.
- > 512 GB RAM: Strongly recommended to avoid performance degradation.
Why Use Huge Pages?
Implementing 1GB Huge Pages provides several advantages:
- Reduced TLB Misses: Fewer page table entries means better CPU performance
- Improved Memory Access: Faster virtual-to-physical address translation
- Enhanced VM Performance: Critical for hypervisors managing large memory allocations
- Lower Overhead: Decreased page table management burden on the system
1G Huge Page Configuration
1. Check Huge Page Support
Run grep -i huge /proc/meminfo command to check the huge page support. You should see an output like this:
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
2. Allocate Huge Pages
Determine how much memory you want to reserve for the VMs. You need to reserve that much memory for huge pages plus a buffer. Note that this memory will be reserved and will be unusable on the host machine.
Calculation Formula:
Number of Huge Pages = (VM Memory in GB + Buffer) / 1GB
For example, if you want to dedicate 2000 GB to virtual machines with a 80 GB buffer, you would need 2080 huge pages.
| Total System RAM | VM Allocation | Buffer | Huge Pages | Left for System |
|---|---|---|---|---|
| 768 GB | 640 (8x80) GB | 60 GB | 700 | 68 GB |
| 1024 GB | 800 (8x100) GB | 80 GB | 880 | 144 GB |
| 1256 GB | 1040 (8x130) GB | 100 GB | 1140 | 116 GB |
| 1512 GB | 1280 (8x160) GB | 120 GB | 1300 | 212 GB |
| 2048 GB | 1760 (8x220) GB | 160 GB | 1920 | 128 GB |
| 4096 GB | 3680 (8*460) GB | 200 GB | 3880 | 216 GB |
Run the following command (it will take a while):
echo 2000 | sudo tee /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
To check that huge pages were allocated, run grep -i huge /proc/meminfo. You should see output like this:
AnonHugePages: 79872 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 2080
HugePages_Free: 1580
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB
Hugetlb: 2181038080 kB
To deallocate, invoke:
echo 0 | sudo tee /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
3. Make Huge Pages Persistent
Edit the /etc/default/grub file and modify the line containing GRUB_CMDLINE_LINUX.
Add default_hugepagesz=1G hugepagesz=1G hugepages=<num> to the GRUB_CMDLINE_LINUX options.
The <num> is the number of huge pages to allocate. For example:
GRUB_CMDLINE_LINUX="... default_hugepagesz=1G hugepagesz=1G hugepages=2080"
Update grub changes and reboot (or do it after the next section).
sudo update-grub
4. (Optional) Enable 5-level Paging
If you're planning to virtualize a large number of GPUs with a large amount of VRAM, like 8 x H100, you'll most likely need to enable 5-level paging. Otherwise, qemu won't be able to allocate large enough chunk of memory for passthrough. Check if your CPU supports 57-bit address space via:
$ lscpu | grep "Address sizes"
You should see an output like:
Address sizes: 52 bits physical, 57 bits virtual
If so, add la57 option to the GRUB_CMDLINE_LINUX_DEFAULT variable in /etc/default/grub:
GRUB_CMDLINE_LINUX_DEFAULT="la57"
GRUB_CMDLINE_LINUX="... default_hugepagesz=1G hugepagesz=1G hugepages=2080"
For this option to work you also need to use a kernel with la57 support. To check if your kernel supports
la57 option run grep CONFIG_CMDLINE /boot/config-$(uname -r). You should see:
CONFIG_CMDLINE_BOOL=y
CONFIG_CMDLINE="la57"
If CONFIG_CMDLINE_BOOL is not set or CONFIG_CMDLINE does not include la57, then your kernel won’t activate la57, even if it’s supported and passed in GRUB. You'll need to build a kernel with those options enabled.
Update grub changes and reboot:
sudo update-grub
sudo reboot
5. Mount Huge Page Table
We need to mount the huge page table to the system, so that Hypervisor can use it.
The mount point location is not important, but we recommend using /mnt/hugepages-1G so that you know
what this mount is used for.
sudo mkdir -p /mnt/hugepages-1G
sudo mount -t hugetlbfs -o pagesize=1G none /mnt/hugepages-1G
Check that mount point is present by running grep hugetlbfs /proc/mounts.
You should see something like:
hugetlbfs /dev/hugepages hugetlbfs rw,nosuid,nodev,relatime,pagesize=1024M 0 0
hugetlbfs /mnt/hugepages-1G hugetlbfs rw,relatime,pagesize=1024M 0 0
6. Persist the Mount on Reboot
Add to /etc/fstab:
none /mnt/hugepages-1G hugetlbfs pagesize=1G 0 0
7. Reboot and Verify that the Configuration Works
Reboot: sudo reboot
Verify that hugepages are allocated and mount point is present.
grep -i huge /proc/meminfo
grep hugetlbfs /proc/mounts
Troubleshooting
Common Issues
Not Enough Memory Available
If you see errors about not being able to allocate the requested number of huge pages:
- Ensure you have enough free memory available (check with
free -h) - Close unnecessary applications that may be consuming memory
- Try allocating a smaller number of huge pages
Mount Point Issues
If the huge page filesystem fails to mount:
- Verify the directory exists:
ls -la /mnt/hugepages-1G - Check for mount errors:
dmesg | grep huge - Ensure kernel support is enabled:
grep CONFIG_HUGETLB_PAGE /boot/config-$(uname -r)
VM Fails to Boot with "unable to map backing store for guest RAM"
VM memory is backed by the 1G huge page pool. Since v0.62.0, the server caps the RAM it offers on a node at the size of the reported huge page pool, so rentals no longer request more memory than the host can back. If you see this error on an older node, or the node offers less RAM than expected, check HugePages_Total and HugePages_Free in /proc/meminfo and allocate more huge pages as described above.
Next Steps
To test the node configuration and make your nodes rentable, proceed to the System Service Setup guide.