Memory Configuration

This section explains how to leverage 1GB Huge Pages for addressing your memory, which is necessary for efficient virtualization on systems with large amounts of RAM. This section is only applicable for VM Mode.

When to Use Huge Pages

Based on your system's RAM capacity:

• < 128 GB RAM: Not recommended. Proceed to Next Steps.
• 128-512 GB RAM: Optional but can provide performance benefits.
• > 512 GB RAM: Strongly recommended to avoid performance degradation.

Why Use Huge Pages?

Implementing 1GB Huge Pages provides several advantages:

• Reduced TLB Misses: Fewer page table entries means better CPU performance
• Improved Memory Access: Faster virtual-to-physical address translation
• Enhanced VM Performance: Critical for hypervisors managing large memory allocations
• Lower Overhead: Decreased page table management burden on the system

1G Huge Page Configuration

1. Check Huge Page Support

Run grep -i huge /proc/meminfo command to check the huge page support. You should see an output like this:

AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB

2. Allocate Huge Pages

Determine how much memory you want to reserve for the VMs. You need to reserve that much memory for huge pages plus a buffer. Note that this memory will be reserved and will be unusable on the host machine.

Calculation Formula:

Number of Huge Pages = (VM Memory in GB + Buffer) / 1GB

For example, if you want to dedicate 2000 GB to virtual machines with a 80 GB buffer, you would need 2080 huge pages.

Total System RAM	VM Allocation	Buffer	Huge Pages	Left for System
768 GB	640 (8x80) GB	60 GB	700	68 GB
1024 GB	800 (8x100) GB	80 GB	880	144 GB
1256 GB	1040 (8x130) GB	100 GB	1140	116 GB
1512 GB	1280 (8x160) GB	120 GB	1300	212 GB
2048 GB	1760 (8x220) GB	160 GB	1920	128 GB
4096 GB	3680 (8*460) GB	200 GB	3880	216 GB

Run the following command (it will take a while):

echo 2000 | sudo tee /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

To check that huge pages were allocated, run grep -i huge /proc/meminfo. You should see output like this:

AnonHugePages:     79872 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:    2080
HugePages_Free:     1580
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB
Hugetlb:        2181038080 kB

To deallocate, invoke:

echo 0 | sudo tee /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

3. Make Huge Pages Persistent

Edit the /etc/default/grub file and modify the line containing GRUB_CMDLINE_LINUX.

Add default_hugepagesz=1G hugepagesz=1G hugepages=<num> to the GRUB_CMDLINE_LINUX options. The <num> is the number of huge pages to allocate. For example:

GRUB_CMDLINE_LINUX="... default_hugepagesz=1G hugepagesz=1G hugepages=2080"

Update grub changes and reboot (or do it after the next section).

sudo update-grub

4. (Optional) Enable 5-level Paging

If you're planning to virtualize a large number of GPUs with a large amount of VRAM, like 8 x H100, you'll most likely need to enable 5-level paging. Otherwise, qemu won't be able to allocate large enough chunk of memory for passthrough. Check if your CPU supports 57-bit address space via:

$ lscpu | grep "Address sizes"

You should see an output like:

Address sizes:                        52 bits physical, 57 bits virtual

If so, add la57 option to the GRUB_CMDLINE_LINUX_DEFAULT variable in /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="la57"
GRUB_CMDLINE_LINUX="... default_hugepagesz=1G hugepagesz=1G hugepages=2080"

For this option to work you also need to use a kernel with la57 support. To check if your kernel supports la57 option run grep CONFIG_CMDLINE /boot/config-$(uname -r). You should see:

CONFIG_CMDLINE_BOOL=y
CONFIG_CMDLINE="la57"

If CONFIG_CMDLINE_BOOL is not set or CONFIG_CMDLINE does not include la57, then your kernel won’t activate la57, even if it’s supported and passed in GRUB. You'll need to build a kernel with those options enabled.

Update grub changes and reboot:

sudo update-grub
sudo reboot

5. Mount Huge Page Table

We need to mount the huge page table to the system, so that Hypervisor can use it.

The mount point location is not important, but we recommend using /mnt/hugepages-1G so that you know what this mount is used for.

sudo mkdir -p /mnt/hugepages-1G
sudo mount -t hugetlbfs -o pagesize=1G none /mnt/hugepages-1G

Check that mount point is present by running grep hugetlbfs /proc/mounts.

You should see something like:

hugetlbfs /dev/hugepages hugetlbfs rw,nosuid,nodev,relatime,pagesize=1024M 0 0
hugetlbfs /mnt/hugepages-1G hugetlbfs rw,relatime,pagesize=1024M 0 0

6. Persist the Mount on Reboot

Add to /etc/fstab:

none /mnt/hugepages-1G hugetlbfs pagesize=1G 0 0

7. Reboot and Verify that the Configuration Works

Reboot: sudo reboot

Verify that hugepages are allocated and mount point is present.

grep -i huge /proc/meminfo
grep hugetlbfs /proc/mounts

Troubleshooting

Common Issues

Not Enough Memory Available

If you see errors about not being able to allocate the requested number of huge pages:

1. Ensure you have enough free memory available (check with free -h)
2. Close unnecessary applications that may be consuming memory
3. Try allocating a smaller number of huge pages

Mount Point Issues

If the huge page filesystem fails to mount:

1. Verify the directory exists: ls -la /mnt/hugepages-1G
2. Check for mount errors: dmesg | grep huge
3. Ensure kernel support is enabled: grep CONFIG_HUGETLB_PAGE /boot/config-$(uname -r)

Next Steps

To test the node configuration and make your nodes rentable, proceed to the System Service Setup guide.