fix(physicalmem): don't claim low addresses

[zyuiop]

Context

While working on patches for AMD SEV-SNP, we use the DeviceMapper with an offset early at boot to allocate a (plaintext) page to communicate with the hypervisor (the GHCB).

We observed that randomly an application compiled with hermit would crash or enter a spin-loop sometimes around the introduction of the IDT. This bug was random in that recompilation with some changes at unrelated places in the code could eliminate the bug. This made debugging "fun" (as in: adding a print makes the bug disappear fun).

This clearly screams of "memory corruption" somewhere. But where, actually? I don't really know, but this patch is an attempt at fixing one possible source.

What this does, and why?

This fix is actually two fixes in one.

• First, map_frame_range (in physicalmem.rs) will, when an offset is used, map all pages also at a fixed offset. This will obviously require spawning new page tables, and it turns out that the physical free list may at this point contain 0x0 as a valid frame, which may get picked up. I don't know if it's bad-bad, but since the page table is identity mapped, this means that accessing the page table entry requires dereferencing virtual address 0, aka the null pointer, which certainly cannot be good.

• Second, it turns out that the SMP boot code has a fixed address of 0x8000. This address is only the 8th frame in the free list, so you can be sure it is going to be used, either by the offset mapping, or later on when mapping the heap. However, when the SMP boot code is loaded, it will overwrite that page table entry, and corrupt the page table. I guess this bug can happen if you use a heap intensive application, it will reach a point where it accesses a part of the heap which is corrupt and crash. Maybe. I'm not certain.

To avoid these two problems, we propose a quickfix: just ignore the first 2 MB of physical addresses.

What else?

The detect_from_fdt method looks a bit "dangerous" to me, in that it detects memory and immediately uses it to map it (sometimes). In most cases, no mapping is created, because UEFI already identity mapped everything, but in the cases where it does use it immediately, it may use pages that should be reserved and would have left the free list later in the procedure. The fact that pages are mapped immediately may cause issues. But not sure.

UPDATE

I now believe that the memory corruption was more likely happening at the end of the stack, which happened to be the page table - see #2462. However, I am still not sure it's a good idea to have 0x0 used as a page table, and the possible corruption with SMP is still there.

[github-actions]

Benchmark Results

Details

Benchmark	Current: 73a2aff	Previous: 2e23902	Performance Ratio
startup_benchmark Build Time	94.53 s	80.34 s	1.18 ❗
startup_benchmark File Size	0.78 MB	0.80 MB	0.98 ❗
Startup Time - 1 core	0.80 s (±0.01 s)	0.75 s (±0.02 s)	1.07 ❗
Startup Time - 2 cores	0.81 s (±0.02 s)	0.74 s (±0.02 s)	1.10 ❗
Startup Time - 4 cores	0.81 s (±0.01 s)	0.74 s (±0.02 s)	1.08 ❗
multithreaded_benchmark Build Time	92.18 s	82.11 s	1.12 ❗
multithreaded_benchmark File Size	0.88 MB	0.86 MB	1.03 ❗
Multithreaded Pi Efficiency - 2 Threads	65.26 % (±5.87 %)	85.89 % (±6.61 %)	0.76 ❗
Multithreaded Pi Efficiency - 4 Threads	40.36 % (±3.31 %)	43.43 % (±2.56 %)	0.93
Multithreaded Pi Efficiency - 8 Threads	19.97 % (±1.23 %)	25.76 % (±1.53 %)	0.78 ❗
micro_benchmarks Build Time	231.98 s	80.40 s	2.89 ❗
micro_benchmarks File Size	0.88 MB	0.86 MB	1.02 ❗
Scheduling time - 1 thread	176.45 ticks (±30.39 ticks)	62.65 ticks (±4.06 ticks)	2.82 ❗
Scheduling time - 2 threads	105.66 ticks (±24.99 ticks)	34.08 ticks (±4.10 ticks)	3.10 ❗
Micro - Time for syscall (getpid)	9.68 ticks (±4.24 ticks)	3.45 ticks (±0.58 ticks)	2.81 ❗
Memcpy speed - (built_in) block size 4096	55773.28 MByte/s (±39910.34 MByte/s)	82448.38 MByte/s (±56997.13 MByte/s)	0.68
Memcpy speed - (built_in) block size 1048576	15267.33 MByte/s (±13389.69 MByte/s)	30585.98 MByte/s (±24707.84 MByte/s)	0.50
Memcpy speed - (built_in) block size 16777216	12035.43 MByte/s (±9942.29 MByte/s)	26340.06 MByte/s (±21720.96 MByte/s)	0.46
Memset speed - (built_in) block size 4096	56006.89 MByte/s (±40072.80 MByte/s)	82292.76 MByte/s (±56891.50 MByte/s)	0.68
Memset speed - (built_in) block size 1048576	15777.75 MByte/s (±13719.32 MByte/s)	31323.85 MByte/s (±25145.86 MByte/s)	0.50
Memset speed - (built_in) block size 16777216	12392.35 MByte/s (±10163.12 MByte/s)	27104.68 MByte/s (±22209.94 MByte/s)	0.46
Memcpy speed - (rust) block size 4096	53788.00 MByte/s (±38483.16 MByte/s)	74097.96 MByte/s (±51811.44 MByte/s)	0.73
Memcpy speed - (rust) block size 1048576	15225.94 MByte/s (±12955.23 MByte/s)	30361.60 MByte/s (±24602.37 MByte/s)	0.50
Memcpy speed - (rust) block size 16777216	12719.00 MByte/s (±10627.24 MByte/s)	27625.34 MByte/s (±22806.88 MByte/s)	0.46
Memset speed - (rust) block size 4096	53946.25 MByte/s (±38598.64 MByte/s)	74373.47 MByte/s (±51976.48 MByte/s)	0.73
Memset speed - (rust) block size 1048576	15590.32 MByte/s (±13163.82 MByte/s)	31110.89 MByte/s (±25033.24 MByte/s)	0.50
Memset speed - (rust) block size 16777216	12999.19 MByte/s (±10764.99 MByte/s)	28386.93 MByte/s (±23265.03 MByte/s)	0.46
alloc_benchmarks Build Time	211.15 s	74.76 s	2.82 ❗
alloc_benchmarks File Size	0.86 MB	0.87 MB	0.98 ❗
Allocations - Allocation success	91.35 %	91.31 %	1.00 ❗
Allocations - Deallocation success	100.00 %	100.00 %	1
Allocations - Pre-fail Allocations	61.55 %	61.44 %	1.00 ❗
Allocations - Average Allocation time	8763.93 Ticks (±2452.81 Ticks)	5860.58 Ticks (±98.43 Ticks)	1.50 ❗
Allocations - Average Allocation time (no fail)	11008.97 Ticks (±2399.69 Ticks)	6554.81 Ticks (±92.86 Ticks)	1.68 ❗
Allocations - Average Deallocation time	3063.76 Ticks (±808.43 Ticks)	1805.01 Ticks (±250.35 Ticks)	1.70 ❗
mutex_benchmark Build Time	215.35 s	79.82 s	2.70 ❗
mutex_benchmark File Size	0.88 MB	0.86 MB	1.02 ❗
Mutex Stress Test Average Time per Iteration - 1 Threads	36.30 ns (±6.04 ns)	12.10 ns (±0.41 ns)	3.00 ❗
Mutex Stress Test Average Time per Iteration - 2 Threads	31.48 ns (±9.04 ns)	40.26 ns (±1.68 ns)	0.78

This comment was automatically generated by workflow using github-action-benchmark.

[mkroening]

Good find! :)

I think it would be preferable to only exclude the null pointer page and allocate memory at the required memory for the boot code. Not sure if that is easily possible with the current boot flow, though. What do you think?

[mkroening]

Thanks and sorry for the wait! :)

I have opened an issue for my concern:

• #2661

We can take care of this later, though. I have rebased and inlined the constant. :)