MTRR and paging

[smolloy]

Hi,
My kernel starts by enabling paging and setting up two tables both pointed at the first 4MB of physical RAM. The two tables are

• An identity mapping
• The 4MB starting at 0xC0000000.

Once I've got everything running, I jump into the C kernel. At this point, everything is located above 0xC0000000, and so I zero the page table that does the identity mapping, and everything works just fine.

After a little thought, I was quite surprised that this actually worked, since I continue writing to video RAM (starting at 0xB8000) without any conversion to take account of the fact that I should be writing to 0xC00B8000 due to the way I set up the paging.

A little googling and browsing of the Intel manuals (volume 3 chapter 11) leads me to believe that this is due to something called the MTRR, and that this acts as a sort of page table that grabs any memory within a certain range and sends it off to (for example) some memory-mapped hardware (such as the video hardware). So poking values into 0xB8000 doesn't cause a page fault since the page tables are never called on to interpret the address -- instead the memory access is caught by the MTRR, which then does the right thing.

Have I understood this correctly?

If so, can I expect this behaviour to exist on real hardware (in particular, the BIOSes found in consumer-level x86 desktops), or do I need to ensure it by setting the appropriate MTRR details myself?

[xenos]

The MTRRs don't know anything about the mapping from virtual to physical addresses. They only control which physical memory regions may be cached and which may not.

The behavior you describe is rather caused by a different cache, know as the TLB. The purpose of the TLB is to cache page tables. When you clear some page table by filling it with zeros, the copy in the TLB remains unchanged and will still be used. In order to tell the CPU that some page table has changed and the copy in the TLB has become invalid, you need to issue the "invlpg" instruction. Have a look at the sections on the TLB and the invlpg instruction in the Intel / AMD manuals on this topic.

[smolloy]

Thanks for the great answer, as well as the super-fast response.

I did a few tests, and get the expected response from invlpg, so I can confirm that you were spot-on with your explanation. Now I'm suffering because I forgot that something else even more important than the video RAM is located in the first 4MB -- the page directory itself!! *smacks-forehead*

I think I can figure out how to solve that on my own, so thanks again for your time.

[iansjack]

Just a small addition. The TLBs can also be cleared by writing a new value to cr3. (Not 100% true, but good enough for the moment.) So if you are switching between different page tables, as you will when implementing tasking, you don't need to do an explcit INVLPG for each page in your page table; just load the address of the new table into cr3.

[Griwes]

Thanks for the great answer, as well as the super-fast response.

I did a few tests, and get the expected response from invlpg, so I can confirm that you were spot-on with your explanation. Now I'm suffering because I forgot that something else even more important than the video RAM is located in the first 4MB -- the page directory itself!! *smacks-forehead*

I think I can figure out how to solve that on my own, so thanks again for your time.

Paging structures use physical, not virtual, addresses. And, as your first 4 MiB are also mapped at 0xC0000000, you can still easily edit it (just by adding 0xC0000000 to its physical address).

[Owen]

One note: INVLPG of large pages normally will clear the entirety of the large page from the TLB. There is one exception: When the page covers the bottom 1MB of RAM, the CPU will break it into multiple smaller pages in the TLB. The size it will break it into is unspecified, so I would assume 4kB pages for safety.

[smolloy]

Now I'm suffering because I forgot that something else even more important than the video RAM is located in the first 4MB -- the page directory itself!! *smacks-forehead*

OK, it took a while, but I figured out the problem. Firstly, it had nothing to do with the location of the page directory. *sigh*

The symptoms were that the CPU would triple-fault almost immediately after the code hits the infinite loop after main() returns (jmp $). Note that this means that breakpoints were almost useless for tracing the error, since it's not really possible to breakpoint a one-line infinite loop.

Investigations showed that the fault only happened after I invlpg'ed from 0x0 to 0x400000. If, however, I only invlpg'ed from 0x140000 to 0x400000, then it didn't triple fault any more (although I imagine that it was a time bomb waiting to go off next time I reloaded CR3).

The big clue came when I realised that keeping the interrupts off (commenting out sti) stopped the triple fault from happening. This hinted that the timer interrupt was responsible, and led me on a bit of a chase around the IDT where everything seemed fine. After checking this, I then checked the GDT, which showed the cause of the error. I had linked the gdt_ptr to a low address that was no longer valid after the invlpg.

Rewriting a new GDT fixed everything, and I can now continue on with the memory management stuff I wanted to do.

I'm still a little confused as to why the fault didn't hit as soon as I invlpg'ed the page containing the gdt_ptr, since I thought that all memory accesses went through the GDT. Perhaps the GDT is only used for far jumps, and it was one of these that was triggered by the timer interrupt?

I think I can figure out how to solve that on my own, so thanks again for your time.

Although I was sorely tempted to post here to get some help, I am very pleased that I figured this one out on my own. As well as helping to improve my knowledge of the bochs debugger, it also helps me to understand my own code, as well as giving me a very smug sense of satisfaction.

[xenos]

I'm still a little confused as to why the fault didn't hit as soon as I invlpg'ed the page containing the gdt_ptr, since I thought that all memory accesses went through the GDT. Perhaps the GDT is only used for far jumps, and it was one of these that was triggered by the timer interrupt?

Actually the GDT is not accessed on every memory access, but only when you do something like a far jump or something similar that involves looking up a segment selector from the GDT. A simple example is reloading a segment register. Whenever you reload, e.g., DS with a new segment selector, a GDT (or LDT) lookup is performed. The information from the GDT such as the segment offset and limit is then stored in a "hidden" portion of DS which is not directly accessible. Whenever you access memory through DS, this stored information is used for offset calculation and limit checking, so there is no need to access the GDT once again. That's also the reason why you need to reload segment registers after you have changed the GDT, because only then the new values will be stored in the hidden portion of the segment registers and used on subsequent memory accesses.

[smolloy]

It struck me today that the only reason I started this thread was because I thought I had figured out the answer to a puzzle -- a puzzle that wasn't causing me any problems at that moment! -- and I wanted confirmation that I was right. As it turns out, I was completely wrong, but even more interesting (for me anyway) is to think about what would have happened had I not tried to track down the true answer. Sooner or later my code would have resulted in the page caches being updated, which would have immediately crashed the OS due to the corrupted GDT, but who knows when that would have happened!

I could have kept happily developing until, all of a sudden, *bang*! And then what? It would have taken far far longer to debug and fix due to all the extra code (isn't the automatic reaction to blame the last compile?).

I guess the point is -- make sure you understand *everything*!! Don't let any question go unanswered! (Although it's probably a good idea to try to answer it yourself before coming here )

Thanks to everyone for your help, especially XenOS.

[Solar]

I guess the point is -- make sure you understand *everything*!! Don't let any question go unanswered!

That, indeed, is a very valuable insight, one that distinguishes the good developer from the average. It is also the core reason why some developers react so poorly to trial & error, "it works for me" approaches to bug fixing: You didn't fix the bug if you didn't understand its cause.