Hi guys,
When writing my VMX code in my OS, I was surprised to see how little resources were available online. Lucky for me, the Intel dev manuals are excellent as always. So I thought I would post my experience here so that other people can learn from it:
http://www.dumais.io/index.php?article= ... 413203fb98
I saw someone in another thread in this forum mention something about the VMX section in the wiki being only a stub and that we need a real example with multi-threading. I think project could serve as a good example. I could probably rewrite my article in the wiki to provide more information on the subject.
BTW: please don't hesitate to point out any misleading information I could have offered. I am still in the process of understand how the TLB works with EPT and VPID.
Here is an example for VMX
Re: Here is an example for VMX
Hi xmm15, your code is well written, I suppose you are updating vm exit handler now. For some vm exits you can restart guest at the same RIP (e.g. for exceptions as you did). For others like hlt, cpuid, etc you have to increase guest instruction pointer to point after the emulated instruction. For already implemented HLT just check IA32_VMX_MISC MSR 485h bit 6. when set indicates support for activity state 1 (HLT), if supported then set guest activity state to 1, then obtain instruction size by vmread vm exit instruction length (for 1-byte instruction like HLT the size should be 1), vmread guest RIP, add them together and vmwrite back to guest RIP, then vmresume. Also remember HLT vm exit occurs only when enabled HLT exiting as you already did. For vm exit controls I suggest to enable acknowledge interrupt on exit (bit 15) so you know interrupt number. For unrestricted guest mode it is recommended to enable vm entry controls bit 15 and then do also the same for vm exit controls and write EFER to VMCS fields for both guest and host (after checking that load/save efer and unrestricted guest features present). Before erasing guest CR0.PE, PG you should check whether hardware supports unrestricted guest mode (otherwise crash on CPUs like core 2 duo) You started guest in real mode from CS:IP 0000:0000, do you have some code at that address (usually there is real mode IDT)? Enable as little of features as possible in early stages of development (my first hypervisor intercepted only unconditional vm exits) and when everything is running OK, then enable them incrementally and check whether OK or some problem.
EPT and VPID are 2 different features, to invalidate them use INVEPT / INVVPID.
EPT is used to translate guest PA to hardware PA, e.g. you can redirect guest PA = 0 to point to some hardware PA containing code which is different from 0, otherwise you can construct identity memory map where guest PA is the same as hardware PA. You must execute INVEPT in vm exit handler at least after modifying EPT in vm exit handler in such way that you restrict access more than was allowed previously in EPT.
VPID is useful when running more VMs as you can tag every VM with unique VPID and invalidate only certain VPID instead of all and then have better performance. Now you are running only one VM (yet) so no worry (yet).
The hardest part of hypervisor development is to find bugs/mistakes/problems, don't be disgusted when stack at some unknown problem for a long time. Keep up your good work.
EPT and VPID are 2 different features, to invalidate them use INVEPT / INVVPID.
EPT is used to translate guest PA to hardware PA, e.g. you can redirect guest PA = 0 to point to some hardware PA containing code which is different from 0, otherwise you can construct identity memory map where guest PA is the same as hardware PA. You must execute INVEPT in vm exit handler at least after modifying EPT in vm exit handler in such way that you restrict access more than was allowed previously in EPT.
VPID is useful when running more VMs as you can tag every VM with unique VPID and invalidate only certain VPID instead of all and then have better performance. Now you are running only one VM (yet) so no worry (yet).
The hardest part of hypervisor development is to find bugs/mistakes/problems, don't be disgusted when stack at some unknown problem for a long time. Keep up your good work.
hypervisor-based solutions developer (Intel, AMD)
Re: Here is an example for VMX
Thanks for the nice words feryno. You seem to to know a few things about the TLB. Could confirm if I understand this correctly?
The way I read it, the TLB will contain entries that can be tagged with EPT (ep4ta as they call it),VPID and PCID.
So let's say I have normal processes running in my OS (not in a VM), they would be tagged with PCID but no EP4TA and VPID=0 right?
So,
invlpg: would invalidate only pages marked with the current PCID. But what if there is an entry with that PCID but also a EP4TA? (a bare--metal process with PCID=242 and a VM process with pcid=242)
invpcid: depending on the mode chosen, I could invalidate all entries marked with a specific PCID or all of them. And this, always regardless of whether there is a EP4TA or not?
invept: would only invalidate for a specific EP4TA.
Am I understanding this correctly?
So if a guest would execute invpcid, I should trap it and invoke invept instead then.
The way I read it, the TLB will contain entries that can be tagged with EPT (ep4ta as they call it),VPID and PCID.
So let's say I have normal processes running in my OS (not in a VM), they would be tagged with PCID but no EP4TA and VPID=0 right?
So,
invlpg: would invalidate only pages marked with the current PCID. But what if there is an entry with that PCID but also a EP4TA? (a bare--metal process with PCID=242 and a VM process with pcid=242)
invpcid: depending on the mode chosen, I could invalidate all entries marked with a specific PCID or all of them. And this, always regardless of whether there is a EP4TA or not?
invept: would only invalidate for a specific EP4TA.
Am I understanding this correctly?
So if a guest would execute invpcid, I should trap it and invoke invept instead then.