Hide elapsed time of VM-exit from a guest

[tandasat]

The VMM does not virtualize the time stamp counter (TSC) or any other timers. This can lead timer interrupt (IDT d1: hal!HalpTimerClockInterrupt, on my test system) immediately after VM-enter when the VMM runs an VM-exit handler overly long time. While this is not an issue by itself, it could cause an infinite between a VM-exit and a timer interrupt handler under certain scenarios. A situation I have seen is as followings:

1. VM-exit occurs at address X
2. VM-exit runs overly long time to address an cause of the VM-exit
3. VM-entry
4. Timer interrupt occurs immediately (before an instruction at X is executed) changing state of the system such that VM-exit will occur at X
5. The timer interrupt ends and X is fetched for execution
6. Return to 1

A quick fix would be streamlining the long run VM-exit handler, but the VMM should not limit what a developer can do on VM-exit in that manner. A more correct way to address this issue is hiding an elapsed time of VM-exit handler from a guest and protect guest context from triggering timer interrupt.

[tandasat]

Looked into details of timer. A fix I thought, which was disabling timer at the entry point of VM-exit did not seem to be straightfoward at all. I will find some time to investigate more, but I am going to treat this issue as lower priority.

Here is my findings on the timer interrupt. An interrupt handler for the timer is 0xd1 on my tested system. This was identified by looking at IP when this issue was seen.

>!idt
...
d1: fffff801085c1bf8 hal!HalpTimerClockInterrupt (KINTERRUPT fffff801084574b0)

I initially thought it was triggered by Local APIC, but below output shows interrupt vector for the timer is 0xff and the Initial Count and Current Count Registers are both 0 indicating the time was disabled accodring to the "APIC Timer" section in the Intel SDM.

kd> !apic
Apic @ ffd0d000  ID:0 (50015)  LogDesc:01000000  DestFmt:ffffffff  TPR F0
TimeCnt: 00000000clk  SpurVec:df  FaultVec:e2  error:0
Ipi Cmd: 01000000`0004002f  Vec:2F  FixedDel    Dest=Self      edg high
Timer..: 00000000`000300ff  Vec:FF  FixedDel    Dest=Self      edg high      m
Linti0.: 00000000`000100d8  Vec:D8  FixedDel    Dest=Self      edg high      m
Linti1.: 00000000`00000400  Vec:00  NMI         Dest=Self      edg high
TMR: 77, 87, 97, B0
IRR: 66, 76, D1
ISR: D1

kd> !dd 0xfee00380
#fee00380 00000000 00000000 00000000 00000000
#fee00390 00000000 00000000 00000000 00000000

Later, I found that the interrupt 0xd1 was registered by IO APIC. I guess this is because timer is managed by HPET, which is a dedicated device located outside a processor.

kd> !ioapic
Controller at 0xffffffffffd004c0 I/O APIC at VA 0xffffffffffd0e000
IoApic @ FEC00000  ID:1 (11)  Arb:1000000
...
Inti02.: 01000000`000008d1  Vec:D1  FixedDel  Lg:01000000      edg high
...

It may be still possible to disable the HPET timer, but I would need to understand HPET well to do it. Moreover, time keeping is not only done by HPET. ACPI Power Management Timer (PM Clock), Local APIC timer or any other timers may be used, and I need to investigate all those posibilities to decide what to cover in this project.

Those are resources could be useful for further investigation.

• Acquiring high-resolution time stamps
  • https://msdn.microsoft.com/en-us/library/windows/desktop/dn553408%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396
• Intel 82093AA I/O Advanced Programmable Interrupt Controller (I/O APIC) Datasheet
  • http://www.intel.com/design/chipsets/datashts/290566.htm
• IA-PC HPET (High Precision Event Timers) Specification
  • http://www.intel.com/content/dam/www/public/us/en/documents/technical-specifications/software-developers-hpet-spec-1-0a.pdf
• Entering the kernel without a driver and getting interrupt information from APIC
  • http://www.codeproject.com/Articles/11363/Entering-the-kernel-without-a-driver-and-getting-i

[koemeet]

I was trying aswell to bypass timing attacks made by apllications running on the Guest OS. My first thought was changing rdtsc in an vmexit handler, but it doesn't seem to be called and the guest os isn't influenced by any changes made in this exit handler.

Did you made some progress on bypassing timing checks in the mean time?

[tandasat]

This is off topic for this thread, but it should be relatively straightforward to modify results of RDTSC. Just change a guest's registers used by RDTSC in VMM. Also, The "Time-Stamp Counter Offset and Multiplier" section might be interesting for your purpose.

[frostiest]

hi @tandasat,
I posted about this issue here tandasat/DdiMon#38 and the more I research the more I get confused as to why there's so many ways to change the TSC. For Intel there's

IA32_TSC_ADJUST
IA32_TSC_OFFSET
IA32_TIME_STAMP_COUNTER
TSC scaling

And amd has an extra one for them
TscRateMsr

Modifying IA32_TIME_STAMP_COUNTER directly accomplishes my end goal which is hiding cpu cycles from rdtsc but the issue is after awhile process windows start blacking out, or refusing to start and the computer is unusable. This happens even if I filter by process

_Use_decl_annotations_ static void VmmpHandleCpuid( GuestContext *guest_context) {
  unsigned int cpu_info[4] = {};
  const auto function_id = static_cast<int>(guest_context->gp_regs->ax);
  const auto sub_function_id = static_cast<int>(guest_context->gp_regs->cx);

  UCHAR Mode = VmmpGetGuestCpl();
  LARGE_INTEGER StampCounter = {};
  int DoChange = 0;

  if (Mode == 3) //usermode
  {
    if (PsGetCurrentProcessId()==2222) {
      StampCounter.QuadPart = UtilReadMsr64(Msr::IA32_TIME_STAMP_COUNTER);
      DoChange = 1;
    }
  }
_cpuidex(reinterpret_cast<int *>(cpu_info), function_id, sub_function_id);
  guest_context->gp_regs->ax = cpu_info[0];
  guest_context->gp_regs->bx = cpu_info[1];
  guest_context->gp_regs->cx = cpu_info[2];
  guest_context->gp_regs->dx = cpu_info[3];

  if (DoChange) 
    UtilWriteMsr64(Msr::IA32_TIME_STAMP_COUNTER, StampCounter.QuadPart - 600);
  

  VmmpAdjustGuestInstructionPointer(guest_context);
}

Or if I filter by driver. Either way eventually(within 5 minutes) the system starts going haywire and isn't usable anymore. Do you have any idea how I can fix this?

[frostiest]

Found this description

17.13.3 Time-Stamp Counter Adjustment
...
Software can modify the value of the time-stamp counter (TSC) of a logical processor by using the WRMSR instruction
to write to the IA32_TIME_STAMP_COUNTER MSR (address 10H). Because such a write applies only to that
logical processor, software seeking to synchronize the TSC values of multiple logical processors must perform these
writes on each logical processor. It may be difficult for software to do this in a way than ensures that all logical
processors will have the same value for the TSC at a given point in time.
The synchronization of TSC adjustment can be simplified by using the 64-bit IA32_TSC_ADJUST MSR ( address
3BH ). Like the IA32_TIME_STAMP_COUNTER MSR, the IA32_TSC_ADJUST MSR is maintained separately for each
logical processor.

and reada comment on stackoverflow that a number of things in your system expect the IA32_TIME_STAMP_COUNTER to be constantly increasing so haven't found a way to keep it stable by decreasing yet.