MdeModulePkg/PciHostBridgeDxe: Add MemoryFence after write.

On AARCH64, there is no ordering guarantee between configuration
space (ECAM) writes and memory space reads (MMIO). ARM AMBA CHI
only guarantees ordering for reads and writes within a single address
region, however, on some systems MMIO and ECAM may be split into
separateaddress regions.

A problem may arise when an ECAM write is issued a completion before a
subsequent MMIO read is issued and receives a completion.

For example, a typical PCI software flow is the following:

1. ECAM write to device command register to enable memory space
2. MMIO read from device memory space for which access was enabled
   in step 1.

There is no guarantee that step 2. will not begin before the completion
of step 1.
on systems where ECAM/MMIO are specified as separate address regions,
even
if both spaces have the memory attributes device-nGnRnE.

- Add a barrier after the final PCI Configuration space write
in RootBridgeIoPciAccess. Configuration space reads should not have
side-efects.

- When configuration space is strongly ordered, this ensures
that program execution cannot continue until the completion
is received for the previous Cfg-Write, which may have side-effects.

- Risk of reading a "write-only" register and causing a CA which leaves
the device unresponsive. The expectation based on the PCI Base Spec
v6.1 section 7.4 is that all PCI Spec-defined registers will be readable,
however, there may exist design-specific registers that fall into
this category.

Signed-off-by: Aaron Pop <[email protected]>

Co-authored-by: Ard Biesheuvel <[email protected]>

MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciRootBridgeIo.c

@@ -1238,6 +1238,16 @@ RootBridgeIoPciAccess (
     }
   }
 
+  //
+  // If the access was a PCI write, it might have side effects that impact how
+  // the PCI device decodes its MMIO regions. Issue a barrier to ensure that
+  // subsequent MMIO accesses to those regions will not be reordered, and will
+  // not arrive before the PCI write.
+  //
+  if (!Read) {
+    MemoryFence ();
+  }
+
   return EFI_SUCCESS;
 }