Tesla M40 won't work on Q77 with 16GB RAM #77
Comments
|
@godcrying Tesla M40 requires 32GB BAR to my knowledge, this is impossible to make work on Ivy Bridge with 32GB of RAM due to the 36-bit (16GB) addressing limitation. The reason the patches cause this issue is because unpatched Ivy Bridge just ignores BARs =>4GB and leaves them unallocated. But if you use Linux it may be possible to use this GPU on Ivy Bridge along with less than 30GB of RAM. You need to turn off 4G decoding and modify DSDT to change When 4G decoding is disabled the now exposed 64-bit MMIO region should be empty so Linux should allocate the 32GB BAR Tesla M40 there without any issues. |
|
@xCuri0 Really thanks for your answer. I've been stuck with this trouble for a long time. |
|
It may work, can't give any guarantees though. You don't need to do any flashing but doing the DSDT patch in BIOS is recommended over having Linux replace it if you want to make it permanent. But a few people had a similar method work on Haswell systems |
|
@xCuri0 Thank you. I will try it. |
|
Post |
|
@xCuri0 I have reduced RAM to 8G and turn off the 4G decoding. The result of And I also add The |
|
@godcrying Can you send output of |
|
@xCuri0 The result is : And I also checked DSDT, which is:
|
|
@godcrying can you send the whole modified DSDT ? because the 64-bit region isn't there |
|
dsdt.dsl.txt I've delete the MM64==Zero |
|
@godcrying Remove And change You should then get full 36-bit range usable then and kernel should allocate M40 there. |
|
Wow! brilliant! 😄😄 @xCuri0 Really appreciate your help ! |
|
@godcrying Nice! I'm suprised it actually worked. You could also send |
|
here it is: The address has been And without By the way, how much RAM space left can I use for my host memory? |
|
@godcrying Anything under 30GB should work |
|
@xCuri0 Finally i found it works only with 8G RAM. When I increase the RAM to 16G, my card doesn't work. Maybe there is a limit that card ram + host ram <= 32G ? |
|
@godcrying can u send full dmesg output and also |
There's the whole 32-64GB region still available, I have no idea why it's saying no space for the 32GB BAR. Maybe realloc will fix it like it says |
|
@xCuri0 I added |
it seems like the PCI bridge needs a 48GB BAR for some reason ? I have no idea why, don't really know about how PCI bridges work. On my LGA1155 system 48GB along with 16GB RAM and other PCIe devices isn't going to fit within 64GB addressing limit. Need to figure out why 48GB bridge window is needed and if it's possible to reduce it. Btw I think you made a mistake in DSDT edit (not cause of this issue though) |
xCuri0
changed the title
From a quick glance it seems your BIOS has dumbly allocated BAR2 at 16GiB address instead of behind BAR1 at 32GiB. Best to fix this or if too hard then turn off 'remap' in BIOS so that you might have at least 15GiB+ of RAM. |
|
This also works for a Xeon Phi on an Asus P8Z77-V LX. Using the |
|
I tried using this method on a Z97-WS (Haswell) and successfully exposed the larger memory space at the PCI root, but am running into a snag with the PLX bridge (PCI Bus 0000:01) present on the chip. It seems no matter what I try, I can't get a large memory window assigned at the bridge so that the gpus underneath can get resized properly with the I have managed to use UEFIPatch + ReBarState to get up to 4G per card, and in that case, the PLX bridge (PCI Bus 0000:01) gets 12G of prefetchable memory assigned, but this is still inadequate for these cards. What I'm talking about: Any ideas how to get the PLX chip memory window opened up too? /cc @xCuri0 EDIT: I managed to get a larger prefetchable memory in the PLX bridge by using the |
|
@zcrypt0 why not use rebaruefi and let the bios do the pci allocation instead of making linux do it ? |
|
@xCuri0 I can get up to 8gb that way, but only when I take out some of the RAM. To enable GPU p2p communication via pcie (thanks to geohotz recent driver mod), I need 32gb BAR per gpu (preferably 4 of them). I was reading here about the 36 bit limit but in reality Haswell has a 39 bit addressable space which led me to the dsdt modifications. I figured since the 39bit address space seems to be available in linux (whereas we are are only getting 36 bits in the bios), this might be a way to get access to the full 39 bits. |
|
yeah the bios will only allocate 36-bit so you have to let linux do the allocation after dsdt mod to unlock 39-bit |
|
@zcrypt0 the first output shows that the nvidia driver isn't resizing your gpus (it's still 256mb), there's a driver argument you need to make it do it. amd gpus do this automatically on linux the plx bridge will get set properly then |
|
I set the driver argument in that case and confirmed it was set with the appropriate commands (I don't have the link readily available, something like |
System
My machine is same with the link #11 (comment). However, after patched my bios following the "Using-UEFIPatch" guide, and flushed the patched bios image, I can't boot my machine with my Nvidia Card pluged-in. I can't even boot to the bios setting UI. The screen has no signal and the number lock of the keyboard has no light.
The machine can boot with my nvidia card pluged-in using unpatched bios, but nvidia-smi reports "no device found error".
The machine can't boot with my nvidia card pluged-in using patched bios, but can boot without my nvidia card.
The text was updated successfully, but these errors were encountered: