[Issue]: Memory copy started by dependency signal for pinned memory exhibits race condition

[nstomlinson]

Problem Description

Scheduling a memory copy with hsa_amd_memory_async_copy and using a dependency signal to start the copy after hsaKmtMapMemoryToGPUNodes has returned seems to result in a race condition. See the attached tarball. Reproduction instructions and other information is in README.md. The exhibiting program is in main.cpp.

2024100201-pincopy.tar.gz

Operating System

Archlinux

CPU

AMD Ryzen Threadripper 1950X

GPU

AMD Radeon Pro W6800

ROCm Version

ROCm 6.2.2

ROCm Component

ROCK-Kernel-Driver

Steps to Reproduce

Unzip the attached tarball, and follow the instructions in README.md.

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

ROCk module is loaded
=====================    
HSA System Attributes    
=====================    
Runtime Version:         1.1
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE                              
System Endianness:       LITTLE                             
Mwaitx:                  DISABLED
DMAbuf Support:          YES

==========               
HSA Agents               
==========               
*******                  
Agent 1                  
*******                  
  Name:                    AMD Ryzen Threadripper 1950X 16-Core Processor
  Uuid:                    CPU-XX                             
  Marketing Name:          AMD Ryzen Threadripper 1950X 16-Core Processor
  Vendor Name:             CPU                                
  Feature:                 None specified                     
  Profile:                 FULL_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        0(0x0)                             
  Queue Min Size:          0(0x0)                             
  Queue Max Size:          0(0x0)                             
  Queue Type:              MULTI                              
  Node:                    0                                  
  Device Type:             CPU                                
  Cache Info:              
    L1:                      32768(0x8000) KB                   
  Chip ID:                 0(0x0)                             
  ASIC Revision:           0(0x0)                             
  Cacheline Size:          64(0x40)                           
  Max Clock Freq. (MHz):   3400                               
  BDFID:                   0                                  
  Internal Node ID:        0                                  
  Compute Unit:            32                                 
  SIMDs per CU:            0                                  
  Shader Engines:          0                                  
  Shader Arrs. per Eng.:   0                                  
  WatchPts on Addr. Ranges:1                                  
  Features:                None
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: FINE GRAINED        
      Size:                    131770752(0x7daa980) KB            
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
    Pool 2                   
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    131770752(0x7daa980) KB            
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
    Pool 3                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    131770752(0x7daa980) KB            
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
  ISA Info:                
*******                  
Agent 2                  
*******                  
  Name:                    gfx1030                            
  Uuid:                    GPU-c5f17292dba09909               
  Marketing Name:          AMD Radeon Pro W6800               
  Vendor Name:             AMD                                
  Feature:                 KERNEL_DISPATCH                    
  Profile:                 BASE_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        128(0x80)                          
  Queue Min Size:          64(0x40)                           
  Queue Max Size:          131072(0x20000)                    
  Queue Type:              MULTI                              
  Node:                    1                                  
  Device Type:             GPU                                
  Cache Info:              
    L1:                      16(0x10) KB                        
    L2:                      4096(0x1000) KB                    
    L3:                      131072(0x20000) KB                 
  Chip ID:                 29603(0x73a3)                      
  ASIC Revision:           1(0x1)                             
  Cacheline Size:          128(0x80)                          
  Max Clock Freq. (MHz):   2555                               
  BDFID:                   3328                               
  Internal Node ID:        1                                  
  Compute Unit:            60                                 
  SIMDs per CU:            2                                  
  Shader Engines:          4                                  
  Shader Arrs. per Eng.:   2                                  
  WatchPts on Addr. Ranges:4                                  
  Coherent Host Access:    FALSE                              
  Features:                KERNEL_DISPATCH 
  Fast F16 Operation:      TRUE                               
  Wavefront Size:          32(0x20)                           
  Workgroup Max Size:      1024(0x400)                        
  Workgroup Max Size per Dimension:
    x                        1024(0x400)                        
    y                        1024(0x400)                        
    z                        1024(0x400)                        
  Max Waves Per CU:        32(0x20)                           
  Max Work-item Per CU:    1024(0x400)                        
  Grid Max Size:           4294967295(0xffffffff)             
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)             
    y                        4294967295(0xffffffff)             
    z                        4294967295(0xffffffff)             
  Max fbarriers/Workgrp:   32                                 
  Packet Processor uCode:: 118                                
  SDMA engine uCode::      83                                 
  IOMMU Support::          None                               
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    31440896(0x1dfc000) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       FALSE                              
    Pool 2                   
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    31440896(0x1dfc000) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       FALSE                              
    Pool 3                   
      Segment:                 GROUP                              
      Size:                    64(0x40) KB                        
      Allocatable:             FALSE                              
      Alloc Granule:           0KB                                
      Alloc Alignment:         0KB                                
      Accessible by all:       FALSE                              
  ISA Info:                
    ISA 1                    
      Name:                    amdgcn-amd-amdhsa--gfx1030         
      Machine Models:          HSA_MACHINE_MODEL_LARGE            
      Profiles:                HSA_PROFILE_BASE                   
      Default Rounding Mode:   NEAR                               
      Default Rounding Mode:   NEAR                               
      Fast f16:                TRUE                               
      Workgroup Max Size:      1024(0x400)                        
      Workgroup Max Size per Dimension:
        x                        1024(0x400)                        
        y                        1024(0x400)                        
        z                        1024(0x400)                        
      Grid Max Size:           4294967295(0xffffffff)             
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)             
        y                        4294967295(0xffffffff)             
        z                        4294967295(0xffffffff)             
      FBarrier Max Size:       32                                 
*** Done ***

Additional Information

No response

[nstomlinson]

Just a thought that occurred to me: I don't think the bug is specific to the SDMA. It's just more easily reproduced that way. If I put a second GPU in the machine, it seems to take longer for the GPU to see the pinned memory - long enough to reproduce the problem with a kernel rather than SDMA copy.

[ppanchad-amd]

Hi @nstomlinson. Internal ticket has been created to investigate your issue. Thanks!

[nstomlinson]

Thanks :) Please let me know if there's anything I can do to help.

[zichguan-amd]

Hi @nstomlinson, I couldn't repro the issue with any/no wait time. Can you send the core dump?

[nstomlinson]

Hello

• I've attached a core dump, binaries with debug information, and new system information.
• I ran the build and test in the Docker image for Ubuntu 24.04 this time to get a core dump that better matches what you might expect.
• I did notice it took several attempts to get the reproducer to crash.
• It should be possible to catch it crashing with for I in {0..99} ; do timeout 10 ./reproducer 0 ; if [ "$?" != "124" ] ; then break ; fi ; done (though in practice, a shorter timeout and fewer iterations should suffice).
• The system is also a bit different this time, in particular, there are now two GPUs in this machine. The tarball includes lspci and rocminfo for this reason.

Thanks!

[zichguan-amd]

Thanks for the information. I can now reproduce the error with the libs that you provided. I'll look into it. Can you tell me how were the .so files built? I also noticed that you have

=====================    
HSA System Attributes    
=====================    
Runtime Version:         1.1

With ROCm 6.2.2 it should look like:

ROCk module version 6.8.5 is loaded
=====================    
HSA System Attributes    
=====================    
Runtime Version:         1.14
Runtime Ext Version:     1.6

I suspect that it's a problem in the kernel driver. Since you are on Arch which is not officially supported, kfd may have unexpected behaviours. Since you need to expose kfd to the docker container, I believe that's also why it fails on Ubuntu. Can you check if the problem if reproducible locally on Ubuntu?

[nstomlinson]

I ran rocminfo from the host machine, which is on the ROCm 6.0.2 userspace. That probably accounts for the difference in HSA system attributes you observed. There's a build.sh script in the original tarball I attached. I built the libraries I provided like ./build.sh Debug.

[zichguan-amd]

Can you update ROCm on the host to 6.2.2 and check if the problem persists?

[nstomlinson]

I did (by building the parts of ROCm userspace that it uses). It does still happen in that case. I haven't tested bare metal Ubuntu yet though.

[zichguan-amd]

You can also try 6.2.1 via AUR, see https://aur.archlinux.org/packages/opencl-amd. Unfortunately, Arch is not officially supported so I recommend you use any of the supported OS listed here if possible.

[zichguan-amd]

Closing due to lack of recent activity. Please update the issue if it persists and I'll reopen it.

[ChrisKitching]

Closing due to lack of recent activity. Please update the issue if it persists and I'll reopen it.

Has a kernel patch been submitted etc., or was this issue just closed due to lack of activity?

If no action has been taken to fix it, then it does indeed persist!

A reproducer has been provided in the initial report. Does this work for you?

[powderluv]

@zichguan-amd please confirm if this was fixed.