Skip to content

Latest commit

 

History

History
477 lines (403 loc) · 21.8 KB

vulkan_dump_resources.md

File metadata and controls

477 lines (403 loc) · 21.8 KB
Raw

Dumping resources

Index

  1. Introduction
    1. General
    2. Block index and object ID
    3. Rules for providing command indices
    4. A simple example
    5. Index vector dimensionality
    6. A more complex example
  2. Command line options and input
    1. gfxrecon-replay command line params
  3. Output
    1. Json file output
    2. Image file output
    3. Buffer file output

Introduction

General

GFXReconstruct offers the capability to dump resources when replaying a capture file. These resources can be:

  1. Render targets

    All images used as render target attachments (both as color and depth attachments) either in render passes or with dynamic rendering.

  2. Vertex and index buffers

    Buffers bound as vertex and index buffers referenced by draw calls which have been marked for dumping.

  3. Results of compute (vkCmdDispatch and its variants) and ray tracing (vkCmdTraceRays) shaders.

    All result images and buffers used as descriptor bindings by dispatch and ray tracing shaders.

  4. Descriptor bindings used as inputs in all shader stages.

    All images and buffers used as descriptor bindings by draw calls, dispatch and ray tracing shaders.

The resources are dumped into files and can either be image files (bmp or png) or binary files.

Dumping can take place only while replaying a capture file either on desktop with the gfxrecon-replay tool or when replaying a capture file on Android with the replay application.

In order to enable the dump resources feature the DrawCalls and/or Dispatch, and/or TraceRays for which the related resources is desired to be dumped, need to be specified in some way. This is can be done by using the block index with which these commands are recorded inside the capture file (for more details see [Block index])(#block-index-and-object-id).

It is possible to dump resources from multiple draw calls and/or Dispatch/TraceRays in a single run by specifying multiple indices.

Block index and object ID

Each command (either a Vulkan command issued by the application or a meta command generated by GFXReconstruct internally) stored in the capture file can be identified by a unique, monotonically increasing number that each command is assigned. This is called the command's block index. In order to specify commands for dumping their resources this index needs to be specified for each command.

The simplest way to find the command block index for each Vulkan command is to use the gfxrecon-convert tool on a capture file which will convert the capture into a human readable json file. Use the --format jsonl option to produce a json file with one command per line. The resulting jsonl file looks something like this:

{"index":301,"function":{"name":"vkBeginCommandBuffer","thread":2,"return":"VK_SUCCESS","args":{"commandBuffer":80, ... }}},
{"index":302,"function":{"name":"vkCmdBeginRenderPass","thread":2,"cmd_index":1,"args":{"commandBuffer":80, ... }}},
{"index":303,"function":{"name":"vkCmdBindPipeline","thread":2,"cmd_index":2,"args":{"commandBuffer":80, ... }}},
{"index":304,"function":{"name":"vkCmdBindDescriptorSets","thread":2,"cmd_index":3,"args":{"commandBuffer":80, ... }}},
{"index":305,"function":{"name":"vkCmdSetViewport","thread":2,"cmd_index":4,"args":{"commandBuffer":80, ... }}},
{"index":306,"function":{"name":"vkCmdSetScissor","thread":2,"cmd_index":5,"args":{"commandBuffer":80, ... }}},
{"index":307,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":80, ... }}},

The block index is the index key used in each entry.

In a similar manner each vulkan object is assigned an object ID inside the capture file. When an object is referenced by a command in capture file, the object is referenced by its ID. In the above example the VkCommandBuffer with object ID 80 is referenced and is common between all commands.

Rules for providing command indices

Apart from the DrawCall/Dispatch/TraceRays indices, indices for several other Vulkan commands need to specified. In summary, the commands that can be specified are the following:

  1. DrawCalls This includes the indices of vkCmdDraw and all supported variants:

    • vkCmdDraw
    • vkCmdDrawIndexed
    • vkCmdDrawIndirect
    • vkCmdDrawIndexedIndirect
    • vkCmdDrawIndirectCount
    • vkCmdDrawIndexedIndirectCount
    • vkCmdDrawIndirectCountKHR
    • vkCmdDrawIndexedIndirectCountKHR

  2. Dispatch This includes the indices of vkCmdDispatch and all supported variants:

    • vkCmdDispatch
    • vkCmdDispatchIndirect

  3. Trace rays This includes the indices of vkCmdTraceRaysKHR and all supported variants:

    • vkCmdTraceRaysKHR
    • VkTraceRaysIndirectCommandKHR

Depending on the type of the commands that dumping is requested for, additional command indices need to be specified. These commands are:

  1. BeginCommandBuffer This is the index of the vkBeginCommandBuffer inside which the indices of the dump-able commands are provided.

  2. Render pass Render pass indices are required only for draw calls. All render pass indices which surround the provided draw calls must be specified - the indices for vkCmdBeginRenderPass, vkCmdNextSubpass (if any), and vkCmdEndRenderPass. In case of dynamic rendering the indices of vkCmdBeginRendering and vkCmdEndRendering must be provided instead.

  3. QueueSubmit The index of the vkQueueSubmit (or vkQueueSubmit2) in which the command buffer that includes the desired commands are submitted needs to be provided.

A simple example

Assuming the following imaginary excerpt from a capture file that contains the following commands:

{"index":301,"function":{"name":"vkBeginCommandBuffer", ... } },
{"index":302,"function":{"name":"vkCmdBeginRenderPass", ... } },
{"index":303,"function":{"name":"vkCmdBindPipeline", ... } },
{"index":304,"function":{"name":"vkCmdBindDescriptorSets", ... } },
{"index":305,"function":{"name":"vkCmdSetViewport", ... } },
{"index":306,"function":{"name":"vkCmdSetScissor", ... } },
{"index":307,"function":{"name":"vkCmdDraw", ... } },
{"index":308,"function":{"name":"vkCmdDraw", ... } },
{"index":309,"function":{"name":"vkCmdDraw", ... } },
{"index":310,"function":{"name":"vkCmdDraw", ... } },
{"index":311,"function":{"name":"vkCmdDraw", ... } },
{"index":312,"function":{"name":"vkCmdDraw", ... } },
{"index":313,"function":{"name":"vkCmdEndRenderPass", ... } },
{"index":314,"function":{"name":"vkEndCommandBuffer", ... } },
{"index":315,"function":{"name":"vkQueueSubmit", ... } },

It is possible to dump the depth and color attachments of all vkCmdDraw commands by providing the following json input file:

{
    "BeginCommandBuffer": [ 301 ],
    "Draw": [ [ 307, 308, 309, 310, 311, 312 ] ],
    "RenderPass": [ [ [ 302, 313 ] ] ],
    "QueueSubmit": [ 315 ]
}

Index vector dimensionality

Indices are provided in GFXReconstruct as vectors of indices. Each index vector, depending on the type of the command it describes can vary and is important, otherwise errors will be generated while parsing the input or the expected commands will not be dumped. Each vector's dimensionality is the following:

  • "BeginCommandBuffer" and "QueueSubmit": 1D

Commands recorded in multiple command buffers can be dumped in a single run. For each command buffer the index of the vkBeginCommandBuffer must be provided. The index of the vkQueueSubmit in which the command buffer is submitted must be provided. Both these vectors must have the same number of indices.

  • "Draw", "Dispatch" and "TraceRays": 2D

These vectors are two dimensional. The first dimension corresponds to BeginCommandBuffer each vector belongs to. I.e.:

{
    "BeginCommandBuffer" :  [ 10, 20 ],
    "TraceRays":            [ [ 220, 230, 240 ], [] ],
    "Dispatch" :            [ [], [ 250, 260, 270 ] ],
    "QueueSubmit" :         [ 350, 360 ]
}

In the example above the vkCmdTraceRays [ 220, 230, 240 ] belong to vkBeginCommandBuffer with block index 10 and are submitted in vkQueueSubmit with block index 350.

vkCmdDispatch with indices [ 250, 260, 270 ] belong to command buffer with vkBeginCommandBuffer with block index 20 and are submitted in vkQueueSubmit with block index 360.

  • "RenderPass": 3D

Inside a command buffer, Vulkan allows multiple render passes with multiple sub-passes. In order to support that for multiple command buffers a 3D array is required.

A more complex example

A hypothetical json output of the gfxrecon-convert tool could look something like the following:

{"index":10,"function":{"name":"vkBeginCommandBuffer","thread":2,"return":"VK_SUCCESS","args":{"commandBuffer":59, ... }}},
{"index":11,"function":{"name":"vkBeginCommandBuffer","thread":2,"return":"VK_SUCCESS","args":{"commandBuffer":60, ... }}},
{"index":12,"function":{"name":"vkCmdBeginRenderPass","thread":2,"cmd_index":1,"args":{"commandBuffer":59, ... }}},
{"index":13,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":14,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":15,"function":{"name":"vkCmdNextSubpass","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":16,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":17,"function":{"name":"vkCmdDraw","thread":3,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":18,"function":{"name":"vkCmdEndRenderPass","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":19,"function":{"name":"vkCmdBeginRenderPass","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":20,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":21,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":22,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":23,"function":{"name":"vkCmdDraw","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":24,"function":{"name":"vkCmdEndRenderPass","thread":2,"cmd_index":6,"args":{"commandBuffer":59, ... }}}
{"index":25,"function":{"name":"vkEndCommandBuffer","thread":2,"return":"VK_SUCCESS","args":{"commandBuffer":59}}}
{"index":26,"function":{"name":"vkCmdBeginRenderPass","thread":2,"cmd_index":1,"args":{"commandBuffer":60, ... }}},
{"index":27,"function":{"name":"vkCmdDrawIndexed","thread":2,"cmd_index":6,"args":{"commandBuffer":60, ... }}}
{"index":28,"function":{"name":"vkCmdDrawIndexed","thread":2,"cmd_index":6,"args":{"commandBuffer":60, ... }}}
{"index":29,"function":{"name":"vkCmdDrawIndexed","thread":2,"cmd_index":6,"args":{"commandBuffer":60, ... }}}
{"index":30,"function":{"name":"vkCmdEndRenderPass","thread":2,"cmd_index":6,"args":{"commandBuffer":60, ... }}}
{"index":31,"function":{"name":"vkEndCommandBuffer","thread":2,"return":"VK_SUCCESS","args":{"commandBuffer":60}}}
...
{"index":50,"function":{"name":"vkQueueSubmit", ... ,"commandBufferCount":1,"pCommandBuffers":[59], ... }},
{"index":51,"function":{"name":"vkQueueSubmit", ... ,"commandBufferCount":1,"pCommandBuffers":[60], ... }},

The indices submitted to gfxrecon-replay for dumping are the following:

{
    "BeginCommandBuffer" : [ 10, 11 ],
    "Draw" :               [ [ 13, 14, 16, 17, 20, 21, 22, 23 ],
                             [ 27, 28, 29 ] ],
    "RenderPass" :         [ [ [ 12, 15, 18 ], [ 19, 24 ] ], [ [ 26, 30 ] ] ],
    "QueueSubmit" :        [ 50, 51 ]
}

In this example two command buffers are submitted for dumping, one with object ID 59 and one with object ID 60.

  • The first command buffer with object ID 59 and vkBeginCommandBuffer with index 10 contains:
    • The draw calls [ 13, 14, 16, 17, 20, 21, 22, 23 ]
    • These draw calls are divided into two render passes:
      • [ 12, 15, 18 ]: vkCmdBeginRenderPass: 12, vkCmdNextSubpass: 15 and vkCmdEndRenderPass: 18
      • [ 19, 24 ]: vkCmdBeginRenderPass: 19 and vkCmdEndRenderPass: 24
  • The second command buffer with object ID 60 and vkBeginCommandBuffer with index 20 contains:
    • The draw calls [ 27, 28, 29 ]
    • One render pass with 1 sub pass:
      • [ 26, 30 ]: vkCmdBeginRenderPass: 26 and vkCmdEndRenderPass: 30
  • Command buffer 59 is submitted for execution with vkQueueSubmit with index 50 and command buffer 60 is submitted in vkQueueSubmit with index 51

Command line options and input

gfxrecon-replay command line params

Dump resources feature can be control in several ways. To do so, a number of parameters can be provided to either to the gfxrecon-replay tool or to the Android application through the gfxrecon.py script:

--dump-resources BeginCommandBuffer=<n>,Draw=<o>,BeginRenderPass=<p>,NextSubPass=<q>,EndRenderPass=<r>,Dispatch=<s>,TraceRays=<t>,QueueSubmit=<u>
              Dump gpu resources after the given vkCmdDraw*, vkCmdDispatch, or vkCmdTraceRaysKHR is replayed. The parameter for
              each is a block index from the capture file.  The additional parameters are used to identify during which occurence
              of the vkCmdDraw/VkCmdDispath/VkCmdTrancRaysKHR resources will be dumped.  NextSubPass can be repeated 0 or more times to
              indicate subpasses withing a render pass.  Note that the minimal set of parameters must be one of:
                  BeginCmdBuffer, Draw, BeginRenderPass, EndRenderPass, QueueSubmit
                  BeginCmdBuffer, Dispatch, QueueSubmit
                  BeginCmdBuffer, TraceRays, QueueSubmit
  --dump-resources <filename>
              Extract --dump-resources args from the specified file, with each line in the file containing a comma or space separated
              list of the parameters as in the above --dump-resources description. The file can contain multiple lines specifying multiple dumps.
  --dump-resources <filename>.json
              Extract --dump-resource args from the specified json file. The format for the json file is documented in detail
              in the gfxreconstruct documentation.
  --dump-resources-image-format <format>
              Image file format to use for image resource dumping.
              Available formats are:
                  bmp         Bitmap file format.  This is the default format.
                  png         Png file format.
  --dump-resources-before-draw
              In addition to dumping gpu resources after the CmdDraw, CmdDispatch and CmdTraceRays calls specified by the
              --dump-resources argument, also dump resources before those calls.
  --dump-resources-scale <scale>
              Scale images generated by dump resources by the given scale factor. The scale factor must be a floating point number
              greater than 0. Values greater than 10 are capped at 10. Default value is 1.0.
  --dump-resources-dir <dir>
              Directory to write dump resources output files. Default is the current working directory.
  --dump-resources-dump-depth-attachment
              Configures whether to dump the depth attachment when dumping draw calls. Default is disabled.
  --dump-resources-dump-color-attachment-index <index>
              Specify which color attachment to dump when dumping draw calls. Index should be an unsigned zero
              based integer. Default is to dump all color attachments.
  --dump-resources-dump-vertex-index-buffers
              Enables dumping of vertex and index buffers while dumping draw call resources.
  --dump-resources-json-output-per-command
              Enables storing a json output file for each dumped command. Overrides default behavior which
              is generating one output json file that contains the information for all dumped commands.
  --dump-resources-dump-immutable-resources
              Enables dumping of resources that are used as inputs in the commands requested for dumping
  --dump-resources-dump-all-image-subresources
              Enables dumping of all image sub resources (mip map levels and array layers)

Output

Json file output

The dump resources feature generates a number of output files. A json file with an entry for each set of resources that are dumped is generated, as well as image and binary files. The json output file name is derived from the input file name, with the ".gfxr" extension replaced with "_rd.json", i.e. an input file name of "vulkanCapture.gfxr" will result in a json output file name of "vulkanCapture_rd.json".

The output json consists of 4 main entries:

  1. Header

The header contains the path to the capture file, GFXR and vulkan version and the dump resources parameters used when replaying.

  1. Draw call commands

The draw calls are listed in an array. Each draw call entry contains information regarding:

  • The Vulkan api command parameters
  • The color and/or depth attachments
  • Vertex and/or index buffers (if requested)
  • The immutable descriptors which can be images and/or buffers
  1. Dispatch commands and Trace Rays commands
  • The Vulkan api command parameters
  • The generated images and/or buffers
  • The immutable descriptors which can be images and/or buffers

Here is an example of a json output file:

[
{
  "header": {
    "source-path": "vulkanCapture.gfxr",
    "gfxrecon-version": "1.0.3"
    "vulkan-version": "1.3.275"
    "dumpResourcesOptions": {
      "scale": 1.0,
      "dumpResourcesOutputDir": "",
      "dumpResourcesColorAttachmentIndex": -1,
      "dumpResourcesBefore": false,
      "dumpResourcesDumpDepth": true,
      "dumpResourcesDumpVertexIndexBuffer": true,
      "dumpResourcesDumpImmutableResources": true,
      "dumpResourcesDumpAllImageSubresources": false
    }
  }
},
{
   "drawCallCommands": [
    {
      "drawIndex": 407,
      "beginCommandBufferIndex": 399,
      "queueSubmitIndex": 461,
      "parameters": {
        "drawCallType": "vkCmdDraw",
        "vertexCount": 24576,
        "instanceCount": 1,
        "firstVertex": 0,
        "firstInstance": 0
      },
      "colorAttachments": [
        {
          "imageId": 15,
          "format": "VK_FORMAT_B8G8R8A8_UNORM",
          "type": "VK_IMAGE_TYPE_2D",
          "aspect": "COLOR",
          "dimensions": [
            1280,
            720,
            1
          ],
          "mipLevel": 0,
          "arrayLayer": 0,
          "file": "Draw_407_qs_461_bcb_399_att_0_aspect_color.bmp"
        }
      ],
      "depthAttachments": [
        {
          "imageId": 32,
          "format": "VK_FORMAT_D32_SFLOAT_S8_UINT",
          "type": "VK_IMAGE_TYPE_2D",
          "aspect": "DEPTH",
          "dimensions": [
            1280,
            720,
            1
          ],
          "mipLevel": 0,
          "arrayLayer": 0,
          "file": "Draw_407_qs_461_bcb_399_depth_att_aspect_depth.bmp"
        },
     ...
      ],
      "vertexBuffers": [
        {
          "bufferId": 75,
          "vertexBufferBinding": 0,
          "file": "VertexBuffers_qs_461_bcb_399_dc_407_binding_0.bin"
        }
      ],
      "descriptors": {
        "vertex": [
          {
            "type": "VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER",
            "set": 0,
            "binding": 2,
            "arrayIndex": 0,
            "descriptor": {
              "bufferId": 79,
              "file": "Buffer_79_qs_461_bcb_399_rp_0.bin"
            }
          }
        ],
        "fragment": [
          {
            "type": "VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER",
            "set": 0,
            "binding": 1,
            "arrayIndex": 0,
            "descriptor": [
              {
                "imageId": 68,
                "format": "VK_FORMAT_R8G8B8A8_UNORM",
                "type": "VK_IMAGE_TYPE_2D",
                "aspect": "COLOR",
                "dimensions": [
                  256,
                  1,
                  1
                ],
                "mipLevel": 0,
                "arrayLayer": 0,
                "file": "Image_68_qs_461_bcb_399_rp_0_aspect_color.bmp"
              }
            ]
          },
          ...
        ]
      }
    },
  ]
},
{
  "dispatchCommands": [
    {
      "dispatchIndex": 294,
      "beginCommandBufferIndex": 290,
      "queueSubmitIndex": 723,
      "parameters": {
        "dispatchType": "vkCmdDispatch",
        "groupCountX": 96,
        "groupCountY": 1,
        "groupCountZ": 1
      },
      "outputs": {
        "buffers": [
          {
            "type": "VK_DESCRIPTOR_TYPE_STORAGE_BUFFER",
            "set": 0,
            "binding": 0,
            "arrayIndex": 0,
            "bufferId": 75,
            "file": "Dispatch_294_qs_723_bcb_290_stage_compute_set_0_binding_0_index_0_buffer.bin"
          }
        ]
      },
      "descriptors": [
        {
          "type": "VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER",
          "set": 0,
          "binding": 1,
          "arrayIndex": 0,
          "descriptor": {
            "bufferId": 90,
            "file": "Buffer_90_qs_723_bcb_290.bin"
          }
        },
        {
          "type": "VK_DESCRIPTOR_TYPE_STORAGE_BUFFER",
          "set": 0,
          "binding": 0,
          "arrayIndex": 0,
          "descriptor": {
            "bufferId": 75,
            "file": "Buffer_75_qs_723_bcb_290.bin"
          }
        }
      ]
    },
    ...
  ]
}]

Image file output

The image files that created are either images (in one of the supported image formats), .astc files, or raw binary files (.bin). Astc compressed images are dumped as .astc files which can be decompressed and converted into a plain image file offline on the host. Raw binary files are created when the dumped resource is an image with a format which cannot be converted into a plain 32bit RGBA layout.

Buffer file output

All buffers are dumped as raw binary files (.bin).