Blender script for overlaying aligned mesh on input images

[vadimkantorov]

Hi @gkioxari!

Following #22, I tried to create a rendering Blender script myself: https://gist.github.com/vadimkantorov/eb53ce740cb80c12444079875facff85

From image

It renders

Since it's my first Blender use, it's mostly a combination of other code found online, so I don't understand everything perfectly. And the rendered image is quite noisy.

I would be very grateful if you could comment or provide feedback:

1. Are mesh vertex coordinates matching normalised image coordinates? (I assumed, yes)
2. How to set camera parameters properly? in particular, what should be clipping bounds? In my example the input image is square, but the camera field of view is rectangular? Should I just set scene.render.resolution_x to input image resolution?
3. Which renderer have you used to produce images in Matching the geometry to the input image #22? I've used Cycles.
4. Which approach have you used to add the original image on background? I found that it's quite non-trivial in Blender. In my example the model is clearly not aligned to the image, is not sharp and is geometrically distorted.

[vadimkantorov]

Using render(...) instead of unused_build_environment_texture_background(...) allowed me to get the following image (gist updated), but I am still not sure if alignment is done correctly. The rendered chair seems smaller than the original image, not sure at what depth the background image was added.

@gkioxari It would be immensely helpful if you could comment on a simple and correct way of adding a background image to blender scene and ensuring correct coordinates. I could not figure out how to resolve problems with environment texture and ended up using snippet that creates nodes. Another useful feature may be partial opacity of the rendered mesh in order to better check object boundaries alignment.

def render(background_path, output_path):
    bpy.context.scene.use_nodes = True
    tree = bpy.context.scene.node_tree
    links = tree.links

    for node in tree.nodes:
        tree.nodes.remove(node)
    
    image_node = tree.nodes.new('CompositorNodeImage')
    scale_node = tree.nodes.new('CompositorNodeScale')
    alpha_over_node = tree.nodes.new('CompositorNodeAlphaOver')
    render_layer_node = tree.nodes.new('CompositorNodeRLayers')
    file_output_node = tree.nodes.new('CompositorNodeOutputFile')

    #enum in [‘RELATIVE’, ‘ABSOLUTE’, ‘SCENE_SIZE’, ‘RENDER_SIZE’], default ‘RELATIVE’
    scale_node.space = 'RENDER_SIZE'
    image_node.image = bpy.data.images.load(background_path)
    file_output_node.base_path = os.path.dirname(output_path)
    file_output_node.file_slots[0].path = os.path.basename(output_path)

    links.new(image_node.outputs[0], scale_node.inputs[0])
    links.new(scale_node.outputs[0], alpha_over_node.inputs[1])
    links.new(render_layer_node.outputs[0], alpha_over_node.inputs[2])
    links.new(alpha_over_node.outputs[0], file_output_node.inputs[0])

    bpy.ops.render.render(write_still = False)

ef unused_build_environment_texture_background(world, background_path, rotation = 0.0):
    world.use_nodes = True
    node_tree = world.node_tree

    environment_texture_node = node_tree.nodes.new(type="ShaderNodeTexEnvironment")
    environment_texture_node.image = bpy.data.images.load(background_path)

    mapping_node = node_tree.nodes.new(type="ShaderNodeMapping")
    mapping_node.inputs["Rotation"].default_value = (0.0, 0.0, rotation)

    tex_coord_node = node_tree.nodes.new(type="ShaderNodeTexCoord")

    node_tree.links.new(tex_coord_node.outputs["Generated"], mapping_node.inputs["Vector"])
    node_tree.links.new(mapping_node.outputs["Vector"], environment_texture_node.inputs["Vector"])
    node_tree.links.new(environment_texture_node.outputs["Color"], node_tree.nodes["Background"].inputs["Color"])

    return environment_texture_node.image.size

[vadimkantorov]

My current attempt (along with estimated rotation of 2D bounding box, the lower plane is considered fixed to XZ (?)):

Still not sure if I got the depth / coordinates correctly

My code is at https://gist.github.com/vadimkantorov/eb53ce740cb80c12444079875facff85

Would be glad to have any feedback on the code / rendering, this is my first use of blender.

Thank you :)

[vadimkantorov]

For the demo image wget https://user-images.githubusercontent.com/4369065/77708628-cda99d00-6f85-11ea-949a-5dad891005ee.jpg, I'm getting the images below, the mesh does not fit the image (as opposed to original results in https://github.com/facebookresearch/meshrcnn/tree/master/demo). So it seems that lens=20 is maybe off. Or some other scaling / depth params? Any ideas? Thank you!

My script is at https://gist.github.com/vadimkantorov/eb53ce740cb80c12444079875facff85