Tutorial: Render Synthetic Objects using Blender 2.8

This page explains and illustrates how to render synthetic objects into photographs for project 4 using Blender 2.8. You can install from here. This page explains how to build your scene from scratch. You can load the provided sample ibl.blend file into Blender as a reference, but please note that the equirectangular image is likely in a different rotation than the one you generated, so there are some differences in camera parameters.

Key concepts

Your 3D scene has the following components: To generate the image, you need to render. First, you set the Render Engine to Cycles using the Render Properties (icon looks kind of like a clipboard or tv). Set the Sampling Render number to 1000 (lower for speed, higher for quality). For your final scene composite, you'll render once with all the objects, once with only the supporting plane, and one more time to generate the object masks. Make sure to save your complete scene before modifying your scene for these different renderings.

Step by step instructions

First clear out initial scene including sample mesh and lighting, and setup camera position

Next, go into perspective camera mode, and setup viewport dimension. You want the same dimension as your background image for this.

We then want to load background image onto this viewport. Click on 'g' to move background image around, and 's' to resize it.

Then, we want to add local scene. That is, add simple geometry (usually planes suffice) to recreate the geometry in the scene near where you'd like to insert objects. For best results, this should be close to where you placed the spherical mirror. Feel free to use the sample scene provided and move the vertices of the plane to match the surface you'd like to recreate (ignore the inserted bunny/teapot/etc for now). Once you're happy with the placement, add materials to the local scene: select a piece of local scene geometry, go to Properties->Materials, add a Diffuse BSDF material, and change the "Color" to roughly match the color from the photograph.

Finally, insert synthetic objects into the scene. There are basic models in Blender, and you can find others online: Turbosquid, Google 3D Warehouse, DModelz, etc. Add interesting materials to your inserted objects as well. Once finished, your scene should now look something like the right image below.
Then, add your equirectangular HDR image that you generated in your Jupyter Notebook: write_hdr_image(eq_image, 'equirectangular.hdr').
In the World tab on property panel, make sure Surface="Background" and Color="Environment Texture". Locate your saved HDR image in the filename field below "Environment Texture".

Once finished, your scene should now look something like the right image below.

Blender scene after: loading background image, modeling local scene, inserting objects


We can now render the scene to see a preview of what the inserted objects will look like. Make sure "Cycles" is the Render Engine in the Render Properties panel, and Render the scene (F12). Under Sampling in the Render Properties, you can set the Render value higher to improve quality or lower for a faster result. Recall that the solution of the HDR image is up to an known scale factor, so you may need to adjust the brightness strength. You can adjust the overall brightness by setting World Properties --> Surface --> Strength. You also may need to adjust your scene, e.g. to pick different materials or make sure that shadows cast by your objects don't go beyond the supporting plane. Once satisfied with the result, save the rendered image to disk.

This is a demo of how the Cycles render will work:

My rendered scene is down below; as you can tell, this is not quite the final result. To seamlessly insert the objects, we need to follow the compositing procedure outlined by Debevec (Section 6 of the paper). This requires rendering the scene twice (both with and without the inserted objects), and creating an inserted object mask.

Next, we'll render the "empty" scene (without inserted objects). Create a copy of your Blender scene and name it something like ibl-empty.blend. Open up the copy, and delete all of the inserted objects (but keep the local scene geometry such as support plane). Render this scene and save the result.

Finally, we need to create an object mask. The mask should be 0 where no inserted objects exist, and greater than 0 otherwise. First, create another duplicate of your scene and open it up (e.g. ibl-mask.blend). We can create the mask quickly using Blender by manipulating object materials and rendering properties:
  1. In the top panel, Choose 'Eevee Render'
  2. Select an object
  3. Go to the materials panel
  4. Remove the Material
  5. Click "New" to add a new material
  6. In the new material properties, under "Emissive"
  7. Repeat for all inserted objects
  8. In World panel, remove equirectangular lighting that was added previously
  9. Render the scene and save your mask as a PNG (or some lossless format)
  10. The PNG will contain a masked image where pixel intensities over 0.5 indicates True for object mask

After these steps, you should have the something akin to the following three rendered images:

Rendered image with objects, rendering without objects, object mask


To finish the insertion, we will use the above rendered images to perform "differential render" compositing. This can be done using a simple pixel-wise equation. Let R be the rendered image with objects, E be the rendered image without objects, M be the object mask, and I be the background image. The final composite is computed with:

composite = M*R + (1-M)*I + (1-M)*(R-E)*c

The first two terms effectively pastes the inserted objects into the background image, and the third term adds the lighting effects of the inserted objects (shadows, caustics, interreflected light, etc), modulated by c. Set c=1 initially, but try different values to get darker or lighter shadows/interreflections. The final compositing result I achieved using my image-based light is at the top of the page.

Some other tips on using Blender