Time Debugging

You have a scene that does render really slow. And you think it should not take that long. (e.g. local render 3min, farm render 20min)


These are some known issues and tips for debugging:

External files - network traffic

It happens often that your fileserver is able to handle artists working on a scene with many file inputs.

But once the scene is send to 10 (30?) clients at a time the network is slowed down.

Try to send the scene to 1 client only. Disable (most of) the other clients that they do not get any job.

Simulation cache files - read only

If you use simulations in your scene, please check that the simulation is not re-simulated.

Some simulators allow to set an option to read cache files only, no re-simulation.

If you are not sure if a setting has this effect, then close the scene, rename the cache files folder and re-open the scene.

If the scene does not simulate, you do not see anything of your simulation.

If you think that the render time should be about the same for all frames, but according to the logs the total frame time increases the higher the frame number send to a client, then this is an indication that the scene is re-simulated.

Simulation cache files - relative paths

If you use relative paths to your cache files (e.g. inside the Maya project folder), then you have to disable "local scene cache" for the job.

Reduce/Remove everything that takes render time

If you remove everything beside the issue you will reduce the chance that you get a false assumption about the issue.

Reduce render resolution and antialiasing.

Breakdown/Slim the scene. See "Scene breakdown" how to keep the object responsible for the render time. 

Render log time/CPU informations

RR and renderer have a few helpful prints in the render log.

For example a default RR+ Maya + Vray render:

++ Log Start +++

10.01 18:15.57

[2013/Oct/1|18:16:03] V-Ray: V-Ray core version is 2.00.01

File read in 4 seconds.

Result: C:/RR_localdata/cachedscenes/0257_0970_lighting_v01_004.mb

V-Ray: Starting render

[2013/Oct/1|18:16:35]

[2013/Oct/1|19:16:54] V-Ray: Rendering.

[2013/Oct/1|19:16:54] V-Ray: Pre-render export.

[2013/Oct/1|19:16:54] V-Ray: Pre-render export

[2013/Oct/1|20:12:35] V-Ray: Pre-render export done.

[2013/Oct/1|20:12:35] V-Ray: Rendering frames.

+++ Render Executable done ++

++ 10.01. 20:14.34 +++

Max core usage of job: 24.00cores (of 24 cores)

Startup to first frame: 59.28 m.s (render app start + scene load + 1st frame preprocessing)


Analysis:

Max CPU usage of the job is 24 of 24, so there is no thread limitation.

Startup to first frame is 59min. So it took a long time until the frame was started. If I compare it to the VRay pre-render export times, then I can see that there is an hour difference. Which confirms the long pre-processing time.

Add time prints

Add as many time prints as possible.

For example as pre/post render scripts.

If you use simulations, add a pre and post-frame script to all sim objects.

Add objects to your scene with a expression/script driven setting (e.g. position noise script) . Then you know when the scene/frame is being prepared after scene load.

Increase renderer verbose

Increase the renderer verbose output.

(Note: If you increase it to much, the log itselfs slows down the rendering. E.g. one log line takes 0.01s. Not much, but if you print 1000 log lines it takes 10s.)

Local workstation/artist settings/caches.

Sometimes a scene is fast for the artist because of local caches.

So it could be different on a different machine or even with a different user.

Please login with the same user that you use for rendering (see render log: ++ Log Start +++ Environment user is renderadmin)

on a render farm machine.

Open the scene and render the first frame. (Should not take long as you have decreased the AA and the resolution).

Then render the last frame of the scene.

How to send debug jobs to the farm

According to the "External files - network traffic" you should not send one scene to many clients at the same time.

If you get a fast render with a few clients and a slow render with many clients (same frames), then your network reached a limit. (Hint: Read rrInstallation section about Jumbo Frames)


You can set sequence divide min/max to:

  • 1/1: You get the full scene load and preprocessing time for every frame.
  • 3/3: You get once the scene load, but multiple times the frame proprocessing and render times.
    If you compare it to 1/1 you can calculate the scene load time.