How a Houdini PDG cook works under the hood

This is an example how Houdini executes a PDG graph.

No matter which scheduler you use.
Even Houdinis "Local Scheduler" works like this.

1. Houdini with PDG graph & TOP nodes:
   There must be one Houdini instance running that has opened the scene 
   and cooks the PDG graph (The play button).
   Houdini creates many files with informations about each task execution.
   
   The PDG graph contains TOP Nodes.
   Each TOP node contains a list of tasks to be executed.
   The tasks of one a TOP node may depend on some or all tasks of the parent TOP node.
   
   
   
2. Scheduler:
   The Scheduler is responsible to start the PDG server and distribute the tasks.
   You will probably use one or both of these Schedulers:
   1) Local Scheduler
       Every Task of the TOP node is executed on this machine
   2) Royal Render Scheduler
       Tasks are send to the farm
   
   
   
3. PDG Server:
   The PDG Server provides
   a) connection between Houdini and the tasks.
   b) connection between tasks in case they are linked (e.g. splitted simulation)
   c) webserver for status information and time statistics
   
   
   
4. Tasks:
   This is the actual execution.

Royal Render Scheduler

The Scheduler has two modes:

1) Task Scheduler

    If you cook (click on the play button) the PDG graph, 
    then Houdini tells RR to start the PDG server and to start the tasks.
    Your Houdini instance controls the PDG execution and must stay running.
    If you do not want Houdini to be open all the time, let an other machine run Houdini with "Submit PDG graph as job"

2) Submit PDG graph as job

    This submits the whole scene to some other machine.
    Note that this may not execute any task on that machine.
    It just opens the scene and cooks the PDG graph.
    It is just a way that you do not have to keep Houdini open on your machine all the time.

   

RR Task Scheduler

The RR Task Scheduler submits multiple jobs:

• Start PDG Server
  Note: This job runs all the time and does not use CPU nor memory (almost).
  Note: The http address of the progress is written into the render log file.
  
  
• Stop PDG Server
  This is just sending a command to the PDG server to shut down.
  
  
• A job for each TOP node with as many frames as tasks of the node.
  
  
  

Note:
All frames(tasks) of these jobs are inactive on submission.
Houdini controls what is started when.

Combine multiple Scheduler

Each TOP node contains an override setting for a Scheduler. 

This way you can use multiple scheduler for different tasks.
Note: If the override is not set, the TOP node uses the first Scheduler it finds.

Examples:

• Use the Local Scheduler to execute all tasks on the same machine.
  Then use the Royal Render Scheduler "Submit PDG graph as job" to execute everything on some other machine.
  
  
• Use the Local Scheduler for very fast TOPs and the Royal Render Scheduler for the more complex TOPs.

TOP with splitted simulation

Note that there are TOPs which split a single simulation into multiple tasks.
In this case multiple machines are working on one simulation.

BUT: multiple machine must work on this simulation.
Which means that if the TOP splits a simulation into 5 tasks 

and 4 rrClients have started a task, then these 4 rrClients stay idle and wait!

Until a 5th client takes the last task.

"Fully Static Cook"

Actually Houdini offers 3 types of Scheduler modes.
The "Task Scheduler", 

the "Submit PDG graph as job" 

and "Fully Static Cook".

RR has not implemented the "Fully Static Cook".

This would submit all tasks of the TOP onto the farm with all required dependencies.
On the RoyalRender side, this would flood the render queue with hundreds of separate 1-frame jobs (depending on the size of your graph)
And on the Houdini side, it has multiple disadvantages. 

Like some TOP nodes are not supported. 
Therefore SideFX states this information for PDG schedulers: 
"....For the above reasons we do not recommend attempting to generate fully static jobs from PDG graphs."

Tutorial 1 - Use your Houdini session as PDG controller

In this tutorial your Houdini interface controls which PDG tasks are being processed and send to Royal Render.

1. Please copy the example file "distributed_flip" from your Houdini help to any location in your network.
   The file is located in this folder:
   c:\Program Files\Side Effects Software\Houdini 19.5.569\houdini\help\files\pdg_examples\top_distributedsim\distributed_flip.hip
   
   
2. Open the scene file and open /obj /topnet in Houdinis network view.

3. Delete the local_scheduler.

4. Add the node royalScheduler.

5. Start cooking:
   Either click on the first play in in the menu "cook the output nodes work items" or choose "cook output node" from the menu.
   Note: 
   If you want to reset the state, then click on the "x" to abort the cook, then choose "Dirty All" from the menu.
   
   
6. In Houdini, click on the small arrows to expand the task view of each TOP Node:
   
   It takes 1 minute to initialize the data for the first TOP node.
   Then you can see which tasks of a TOP node have been send to Royal Render (•) and which are done (•).
   As well as how many tasks have been send to Royal Render in total (in this case 54) and which are about to be send to Royal Render (In this case 1 )
   
   
   For more information, please see https://www.sidefx.com/docs/houdini/tops/ui.html

7. RR creates 5 jobs:
   2 jobs to start and stop the PDG server.
   3 Jobs, one for each TOP node.
   The state is set to "finished" as they should not start directly.
   
   
8. Please select the TOP job "distributed_flip" or "render_scene" in rrControl.
   The framebar shows you which tasks (in RR: frames) have been allowed by Houdini to process.
   
   
   
   The frames done in RR match the tasks done in Houdini.

Tutorial 2 - Use some other machine as PDG controller

This tutorial is almost the same as tutorial 1.

But instead of using your Houdini session as PDG controller, you let RR start Houdini on some farm machine.

1. Use the scene as prepared in the last tutorial (Step 1-4).
   Note: 
   If you want to reset the state, then click on the "x" to abort any cook, then choose "Dirty All" from the menu.
   
   
2. In the RoyalScheduler, click on "Submit PDG as a Job".
   Royal Render submits one job.
   This jobs starts Houdini to load the scene and cook the PGD tree. 
   IMPORTANT: 
   All Job options that you set in the rrSubmitter UI are applied to this PDG controller job only!
   It does not affect the PDG/TOP jobs that are generated later.
   
   
3. Optional: Close your Houdini interface.
   
   
4. From now on now the same as in tutorial 1, just not on your machine.
   Houdini controls the PDG tree.
   That Houdini session submits 5 new jobs as in tutorial 1.

Tutorial 3 - Mix PDG schedulers

In this case we use a local scheduler for the first TOP node.
The tasks of the distributed_flip.
This is useful if the tasks are executed very fast.
So fast that it is faster to process all tasks of the TOP node on the PDG controlling machine 

than to create a job in Royal Render, let it start Houdini, load the data and process the task.

1. Use the scene as prepared in the first last tutorial (Step 1-4).
   Note: 
   If you want to reset the state, then click on the "x" to abort any cook, then choose "Dirty All" from the menu.
   
   
2. Add a localScheduler node.
   
   
3. Select TOP node distributed_flip.
   Open tab Schedulers.
   Click on the button "Open floating operator chooser" to pick the localScheduler.

4. Select TOP node render_scene.
   Open tab Schedulers.
   Click on the button "Open floating operator chooser" to pick the royalScheduler.
   
   
5. Press the play button to start cooking.

Change rrJob Settings for the TOP task jobs

As the jobs for TOP nodes are not send via the rrSubmitter UI, 

you cannot use the UI to change job settings like a client assignment.
Therefore you have to use the "rrJob Settings" tab in the royalScheduler.

Example:

To change the clients that should run the TOP Tasks to group "myHoudiniClients" and increase the priority to 70, 

please add this string to "Job Settings":

DefaultClientGroup=0~myHoudiniClients; Priority=0~70

If you are searching for the name of a setting, please refer to this page:

How to get an options name?

Note:
It is possible to change the default settings for all PDG TOP jobs as well.
Please create these 2 files:
RR/render_apps/_config_inhouse/3D06__Houdini_PDG_Nodes_Py__inhouse.inc

RR/render_apps/_config_inhouse/3D06__Houdini_PDG_Nodes__inhouse.inc

Then add your settings. One line for each setting.