OpendTect User Documentation version 4.2
Prev		Next

7.5. Batch Processing

7.5.1. Single machine batch processing window

In single mode processing, the user has the option to process the data on a local machine or on a remote host. In the log file, all relevant information on the progress of the calculation will be stored. This is the same information that is displayed in the progress window that appears when using local processing, (See Batch log file).

7.5.2. Multi-machine batch processing window

The multi-machine batch processing window controls on which machines a volume output or steering cube batch job will be processed. Jobs are distributed over the Used hosts on a line-by-line basis (the number of inlines per job can be specified). Hosts can be Added and/or Stopped at all times. Processed results are stored in a Temporary storage directory.

At the end of the processing sequence, OpendTect will merge all processed lines and store the data in the output file that was specified in the Volume output or Create steering cube window, and it will delete the temporary disk files. If for any reason OpendTect fails to perform this data transfer, this can also be done manually in the File - Manage module. The temporary data store appears with a name starting with Proc_. Select this item and copy it to a new cube.

It is possible that at the end of a multi-machine batch job not all data was processed successfully. Some jobs may have failed (e.g. because one of the hosts was temporarily not available). OpendTect will come back with a warning message stating which jobs (i.e. which inlines) have not been processed yet. The user is then advised to re-submit these jobs until all data are processed. The Auto-fill option automatically scans and fills gaps in the processed volume.

The Nice level sets the priority the process gets. With the nice level set on 19 the process has very low priority and other processes that run on the same computer get priority. If the nice level is set to 1 the process gets the highest priority.

The Processes menu allows the user to set the Run, Pause, or Go - Only between options. The Go - Only between option, pauses and runs the processes at user defined times.

Note that OpendTect calls the system utilities of the 'hostent' (sethostent, gethostent, etc.) type to get a table of hosts for the user to select from. How the Operating System builds the lists is dependent on the particular system setup; most likely /etc/hosts and/or the NIS tables are consulted. OpendTect supports multi-threading which means that all processors of multi-processor machines will be used.

Multi-machine processing on Windows OS

The new system works with a Daemon Service running in background on every remote machine to be used for processing. The communication works with TCP/IP and requires some configurations to actually make things working.

Mapping of Survey folder: We assume a DataRoot folder ($DTECT_DATA) to be in a centralised server. This folder has to be mapped on the same drive in every PC to be used as hosts. For example, we have 4 PCs (A,B,C,D). A is a server and the ODData is in drive D. Then map D:ODData on e.g. S: on B, C and D. PCs B, C, and D can then be used as your processing nodes.

OpendTect installation: You need to have OpendTect installation in all hosts, and make sure they all use the same survey at the same time. For example if B is using F3_Demo and want to process something in F3_Demo. then it has to be made sure that the rest of the two PCs also uses the same folder as long as the processing is needed. (will be fixed before the release)

BatchHosts file: Add the IP addresses of C and D in the BatchHosts file inside the application data folder.

Start the daemon: If the User wants to launch process from B to the other two, then B is his local machine, C and D are remote machine. In this case the Daemon service ( odremoteservice ) application has to be launched from binwin folder ( win32/win64 ) only in the remote machines and not in the local machine ( B ). Please note the odremoteservice.exe not to be run directly instead a launching tool will be found in win32mmod_remote_service_manager. Use od_remote_service_manager to launch the daemon which will also add anotification icon to the system tray. Once the service starts, the remote machines are ready.

Start processing: Select the PC’s B, C and D from the list of machines in the multimachine launch window and start processing.

7.5.3. Batch log file

A batch log file is produced for every volume output run. The information is streamed to a file if the batch job is executed on a remote computer. If the processing is done locally the log file is either streamed to a new, dedicated window, or to the standard output window. Every N traces the program will output a symbol to reflect the progress. There are five symbols the program can use. Which symbol it uses depends on the speed in number of traces times N per sec (given at the end of a line in brackets after the cumulative number of traces processed). The symbols indicate the following:

a period (.) means 1 trace processed
a colon (:) means 10 traces processed
an equal sign (=) means 100 traces processed
a pipe marker (|) means 1000 traces processed
an asterisk (*) means 10000 traces processed

The following options are available:

Pressing this icon will stop the process.

This will save the log file.

This icon will bring you to the help menu.

7.5.4. Cluster processing

Batch jobs can be run from OpendTect to cluster management tools. So far dGB has successfully tested SLURM which is easy to install and even easier to use.

Cluster processing is enabled by following this prerequisites:

The environment variable DTECT_CLUSTER_PROC is set to yes before starting OpendTect
The cluster management tool must be up and running. In the case of SLURM it means that the daemon 'slurmctld' should be running on the master computer and that the daemon 'mlurmd' must be running on every node.
This cluster management tool (SLURM) bin directory is included in your PATH variable

Cluster processing is started by choosing "Cluster" in the 'Submit' options menu of the volume processing window. Only 3D attributes can make use of cluster computing so far.

A new window will pop up that will list a number of directories use for the storage of temporary files. The jobs will be split using a user-defined number of inlines. The field named 'Cluster Processing command' represents the name of the binary from the cluster management tool used to run a process.

You can run the "Main script file" (default: ~/yoursurvey/Proc/clusterprocscript) from a command line which will run each job one-by-one using the above command and will also launch the UI to show progress and do post-processing merging of temporary data.

Prev	Home	Next
Generic Mapping Tools (GMT)		Menu - Scenes