Gabriele Cosmo
226 lines
8.9 KiB
Plaintext
226 lines
8.9 KiB
Plaintext
Geant4 Tasking
|
||
|
||
This directory contains a Geant4 run manager which uses a tasking system
|
||
for the G4Event loop. This tasking system is fully compatible with TBB
|
||
if GEANT4_USE_TBB=ON is specified when configuring CMake. The default
|
||
behavior, however, is to submit the tasks to an internal thread-pool and
|
||
task-queue.
|
||
|
||
G4TaskRunManager
|
||
|
||
G4TaskRunManager multiply inherits from G4MTRunManager and
|
||
PTL::TaskRunManager. PTL::TaskRunManager holds the thread-pool instance,
|
||
the size of the thread-pool, and the default task-queue. The constructor
|
||
of G4TaskRunManager takes a G4VUserTaskQueue pointer (can be nullptr), a
|
||
boolean for whether to use TBB if available, and a grainsize.
|
||
|
||
Concepts
|
||
|
||
Grainsize
|
||
|
||
Environment Variable: G4FORCE_GRAINSIZE=N
|
||
|
||
The grainsize is essentially the number of tasks. If set to 0, the
|
||
default grainsize will be poolSize and each thread will get
|
||
numEvents / poolSize events. If the grainsize is set to 1, then all the
|
||
events will be submitted as one task (i.e. be processed serially by one
|
||
thread in the pool). If the grainsize is set to 50 and there are 500
|
||
events, then 50 tasks of 10 events will be submitted.
|
||
|
||
Events Per Tasks
|
||
|
||
Environment Variable: G4FORCE_EVENTS_PER_TASK=N
|
||
|
||
Sometimes is easier to specify the number of events in a task instead of
|
||
the grainsize. If the events-per-task is set to 10 and there are 500
|
||
events, then 50 tasks of 10 events will be submitted.
|
||
|
||
Default Constructor
|
||
|
||
G4TaskRunManager(G4VUserTaskQueue* = nullptr, bool useTBB = false, G4int grainsize = 0);
|
||
|
||
G4RunManagerCreator
|
||
|
||
An enumeration G4RunManagerType and a function
|
||
G4RunManagerCreator::CreateRunManager(...) was added to
|
||
"G4RunManagerCreator.hh" to simplify the selection of the various run
|
||
managers. The first parameter is either one of the enumerated
|
||
G4RunManagerType or a string identifier
|
||
|
||
Enumeration String ID Class
|
||
--------------------------- ----------- ---------------------
|
||
G4RunManagerType::Serial "Serial" G4RunManager
|
||
G4RunManagerType::MT "MT" G4MTRunManager
|
||
G4RunManagerType::Tasking "Tasking" G4TaskRunManager
|
||
G4RunManagerType::TBB "TBB" G4TaskRunManager
|
||
G4RunManagerType::Default "Default" Environment setting
|
||
|
||
The Default enumeration value will defer to the following environment
|
||
variable G4RUN_MANAGER_TYPE if specified and will default to "Tasking"
|
||
if MT is supported and serial if MT is not supported. If the
|
||
G4FORCE_RUN_MANAGER_TYPE environment variable is set, this variable will
|
||
override the value passed to the CreateRunManager function.
|
||
|
||
--------------------------------------------------------------------------------------
|
||
Environment Variable Options Description
|
||
-------------------------- ------------------ ----------------------------------------
|
||
G4RUN_MANAGER_TYPE "Serial", "MT", Only applicable when
|
||
"Tasking", "TBB" G4RunManagerType::Default is used
|
||
|
||
G4FORCE_RUN_MANAGER_TYPE "Serial", "MT", Will override explicitly specifed
|
||
"Tasking", "TBB" G4RunManagerType and fail if type is not
|
||
available
|
||
--------------------------------------------------------------------------------------
|
||
|
||
Creating the G4RunManager
|
||
|
||
- The G4RunManagerCreator::CreateRunManager(...) function takes either
|
||
G4RunManagerType enumerated type or string to specify the desired
|
||
G4RunManager
|
||
- If a string is used, regex matching is used which is
|
||
case-insensitive
|
||
- Returns a G4RunManager*
|
||
- Various overloads exist which just reorder passing in:
|
||
- int numberOfThreads - executes
|
||
G4MTRunManager::SetNumberOfThreads(numberOfThreads) before
|
||
returning if > 0
|
||
- default: 0
|
||
- bool fail_if_unavail - will cause a runtime failure if
|
||
requested type is not available with Geant4 build
|
||
- default: true
|
||
- G4VTaskQueue* - a task-queue manager
|
||
- default: nullptr
|
||
|
||
#include "G4RunManagerCreator.hh"
|
||
|
||
int main()
|
||
{
|
||
// specify {Serial, MT, Tasking, TBB} as the default, can be overridden
|
||
// with "G4FORCE_RUN_MANAGER_TYPE" env variable
|
||
auto* runmanager = G4RunManagerCreator::CreateRunManager(G4RunManagerType::Serial);
|
||
auto* runmanager = G4RunManagerCreator::CreateRunManager(G4RunManagerType::MT);
|
||
auto* runmanager = G4RunManagerCreator::CreateRunManager(G4RunManagerType::Tasking);
|
||
auto* runmanager = G4RunManagerCreator::CreateRunManager(G4RunManagerType::TBB);
|
||
|
||
// defer to "G4RUN_MANAGER_TYPE" env variable and default to Tasking if
|
||
// env variable is not set
|
||
auto* runmanager = G4RunManagerCreator::CreateRunManager(G4RunManagerType::Default);
|
||
|
||
// same as above
|
||
auto* runmanager = G4RunManagerCreator::CreateRunManager();
|
||
}
|
||
|
||
Using the Tasking System
|
||
|
||
With G4TaskRunManager, Geant4 events will be launched asynchronously as
|
||
tasks. These tasks are placed into a queue until one of the thread in
|
||
the pool is available to execute the task. Users can take advantage of
|
||
this system to load-balance expensive sub-event calculations which might
|
||
have previously resulted in serial bottlenecks. For example, if an
|
||
application needs to do extensive event analysis on electrons and thread
|
||
#1 ends up with 10x as many of these events, the other threads might
|
||
finish their G4Run significantly eariler and be idle while thread #1 has
|
||
a lot of work. Tasking allows these analysis calculations to be offload
|
||
back into the queue so that other threads can contribute to their
|
||
completion.
|
||
|
||
Option 1 - Submit Directly to Thread-Pool
|
||
|
||
- To execute the function foo(int, double) asynchronously:
|
||
|
||
// get the task manager
|
||
auto* task_manager = G4TaskRunManager::GetTaskManager();
|
||
|
||
// submit task to thread-pool and receive a future for when the result is need
|
||
std::future<void> _fvoid = task_manager->async<void>(foo, 1, 1.0);
|
||
std::future<int> _fint = task_manager->async<int>(bar, 1.0);
|
||
|
||
// wait for task to execute
|
||
_fvoid.wait();
|
||
_fint.wait();
|
||
|
||
// get the result (if non-void)
|
||
auto result = _fint.get();
|
||
|
||
Option 2 - Submit to task-group
|
||
|
||
- Obtain a pointer to the thread-pool instance
|
||
- Create a task_group<T> object where T is the return type of all the
|
||
functions in the group
|
||
- If T is non-void, you must provide a join functor who return
|
||
type and first argument are both references to the joined type
|
||
and the second argument is type T, e.g. task_group<int> can
|
||
provide a join functor with vector<int>& as the return type and
|
||
T as the second argument or int& as the return and first
|
||
argument and int as the second argument
|
||
- If T is void, the join functor is optional and can be treated as
|
||
a final synchronization operation after all the tasks have been
|
||
completed.
|
||
|
||
NOTE: The join functor for task-groups are called sequentially on the
|
||
thread that is waiting on task_group<T>::join() member function.
|
||
|
||
Global Definitions for Examples
|
||
|
||
// obtain thread-pool instance from task manager
|
||
static auto* thread_pool = G4TaskRunManager::GetThreadPool();
|
||
|
||
// trivial int function which just returns value passed
|
||
int foo(int v) { return v; }
|
||
|
||
// function which launches CUDA kernel
|
||
void bar(int v)
|
||
{
|
||
cuda_bar<<<512, 1>>>(v);
|
||
}
|
||
|
||
Example with non-void return types from tasks
|
||
|
||
// put all return values from tasks into an array
|
||
auto join_vec = [](std::vector<int>& lhs, int rhs) { lhs.push_back(rhs); return lhs; };
|
||
|
||
// sum the values returned by tasks
|
||
auto sum_int = [](int& lhs, int rhs) { return lhs += rhs; };
|
||
|
||
// task group which applies 'join_vec' to all task return values
|
||
task_group<int> vec_tg(join_vec, thread_pool);
|
||
// task group with applies 'sum_int' to all task return values
|
||
task_group<int> sum_tg(sum_int, thread_pool);
|
||
|
||
// submit work to task-groups
|
||
vec_tg.exec(foo, 1);
|
||
vec_tg.exec(foo, 2);
|
||
sum_tg.exec(foo, 1);
|
||
sum_tg.exec(foo, 2);
|
||
|
||
// produces std::vector{ 1, 2 };
|
||
auto vec_result = vec_tg.join();
|
||
|
||
// produces 1 + 2 = 3
|
||
auto sum_result = sum_tg.join();
|
||
|
||
Example with void return type from tasks
|
||
|
||
// wait for the GPU to finish
|
||
auto sync = []() { cudaDeviceSynchronize(); };
|
||
|
||
// task group which applies 'sync' after all tasks have been executed
|
||
task_group<void> gpu_tg(sync, thread_pool);
|
||
// generic task group w/o a join functor
|
||
task_group<void> general_tg(thread_pool);
|
||
|
||
// submit work to task-groups
|
||
gpu_tg.exec(bar, 1);
|
||
gpu_tg.exec(bar, 2);
|
||
general_tg.exec(bar, 1);
|
||
general_tg.exec(bar, 2);
|
||
|
||
// 'sync()' will get called after all tasks in group have executed
|
||
// (i.e. return from 'bar' function). 'sync' will then block until
|
||
// all GPU work has been completed
|
||
gpu_tg.join();
|
||
|
||
// will block only until all tasks in group have been executed
|
||
// (i.e. returned from 'bar' function)
|
||
generic_tg.join();
|