Geant4 Tasking This directory contains a Geant4 run manager which uses a tasking system for the G4Event loop. This tasking system is fully compatible with TBB if GEANT4_USE_TBB=ON is specified when configuring CMake. The default behavior, however, is to submit the tasks to an internal thread-pool and task-queue. G4TaskRunManager G4TaskRunManager multiply inherits from G4MTRunManager and PTL::TaskRunManager. PTL::TaskRunManager holds the thread-pool instance, the size of the thread-pool, and the default task-queue. The constructor of G4TaskRunManager takes a G4VUserTaskQueue pointer (can be nullptr), a boolean for whether to use TBB if available, and a grainsize. Concepts Grainsize Environment Variable: G4FORCE_GRAINSIZE=N The grainsize is essentially the number of tasks. If set to 0, the default grainsize will be poolSize and each thread will get numEvents / poolSize events. If the grainsize is set to 1, then all the events will be submitted as one task (i.e. be processed serially by one thread in the pool). If the grainsize is set to 50 and there are 500 events, then 50 tasks of 10 events will be submitted. Events Per Tasks Environment Variable: G4FORCE_EVENTS_PER_TASK=N Sometimes is easier to specify the number of events in a task instead of the grainsize. If the events-per-task is set to 10 and there are 500 events, then 50 tasks of 10 events will be submitted. Default Constructor G4TaskRunManager(G4VUserTaskQueue* = nullptr, bool useTBB = false, G4int grainsize = 0); G4RunManagerCreator An enumeration G4RunManagerType and a function G4RunManagerCreator::CreateRunManager(...) was added to "G4RunManagerCreator.hh" to simplify the selection of the various run managers. The first parameter is either one of the enumerated G4RunManagerType or a string identifier Enumeration String ID Class --------------------------- ----------- --------------------- G4RunManagerType::Serial "Serial" G4RunManager G4RunManagerType::MT "MT" G4MTRunManager G4RunManagerType::Tasking "Tasking" G4TaskRunManager G4RunManagerType::TBB "TBB" G4TaskRunManager G4RunManagerType::Default "Default" Environment setting The Default enumeration value will defer to the following environment variable G4RUN_MANAGER_TYPE if specified and will default to "Tasking" if MT is supported and serial if MT is not supported. If the G4FORCE_RUN_MANAGER_TYPE environment variable is set, this variable will override the value passed to the CreateRunManager function. -------------------------------------------------------------------------------------- Environment Variable Options Description -------------------------- ------------------ ---------------------------------------- G4RUN_MANAGER_TYPE "Serial", "MT", Only applicable when "Tasking", "TBB" G4RunManagerType::Default is used G4FORCE_RUN_MANAGER_TYPE "Serial", "MT", Will override explicitly specifed "Tasking", "TBB" G4RunManagerType and fail if type is not available -------------------------------------------------------------------------------------- Creating the G4RunManager - The G4RunManagerCreator::CreateRunManager(...) function takes either G4RunManagerType enumerated type or string to specify the desired G4RunManager - If a string is used, regex matching is used which is case-insensitive - Returns a G4RunManager* - Various overloads exist which just reorder passing in: - int numberOfThreads - executes G4MTRunManager::SetNumberOfThreads(numberOfThreads) before returning if > 0 - default: 0 - bool fail_if_unavail - will cause a runtime failure if requested type is not available with Geant4 build - default: true - G4VTaskQueue* - a task-queue manager - default: nullptr #include "G4RunManagerCreator.hh" int main() { // specify {Serial, MT, Tasking, TBB} as the default, can be overridden // with "G4FORCE_RUN_MANAGER_TYPE" env variable auto* runmanager = G4RunManagerCreator::CreateRunManager(G4RunManagerType::Serial); auto* runmanager = G4RunManagerCreator::CreateRunManager(G4RunManagerType::MT); auto* runmanager = G4RunManagerCreator::CreateRunManager(G4RunManagerType::Tasking); auto* runmanager = G4RunManagerCreator::CreateRunManager(G4RunManagerType::TBB); // defer to "G4RUN_MANAGER_TYPE" env variable and default to Tasking if // env variable is not set auto* runmanager = G4RunManagerCreator::CreateRunManager(G4RunManagerType::Default); // same as above auto* runmanager = G4RunManagerCreator::CreateRunManager(); } Using the Tasking System With G4TaskRunManager, Geant4 events will be launched asynchronously as tasks. These tasks are placed into a queue until one of the thread in the pool is available to execute the task. Users can take advantage of this system to load-balance expensive sub-event calculations which might have previously resulted in serial bottlenecks. For example, if an application needs to do extensive event analysis on electrons and thread #1 ends up with 10x as many of these events, the other threads might finish their G4Run significantly eariler and be idle while thread #1 has a lot of work. Tasking allows these analysis calculations to be offload back into the queue so that other threads can contribute to their completion. Option 1 - Submit Directly to Thread-Pool - To execute the function foo(int, double) asynchronously: // get the task manager auto* task_manager = G4TaskRunManager::GetTaskManager(); // submit task to thread-pool and receive a future for when the result is need std::future _fvoid = task_manager->async(foo, 1, 1.0); std::future _fint = task_manager->async(bar, 1.0); // wait for task to execute _fvoid.wait(); _fint.wait(); // get the result (if non-void) auto result = _fint.get(); Option 2 - Submit to task-group - Obtain a pointer to the thread-pool instance - Create a task_group object where T is the return type of all the functions in the group - If T is non-void, you must provide a join functor who return type and first argument are both references to the joined type and the second argument is type T, e.g. task_group can provide a join functor with vector& as the return type and T as the second argument or int& as the return and first argument and int as the second argument - If T is void, the join functor is optional and can be treated as a final synchronization operation after all the tasks have been completed. NOTE: The join functor for task-groups are called sequentially on the thread that is waiting on task_group::join() member function. Global Definitions for Examples // obtain thread-pool instance from task manager static auto* thread_pool = G4TaskRunManager::GetThreadPool(); // trivial int function which just returns value passed int foo(int v) { return v; } // function which launches CUDA kernel void bar(int v) { cuda_bar<<<512, 1>>>(v); } Example with non-void return types from tasks // put all return values from tasks into an array auto join_vec = [](std::vector& lhs, int rhs) { lhs.push_back(rhs); return lhs; }; // sum the values returned by tasks auto sum_int = [](int& lhs, int rhs) { return lhs += rhs; }; // task group which applies 'join_vec' to all task return values task_group vec_tg(join_vec, thread_pool); // task group with applies 'sum_int' to all task return values task_group sum_tg(sum_int, thread_pool); // submit work to task-groups vec_tg.exec(foo, 1); vec_tg.exec(foo, 2); sum_tg.exec(foo, 1); sum_tg.exec(foo, 2); // produces std::vector{ 1, 2 }; auto vec_result = vec_tg.join(); // produces 1 + 2 = 3 auto sum_result = sum_tg.join(); Example with void return type from tasks // wait for the GPU to finish auto sync = []() { cudaDeviceSynchronize(); }; // task group which applies 'sync' after all tasks have been executed task_group gpu_tg(sync, thread_pool); // generic task group w/o a join functor task_group general_tg(thread_pool); // submit work to task-groups gpu_tg.exec(bar, 1); gpu_tg.exec(bar, 2); general_tg.exec(bar, 1); general_tg.exec(bar, 2); // 'sync()' will get called after all tasks in group have executed // (i.e. return from 'bar' function). 'sync' will then block until // all GPU work has been completed gpu_tg.join(); // will block only until all tasks in group have been executed // (i.e. returned from 'bar' function) generic_tg.join();