Import Geant4 11.0.0 source tree
This commit is contained in:
@@ -0,0 +1,124 @@
|
||||
|
||||
///\file "parallel/TopC/ParN02/.README.N02.txt"
|
||||
///\brief Example N02 (in ParN02) README page
|
||||
|
||||
/*! \page ExampleN02InParN02 Example N02 in ParN02
|
||||
|
||||
|
||||
This example simulates a simplified fixe target experiment.
|
||||
Read \link ExampleParN02 Example ParN02 \endlink for a description
|
||||
of how to run it in parallel.
|
||||
|
||||
\section ExampleN02InParN02_s1 GEOMETRY DEFINITION
|
||||
|
||||
The setup consists of a target followed by six chambers of increasing
|
||||
transverse size. These chambers are located in a region called Tracker
|
||||
region. Their shape are boxes, constructed as parametrised volumes
|
||||
(ChamberParametrisation class).
|
||||
|
||||
The default geometry is constructed in DetectorConstruction class.
|
||||
One can change the material of the target and of the chambers
|
||||
interactively via the commands defined in the DetectorMessenger class.
|
||||
|
||||
In addition a transverse uniform magnetic field can be applied (see
|
||||
N02MagneticField and DetectorMessenger classes).
|
||||
|
||||
|
||||
\section ExampleN02InParN02_s2 PHYSICS LIST
|
||||
|
||||
The particle's type and the physic processes which will be available
|
||||
in this example are set in PhysicsList class.
|
||||
|
||||
In this example, all the so called 'electromagnetic processes' are
|
||||
introduced for gamma, charged leptons, and charged hadrons (see the
|
||||
method PhysicsList::ConstructEM()).
|
||||
|
||||
An important data member of this class is the defaultCutValue which
|
||||
defines the production threshold of secondary particles
|
||||
(mainly Ionisation and Bremsstrahlung processes are concerned by this
|
||||
CutValue).
|
||||
Notice that the CutValue must be given in unit of length, corresponding
|
||||
to the stopping range of the particle. It is automatically converted
|
||||
in energy for each material, and a table is printed in the method
|
||||
PhysicsList::SetCuts()
|
||||
|
||||
In addition the build-in interactive command:
|
||||
\verbatim
|
||||
/process/(in)activate processName
|
||||
\endverbatim
|
||||
allows to activate/inactivate the processes one by one.
|
||||
|
||||
|
||||
\section ExampleN02InParN02_s3 RUNS and EVENTS
|
||||
|
||||
The primary kinematic consists of a single particle which hits the
|
||||
target perpendicular to the input face. The type of the particle
|
||||
and its energy are set in the PrimaryGeneratorAction class, and can
|
||||
be changed via the G4 build-in commands of ParticleGun class.
|
||||
|
||||
A RUN is a set of events.
|
||||
|
||||
The user has control:
|
||||
-at Begin and End of each run (class RunAction)
|
||||
-at Begin and End of each event (class EventAction)
|
||||
-at Begin and End of each track (class TrackingAction, not used here)
|
||||
-at End of each step (class SteppingAction)
|
||||
|
||||
The class SteppingVerbose prints some informations step per step,
|
||||
under the control of the command: /tracking/verbose 1
|
||||
It inherits from G4SteppingVerbose, and has been setup here in order
|
||||
to illustrate how to extract informations from the G4 kernel during
|
||||
the tracking of a particle.
|
||||
|
||||
|
||||
\section ExampleN02InParN02_s4 DETECTOR RESPONSE
|
||||
|
||||
A HIT is a record, track per track (even step per step), of all the
|
||||
informations needed to simulate and analyse the detector response.
|
||||
|
||||
In this example the Tracker chambers are considered as the detector.
|
||||
Therefore the chambers are declared 'sensitive detectors' (SD) in
|
||||
the DetectorConstruction class.
|
||||
|
||||
Then, a Hit is defined as a set of 4 informations per step, inside
|
||||
the chambers, namely:
|
||||
- the track identifier (an integer),
|
||||
- the chamber number,
|
||||
- the total energy deposit in this step,
|
||||
- the position of the deposit.
|
||||
|
||||
A given hit is an instance of the class TrackerHit which is created
|
||||
during the tracking of a particle, step by step, in the method
|
||||
TrackerSD::ProcessHits(). This hit is inserted in a HitsCollection.
|
||||
|
||||
The HitsCollection is printed at the end of event (via the method
|
||||
TrackerSD::EndOfEvent()), under the control of the command: /hits/verbose 1
|
||||
|
||||
|
||||
\section ExampleN02InParN02_s5 VISUALIZATION
|
||||
|
||||
The Visualization Manager is set in the main () (see exampleN02.cc).
|
||||
The initialisation of the drawing is done via a set of /vis/ commands
|
||||
in the macro vis.mac. This macro is automatically read from
|
||||
the main when running in interactive mode.
|
||||
|
||||
The tracks are automatically drawn at the end of event and erased at
|
||||
the beginning of the next run.
|
||||
|
||||
The visualization (with OpenGL driver) assumes two things:
|
||||
1- the visualisation & interfaces categories have been compiled
|
||||
with the environment variable G4VIS_BUILD_OPENGLX_DRIVER.
|
||||
2- ParN02.cc has been compiled with G4VIS_USE_OPENGLX.
|
||||
|
||||
(The same with DAWNFILE instead of OPENGLX)
|
||||
|
||||
|
||||
\section ExampleN02InParN02_s6 USER INTERFACES
|
||||
|
||||
The default command interface, called G4UIterminal, is done via
|
||||
standart cin/G4cout.
|
||||
On Linux and Sun-cc on can use a smarter command interface G4UItcsh.
|
||||
It is enough to set the environment variable G4UI_USE_TCSH before
|
||||
compiling ParN02.cc
|
||||
|
||||
*/
|
||||
@@ -0,0 +1,199 @@
|
||||
|
||||
///\file "parallel/TopC/ParN02/.README.txt"
|
||||
///\brief Example ParN02 README page
|
||||
|
||||
/*! \page ExampleParN02 Example ParN02
|
||||
|
||||
ParGeant4: Geant4/TOP-C, a parallelization of Geant4
|
||||
(event-level parallelism)
|
||||
|
||||
\author Gene Cooperman,
|
||||
Northeastern University,
|
||||
gene@ccs.neu.edu
|
||||
|
||||
For the latest information on ParGeant4, see: \n
|
||||
http://www.ccs.neu.edu/home/gene/pargeant4.html
|
||||
|
||||
Note that a version now exists that runs Geant4 over the Grid.
|
||||
Please write to gene@ccs.neu.edu for further information.
|
||||
To port other applications to a parallel version, read the
|
||||
files ../../info/PAR_INSTALL and ../../info/PAR_README.
|
||||
|
||||
<hr>
|
||||
|
||||
See the beginning of GNUmakefile for reasonable `make' targets to run it.
|
||||
To run it: \n
|
||||
-#
|
||||
- a. Follow the standard Geant4 installation procedure.
|
||||
- b. Download and install TOP-C. \n
|
||||
The TOP-C home page is at http://www.ccs.neu.edu/home/gene/topc.html
|
||||
\verbatim
|
||||
cd <TOPC_INSTALL_DIR>
|
||||
gzip -dc topc.tar.gz | tar -xvf -
|
||||
cd topc
|
||||
./configure
|
||||
make
|
||||
make check
|
||||
[ Copy bin/topc-config to your path ]
|
||||
\endverbatim
|
||||
- c. Verify that the Geant4 example installs:
|
||||
\verbatim
|
||||
cd $G4INSTALL/examples/extended/parallel/ParN02
|
||||
make
|
||||
$G4WORKDIR/bin/$G4SYSTEM/ParN02 ParN02.in
|
||||
\endverbatim
|
||||
\n
|
||||
-#
|
||||
\verbatim
|
||||
make run
|
||||
\endverbatim
|
||||
[ By default, the included `procgroup' file creates two slave processes
|
||||
on localhost. ] \n
|
||||
[ Note that in addition to output on master,
|
||||
$G4WORKDIR/bin/$G4SYSTEM/slave*.out contains slave output. ]\n
|
||||
[ To remove intermediate files and start over: make parclean ]
|
||||
\n
|
||||
-#
|
||||
\n
|
||||
Try running it with slave processes on remote processes.
|
||||
First, test that your local environment is set up correctly.
|
||||
Try:
|
||||
\verbatim
|
||||
ssh <REMOTE_HOSTNAME> $G4WORKDIR/bin/$G4SYSTEM/ParN02 `pwd`/ParN02.in
|
||||
\endverbatim
|
||||
The above command needs to work without asking for a password.
|
||||
[ If you use dynamic libraries (*.so), make sure the LD_LIBRARY_PATH
|
||||
in your shell startup file (e.g. ~.tcshrc) includes both:
|
||||
$G4INSTALL/lib/$G4SYSTEM and $CLHEP_BASE_DIR/lib
|
||||
If you use AFS, you may need to type 'klog' to renew your AFS token. ]
|
||||
In `procgroup' file, replace `localhost' by desired remote hosts;
|
||||
Add additional remote hosts (additional slaves) if you like.
|
||||
Then:
|
||||
\verbatim
|
||||
make run
|
||||
\endverbatim
|
||||
|
||||
<hr>
|
||||
|
||||
If you read ParGNUmakefile, you'll find other things that you can
|
||||
modify.
|
||||
- For example, all TOP-C additions are in conditionals:
|
||||
remove -DG4USE_TOPC from ParGNUmakefile and:
|
||||
\verbatim
|
||||
make parclean; make run
|
||||
\endverbatim
|
||||
in order to re-compile and rerun without TOP-C.
|
||||
|
||||
- Define REMOTE_SHELL differently if you don't use `ssh' for a remote shell.
|
||||
(If undefined, ParGNUmakefile defines it to be `ssh')
|
||||
|
||||
- Define MACROFILE diferently to use a different set of input commands.
|
||||
|
||||
- Define MEM_MODEL=--seq
|
||||
to run with TOP-C, but using a single (sequential) process, suitable
|
||||
for easy debugging (via gdb, for example).
|
||||
|
||||
- Try: pushd $G4WORKDIR/bin/$G4SYSTEM/; ./ParN02 --TOPC-help
|
||||
to see TOP-C run-time options that can be invoked, such as
|
||||
pushd $G4WORKDIR/bin/$G4SYSTEM/; ./ParN02 --TOPC-num-slaves=5 ParN02.in
|
||||
|
||||
- Alternatively, modify TOPC_OPTIONS in ParGNUmakefile for the same effect.
|
||||
|
||||
You can also try other targets: make run-debug
|
||||
This will run it under gdb, so you can single step to see what happens.
|
||||
make parclean - Start over with clean set of files.
|
||||
|
||||
<hr>
|
||||
|
||||
New or modified files:
|
||||
\verbatim
|
||||
- ParN02.cc - Adds one line: #include "ParN02.icc"
|
||||
ParExample.icc inserts: #include "topc.h"
|
||||
and causes main to calls TOPC_init, TOPC_finalize,
|
||||
and to use: `new ParRunManager' instead of `new G4RunManager'
|
||||
|
||||
- GNUmakefile - Adds one line at beginning: include ParGNUmakefile
|
||||
ParGNUmakefile defines EXTRALIBS and CPPFLAGS so as to
|
||||
modify behavior of config/binmake.gmk
|
||||
in order to use TOP-C libraries and includes
|
||||
|
||||
- procgroup - Specifies which slave hosts to use, and where to put output
|
||||
For example: localhost 1 - > slave1.out
|
||||
host=`localhost', executable=`same as master',
|
||||
params of slave=`> slave1.out' (redirect output)
|
||||
If output not redirected, it goes to stdout on master.
|
||||
|
||||
- src/ParRunManager.cc - ParRunManger derived from G4RunManager
|
||||
replaces Gr4RunManager::DoEventLoop w/ TOP-C parallel loop,
|
||||
Adds certain local vars of DoEventLoop as ParRunManager members
|
||||
|
||||
- include/MarshaledObj.h - run-time utilities for marshalling
|
||||
- include/MarshaledEx*Hit.h - marshals N02 hits (calorimeter hits)
|
||||
- include/MarshaledG4*.h - marshaling routines for Geant4 data structures
|
||||
|
||||
- ~/slave*.out - Contains outputs of slave1, slave2, etc.
|
||||
Generated each time parallel ParN02 is executed.
|
||||
These files are specified in the file procgroup.
|
||||
\endverbatim
|
||||
|
||||
This version passes an event number to the slave and lets the
|
||||
slave generate the event. The slave passes back marshaled hits to
|
||||
the master.
|
||||
|
||||
I will integrate the track level parallelism into this scenario at
|
||||
a later date. For the track level, I will generate several
|
||||
secondary tracks on the master, and then convert the secondary tracks
|
||||
to new events that can be passed to slaves. I will do this only if
|
||||
I detect that there are not enough initial events to fully occupy all
|
||||
the slaves. This scheme has the drawback that we are splitting an event
|
||||
into many events, which may make the summarization, histogram, and so
|
||||
on more difficult. However, track level parallelism will be triggered
|
||||
only when a very small number of events are generated.
|
||||
|
||||
I also want to support postponing
|
||||
a track to the next event ( G4ClassificationOfNewTrack::fPostpone .
|
||||
To do this, each slave will wait to retire an event until it knows that
|
||||
the previous event has been retired.
|
||||
|
||||
In addition, I plan to have only the master read commands and pass
|
||||
them to the slaves. Currently, the master and slaves each read
|
||||
identical commands.
|
||||
|
||||
<hr>
|
||||
|
||||
If you are curious about some of the layers, the following
|
||||
stack trace [somewhat out of date now] gives some idea.
|
||||
- G4RunManager::BeamOn calls ParRunManager::DoEventLoop
|
||||
(since G4RunManager::DoEventLoop is virtual)
|
||||
- ParRunManager::DoEventLoop calls TOPC_master_slave
|
||||
- TOPC_master_slave calls submit_task_input
|
||||
- submit_task_input eventually calls COMM_send_msg which calls MPI_Send
|
||||
(COMM_send_msg is the communication layer of TOPC;
|
||||
ParN02.cc was linked with the TOP-C MPI communication layer.
|
||||
The same source could have been linked with a POSIX threads layer,
|
||||
a communication layer, or some other communication layer.
|
||||
)
|
||||
- MPI_send calls send
|
||||
(where send is the socket system call of libc.so)
|
||||
<pre>
|
||||
(gdb) where
|
||||
#0 0x41946c62 in send () from /lib/libc.so.6
|
||||
#1 0x400839c1 in send () at wrapsyscall.c:186
|
||||
#2 0x805c547 in MPI_Send (buf=0x82690fc, count=4, datatype=3, dest=2, tag=1, comm=0) at sendrecv.c:236
|
||||
#3 0x805a0b5 in COMM_send_msg (msg=0x82690fc, msg_size=4, dst=2, tag=TASK_INPUT_TAG) at comm-mpi.c:224
|
||||
#4 0x805774e in send_task_input (slave=2, input={data = 0x82690fc, data_size = 4}, tag=TASK_INPUT_TAG) at topc.c:560
|
||||
#5 0x8057aa8 in submit_task_input (input={data = 0x82690fc, data_size = 4}) at topc.c:659
|
||||
#6 0x805813c in TOPC_master_slave (generate_task_input_=0x4003d2e4 <ParRunManager::GenerateEventInput(void)>,
|
||||
do_task_=0x4003d350 <ParRunManager::DoEvent(int *)>, check_task_result_=0x4003d420 <ParRunManager::CheckEventResult(int *, void *)>,
|
||||
update_shared_data_=0) at topc.c:922
|
||||
#7 0x4003d18c in ParRunManager::DoEventLoop (this=0x80c0bf0, n_event=1, macroFile=0x0, n_select=-1) at src/ParRunManager.cc:51
|
||||
#8 0x400b14d1 in G4RunManager::BeamOn () from /afs/cern.ch/user/c/cooperma/scratch-pcitapi07/geant4/lib/libG4run.so
|
||||
#9 0x400b870a in G4RunMessenger::SetNewValue () from /afs/cern.ch/user/c/cooperma/scratch-pcitapi07/geant4/lib/libG4run.so
|
||||
#10 0x4167157b in G4UIcommand::DoIt () from /afs/cern.ch/user/c/cooperma/scratch-pcitapi07/geant4/lib/libG4intercoms.so
|
||||
#11 0x416810a3 in G4UImanager::ApplyCommand () from /afs/cern.ch/user/c/cooperma/scratch-pcitapi07/geant4/lib/libG4intercoms.so
|
||||
#12 0x805db7e in G4UIterminal::ExecuteCommand () at /afs/cern.ch/sw/lhcxx/specific/redhat61/3.2.0/include/CLHEP/Random/Randomize.h:64
|
||||
#13 0x805d42d in G4UIterminal::SessionStart () at /afs/cern.ch/sw/lhcxx/specific/redhat61/3.2.0/include/CLHEP/Random/Randomize.h:64
|
||||
#14 0x8056a3d in main (argc=1, argv=0x80bfa00) at ParN02.cc:98
|
||||
</pre>
|
||||
|
||||
*/
|
||||
@@ -0,0 +1,47 @@
|
||||
The files in this directory are the files that were used to build
|
||||
../include/Marshaled*.hh . The files ../include/Marshaled*.hh contain
|
||||
marshaling (serialization) routines that are used by Parallel Geantr4.
|
||||
|
||||
Marshalgen is a package that allows one to add a small number of annotations
|
||||
to the original sequential code, in order to create marshalling or
|
||||
serialization routines. Those marshalling routines are then used
|
||||
by ParGeant4 to pass data between slave processes and the master process.
|
||||
|
||||
INPUT FILES:
|
||||
1. G4*.hh : The files *.hh are taken from Geant4. They include annotations
|
||||
(comments) that describe how to marshal individual fields of the
|
||||
given classes. These files have additional annotations that describe
|
||||
how to marshal individual fields of the given classes. For a new
|
||||
application, you have to change any references to Ex* to your
|
||||
own example/application include files. These files are then reused
|
||||
in the new Geant4 parallel application.
|
||||
2. ../include/Ex*Hit.hh : These files are the original sequential Geant4
|
||||
application files that describe the application-defined hits.
|
||||
They remain in the application include directory because they may
|
||||
depend on other files in the include directory.
|
||||
For most new parallel Geant4 applications, these files are sufficiently
|
||||
simple, that one needs only to add Marshalgen begin and end comments
|
||||
bracketing the class that needs to be marshalled, and a small number
|
||||
of annotations specifying accessor functions to get and set fields
|
||||
in the class. It may also be necessary to include marshalling
|
||||
include files, if it is necessary to marshal other Geant4 data
|
||||
structures.
|
||||
Information on how to marshal the fields, etc., is in the Marshalgen
|
||||
manual, along with the Marshalgen package:
|
||||
http://www.ccs.neu.edu/home/gene/marshalgen
|
||||
|
||||
|
||||
Marshalgen was then called on the above input files. One calls:
|
||||
./marshalgen *.hh
|
||||
./marshalgen ../include/Ex*Hit.hh
|
||||
|
||||
OUTPUT FILES:
|
||||
3. *.msh : These files are intermediate files generated automatically
|
||||
by marshalgen ( http://www.ccs.neu.edu/home/gene/marshalgen )
|
||||
These files can be deleted if desired.
|
||||
4. Marshaled*.h : These files are generated by Marshalgen. They
|
||||
include of type both MarshaledEx*.h and MarshaledG4*.h .
|
||||
(In addition, the file MarshaledObj.h is copied directly from
|
||||
the Marshalgen distribution.) These files are all copied to
|
||||
the include directory of the Geant4 parallel application.
|
||||
They provide the marshalling functions that are then used by Geant4.
|
||||
@@ -0,0 +1,155 @@
|
||||
|
||||
ParGeant4: Geant4/TOP-C, a parallelization of Geant4
|
||||
(event-level parallelism)
|
||||
|
||||
Gene Cooperman
|
||||
Northeastern University
|
||||
gene@ccs.neu.edu,
|
||||
|
||||
For the latest information on ParGeant4, see:
|
||||
http://www.ccs.neu.edu/home/gene/pargeant4.html
|
||||
Note that a version now exists that runs Geant4 over the Grid.
|
||||
Please write to gene@ccs.neu.edu for further information.
|
||||
To port other applications to a parallel version, read the
|
||||
files ../../info/PAR_INSTALL and ../../info/PAR_README.
|
||||
|
||||
See the beginning of GNUmakefile for reasonable `make' targets to run it.
|
||||
To run it:
|
||||
0. a. Follow the standard Geant4 installation procedure.
|
||||
b. Download and install TOP-C
|
||||
The TOP-C home page is at http://www.ccs.neu.edu/home/gene/topc.html
|
||||
cd <TOPC_INSTALL_DIR>
|
||||
gzip -dc topc.tar.gz | tar -xvf -
|
||||
cd topc
|
||||
./configure
|
||||
make
|
||||
make check
|
||||
[ Copy bin/topc-config to your path ]
|
||||
c. Verify that the Geant4 example installs:
|
||||
cd $G4INSTALL/examples/extended/parallel/ParN02
|
||||
make
|
||||
$G4WORKDIR/bin/$G4SYSTEM/ParN02 ParN02.in
|
||||
2. make run
|
||||
[ By default, the included `procgroup' file creates two slave processes
|
||||
on localhost. ]
|
||||
[ Note that in addition to output on master,
|
||||
$G4WORKDIR/bin/$G4SYSTEM/slave*.out contains slave output. ]
|
||||
[ To remove intermediate files and start over: make parclean ]
|
||||
3. Try running it with slave processes on remote processes.
|
||||
First, test that your local environment is set up correctly.
|
||||
Try:
|
||||
ssh <REMOTE_HOSTNAME> $G4WORKDIR/bin/$G4SYSTEM/ParN02 `pwd`/ParN02.in
|
||||
The above command needs to work without asking for a password.
|
||||
[ If you use dynamic libraries (*.so), make sure the LD_LIBRARY_PATH
|
||||
in your shell startup file (e.g. ~.tcshrc) includes both:
|
||||
$G4INSTALL/lib/$G4SYSTEM and $CLHEP_BASE_DIR/lib
|
||||
If you use AFS, you may need to type 'klog' to renew your AFS token. ]
|
||||
In `procgroup' file, replace `localhost' by desired remote hosts;
|
||||
Add additional remote hosts (additional slaves) if you like.
|
||||
Then: make run
|
||||
|
||||
============================================================================
|
||||
If you read ParGNUmakefile, you'll find other things that you can
|
||||
modify. For example, all TOP-C additions are in conditionals:
|
||||
remove -DG4USE_TOPC from ParGNUmakefile and:
|
||||
make parclean; make run
|
||||
in order to re-compile and rerun without TOP-C.
|
||||
Define REMOTE_SHELL differently if you don't use `ssh' for a remote shell.
|
||||
(If undefined, ParGNUmakefile defines it to be `ssh')
|
||||
Define MACROFILE diferently to use a different set of input commands.
|
||||
Define MEM_MODEL=--seq
|
||||
to run with TOP-C, but using a single (sequential) process, suitable
|
||||
for easy debugging (via gdb, for example).
|
||||
Try: pushd $G4WORKDIR/bin/$G4SYSTEM/; ./ParN02 --TOPC-help
|
||||
to see TOP-C run-time options that can be invoked, such as
|
||||
pushd $G4WORKDIR/bin/$G4SYSTEM/; ./ParN02 --TOPC-num-slaves=5 ParN02.in
|
||||
Alternatively, modify TOPC_OPTIONS in ParGNUmakefile for the same effect.
|
||||
|
||||
You can also try other targets: make run-debug
|
||||
This will run it under gdb, so you can single step to see what happens.
|
||||
make parclean - Start over with clean set of files.
|
||||
|
||||
============================================================================
|
||||
New or modified files:
|
||||
ParN02.cc - Adds one line: #include "ParN02.icc"
|
||||
ParExample.icc inserts: #include "topc.h"
|
||||
and causes main to calls TOPC_init, TOPC_finalize,
|
||||
and to use: `new ParRunManager' instead of `new G4RunManager'
|
||||
GNUmakefile - Adds one line at beginning: include ParGNUmakefile
|
||||
ParGNUmakefile defines EXTRALIBS and CPPFLAGS so as to
|
||||
modify behavior of config/binmake.gmk
|
||||
in order to use TOP-C libraries and includes
|
||||
procgroup - Specifies which slave hosts to use, and where to put output
|
||||
For example: localhost 1 - > slave1.out
|
||||
host=`localhost', executable=`same as master',
|
||||
params of slave=`> slave1.out' (redirect output)
|
||||
If output not redirected, it goes to stdout on master.
|
||||
src/ParRunManager.cc - ParRunManger derived from G4RunManager
|
||||
replaces Gr4RunManager::DoEventLoop w/ TOP-C parallel loop,
|
||||
Adds certain local vars of DoEventLoop as ParRunManager members
|
||||
include/MarshaledObj.h - run-time utilities for marshalling
|
||||
include/MarshaledEx*Hit.h - marshals N02 hits (calorimeter hits)
|
||||
include/MarshaledG4*.h - marshaling routines for Geant4 data structures
|
||||
|
||||
~/slave*.out - Contains outputs of slave1, slave2, etc.
|
||||
Generated each time parallel ParN02 is executed.
|
||||
These files are specified in the file procgroup.
|
||||
|
||||
====================================================================
|
||||
This version passes an event number to the slave and lets the
|
||||
slave generate the event. The slave passes back marshaled hits to
|
||||
the master.
|
||||
|
||||
I will integrate the track level parallelism into this scenario at
|
||||
a later date. For the track level, I will generate several
|
||||
secondary tracks on the master, and then convert the secondary tracks
|
||||
to new events that can be passed to slaves. I will do this only if
|
||||
I detect that there are not enough initial events to fully occupy all
|
||||
the slaves. This scheme has the drawback that we are splitting an event
|
||||
into many events, which may make the summarization, histogram, and so
|
||||
on more difficult. However, track level parallelism will be triggered
|
||||
only when a very small number of events are generated.
|
||||
|
||||
I also want to support postponing
|
||||
a track to the next event ( G4ClassificationOfNewTrack::fPostpone .
|
||||
To do this, each slave will wait to retire an event until it knows that
|
||||
the previous event has been retired.
|
||||
|
||||
In addition, I plan to have only the master read commands and pass
|
||||
them to the slaves. Currently, the master and slaves each read
|
||||
identical commands.
|
||||
|
||||
====================================================================
|
||||
If you are curious about some of the layers, the following
|
||||
stack trace [somewhat out of date now] gives some idea.
|
||||
G4RunManager::BeamOn calls ParRunManager::DoEventLoop
|
||||
(since G4RunManager::DoEventLoop is virtual)
|
||||
ParRunManager::DoEventLoop calls TOPC_master_slave
|
||||
TOPC_master_slave calls submit_task_input
|
||||
submit_task_input eventually calls COMM_send_msg which calls MPI_Send
|
||||
(COMM_send_msg is the communication layer of TOPC;
|
||||
ParN02.cc was linked with the TOP-C MPI communication layer.
|
||||
The same source could have been linked with a POSIX threads layer,
|
||||
a communication layer, or some other communication layer.
|
||||
)
|
||||
MPI_send calls send
|
||||
(where send is the socket system call of libc.so)
|
||||
|
||||
(gdb) where
|
||||
#0 0x41946c62 in send () from /lib/libc.so.6
|
||||
#1 0x400839c1 in send () at wrapsyscall.c:186
|
||||
#2 0x805c547 in MPI_Send (buf=0x82690fc, count=4, datatype=3, dest=2, tag=1, comm=0) at sendrecv.c:236
|
||||
#3 0x805a0b5 in COMM_send_msg (msg=0x82690fc, msg_size=4, dst=2, tag=TASK_INPUT_TAG) at comm-mpi.c:224
|
||||
#4 0x805774e in send_task_input (slave=2, input={data = 0x82690fc, data_size = 4}, tag=TASK_INPUT_TAG) at topc.c:560
|
||||
#5 0x8057aa8 in submit_task_input (input={data = 0x82690fc, data_size = 4}) at topc.c:659
|
||||
#6 0x805813c in TOPC_master_slave (generate_task_input_=0x4003d2e4 <ParRunManager::GenerateEventInput(void)>,
|
||||
do_task_=0x4003d350 <ParRunManager::DoEvent(int *)>, check_task_result_=0x4003d420 <ParRunManager::CheckEventResult(int *, void *)>,
|
||||
update_shared_data_=0) at topc.c:922
|
||||
#7 0x4003d18c in ParRunManager::DoEventLoop (this=0x80c0bf0, n_event=1, macroFile=0x0, n_select=-1) at src/ParRunManager.cc:51
|
||||
#8 0x400b14d1 in G4RunManager::BeamOn () from /afs/cern.ch/user/c/cooperma/scratch-pcitapi07/geant4/lib/libG4run.so
|
||||
#9 0x400b870a in G4RunMessenger::SetNewValue () from /afs/cern.ch/user/c/cooperma/scratch-pcitapi07/geant4/lib/libG4run.so
|
||||
#10 0x4167157b in G4UIcommand::DoIt () from /afs/cern.ch/user/c/cooperma/scratch-pcitapi07/geant4/lib/libG4intercoms.so
|
||||
#11 0x416810a3 in G4UImanager::ApplyCommand () from /afs/cern.ch/user/c/cooperma/scratch-pcitapi07/geant4/lib/libG4intercoms.so
|
||||
#12 0x805db7e in G4UIterminal::ExecuteCommand () at /afs/cern.ch/sw/lhcxx/specific/redhat61/3.2.0/include/CLHEP/Random/Randomize.h:64
|
||||
#13 0x805d42d in G4UIterminal::SessionStart () at /afs/cern.ch/sw/lhcxx/specific/redhat61/3.2.0/include/CLHEP/Random/Randomize.h:64
|
||||
#14 0x8056a3d in main (argc=1, argv=0x80bfa00) at ParN02.cc:98
|
||||
@@ -0,0 +1,123 @@
|
||||
-------------------------------------------------------------------
|
||||
|
||||
=========================================================
|
||||
Geant4 - an Object-Oriented Toolkit for Simulation in HEP
|
||||
=========================================================
|
||||
|
||||
ParN02
|
||||
------
|
||||
|
||||
|
||||
This example simulates a simplified fixe target experiment.
|
||||
Read README for a description of how to run it in parallel.
|
||||
|
||||
1- GEOMETRY DEFINITION
|
||||
|
||||
The setup consists of a target followed by six chambers of increasing
|
||||
transverse size. These chambers are located in a region called Tracker
|
||||
region. Their shape are boxes, constructed as parametrised volumes
|
||||
(ChamberParametrisation class).
|
||||
|
||||
The default geometry is constructed in DetectorConstruction class.
|
||||
One can change the material of the target and of the chambers
|
||||
interactively via the commands defined in the DetectorMessenger class.
|
||||
|
||||
In addition a transverse uniform magnetic field can be applied (see
|
||||
N02MagneticField and DetectorMessenger classes).
|
||||
|
||||
|
||||
2- PHYSICS LIST
|
||||
|
||||
The particle's type and the physic processes which will be available
|
||||
in this example are set in PhysicsList class.
|
||||
|
||||
In this example, all the so called 'electromagnetic processes' are
|
||||
introduced for gamma, charged leptons, and charged hadrons (see the
|
||||
method PhysicsList::ConstructEM()).
|
||||
|
||||
An important data member of this class is the defaultCutValue which
|
||||
defines the production threshold of secondary particles
|
||||
(mainly Ionisation and Bremsstrahlung processes are concerned by this
|
||||
CutValue).
|
||||
Notice that the CutValue must be given in unit of length, corresponding
|
||||
to the stopping range of the particle. It is automatically converted
|
||||
in energy for each material, and a table is printed in the method
|
||||
PhysicsList::SetCuts()
|
||||
|
||||
In addition the build-in interactive command:
|
||||
/process/(in)activate processName
|
||||
allows to activate/inactivate the processes one by one.
|
||||
|
||||
|
||||
3- RUNS and EVENTS
|
||||
|
||||
The primary kinematic consists of a single particle which hits the
|
||||
target perpendicular to the input face. The type of the particle
|
||||
and its energy are set in the PrimaryGeneratorAction class, and can
|
||||
be changed via the G4 build-in commands of ParticleGun class.
|
||||
|
||||
A RUN is a set of events.
|
||||
|
||||
The user has control:
|
||||
-at Begin and End of each run (class RunAction)
|
||||
-at Begin and End of each event (class EventAction)
|
||||
-at Begin and End of each track (class TrackingAction, not used here)
|
||||
-at End of each step (class SteppingAction)
|
||||
|
||||
The class SteppingVerbose prints some informations step per step,
|
||||
under the control of the command: /tracking/verbose 1
|
||||
It inherits from G4SteppingVerbose, and has been setup here in order
|
||||
to illustrate how to extract informations from the G4 kernel during
|
||||
the tracking of a particle.
|
||||
|
||||
|
||||
4- DETECTOR RESPONSE
|
||||
|
||||
A HIT is a record, track per track (even step per step), of all the
|
||||
informations needed to simulate and analyse the detector response.
|
||||
|
||||
In this example the Tracker chambers are considered as the detector.
|
||||
Therefore the chambers are declared 'sensitive detectors' (SD) in
|
||||
the DetectorConstruction class.
|
||||
|
||||
Then, a Hit is defined as a set of 4 informations per step, inside
|
||||
the chambers, namely:
|
||||
- the track identifier (an integer),
|
||||
- the chamber number,
|
||||
- the total energy deposit in this step,
|
||||
- the position of the deposit.
|
||||
|
||||
A given hit is an instance of the class TrackerHit which is created
|
||||
during the tracking of a particle, step by step, in the method
|
||||
TrackerSD::ProcessHits(). This hit is inserted in a HitsCollection.
|
||||
|
||||
The HitsCollection is printed at the end of event (via the method
|
||||
TrackerSD::EndOfEvent()), under the control of the command: /hits/verbose 1
|
||||
|
||||
|
||||
5- VISUALIZATION
|
||||
|
||||
The Visualization Manager is set in the main().
|
||||
The initialisation of the drawing is done via a set of /vis/ commands
|
||||
in the macro vis.mac. This macro is automatically read from
|
||||
the main when running in interactive mode.
|
||||
|
||||
The tracks are automatically drawn at the end of event and erased at
|
||||
the beginning of the next run.
|
||||
|
||||
The visualization (with OpenGL driver) assumes two things:
|
||||
1- the visualisation & interfaces categories have been compiled
|
||||
with the environment variable G4VIS_BUILD_OPENGLX_DRIVER.
|
||||
2- ParN02.cc has been compiled with G4VIS_USE_OPENGLX.
|
||||
|
||||
(The same with DAWNFILE instead of OPENGLX)
|
||||
|
||||
|
||||
6- USER INTERFACES
|
||||
|
||||
The default command interface, called G4UIterminal, is done via
|
||||
standart cin/G4cout.
|
||||
On Linux and Sun-cc on can use a smarter command interface G4UItcsh.
|
||||
It is enough to set the environment variable G4UI_USE_TCSH before
|
||||
compiling ParN02.cc
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
|
||||
///\file "parallel/TopC/ParN04/.README.N04"
|
||||
///\brief Example N04 (in ParN04) README page
|
||||
|
||||
/*! \page ExampleN04InParN04 Example N04 in ParN04
|
||||
|
||||
ParN04 has a simplified collider detector geometry.
|
||||
This example demonstrates the following features.
|
||||
[ Read \link ExampleParN02 Example ParN02 \endlink for a description
|
||||
of how to run it in parallel. ]
|
||||
|
||||
\section ExampleN04InParN04_s1 PYTHIA primary events.
|
||||
|
||||
ExN04PrimaryGeneratorAction has G4HEPEvtInterface as the generator.
|
||||
G4HEPEvtInterface accesses to "pythia_event.data", which contains three
|
||||
events of Higgs generation produced by PYTHIA. "pythia_main.f" is an
|
||||
example FORTRAN code of PYTHIA for generating this event sample.
|
||||
|
||||
\section ExampleN04InParN04_s2 Readout geometry
|
||||
|
||||
ExN04DetectorConstruction defines a simplified collider detecor
|
||||
geometry, tracker made of cylindrical tubes, calorimeter made of
|
||||
cylindrical tubes, and muon trackers made of planes.
|
||||
Cylindrical calorimeter is made of tubes of lead and scintirator
|
||||
without cut in phi nor z direction. Energy deposition in scintirator
|
||||
is accumurated by ExN04CalorimeterSD sensitive detector, which has
|
||||
a readout geometry to find the phi-z cell.
|
||||
|
||||
\section ExampleN04InParN04_s3 Full set of "ordinary" physics processes
|
||||
|
||||
ExN04PhysicsList defines almost all of leptons and hadrons which
|
||||
Geant4 has dedicated classes for. Also almost all physics processes
|
||||
Geant4 has are defined.
|
||||
|
||||
\section ExampleN04InParN04_s4 Event filtering by the stacking mechanism.
|
||||
|
||||
Higgs events in "pythia_event.data" have two lepton pairs produced
|
||||
by the Higgs decay via Z0. At the first stage of each event, only the
|
||||
primary muons are tracked without tracking secondaries. then the number
|
||||
of hits on the muon trackers are examined. At the next stage, only
|
||||
the primary charged particles are tracked only inside the barrel
|
||||
tracking area and the isolation of the primary muons are examined.
|
||||
At the third stage, all particles in the RoI (Region of Interest) along
|
||||
the isolated muons are tracked. All these examinations are applied in
|
||||
ExN04StackingAction.
|
||||
|
||||
*/
|
||||
@@ -0,0 +1,47 @@
|
||||
|
||||
///\file "parallel/TopC/ParN04/.README.N04.txt"
|
||||
///\brief Example N04 (in ParN04) README page
|
||||
|
||||
/*! \page ExampleN04InParN04 Example N04 in ParN04
|
||||
|
||||
ParN04 has a simplified collider detector geometry.
|
||||
This example demonstrates the following features.
|
||||
[ Read \link ExampleParN04 Example ParN02 \endlink for a description
|
||||
of how to run it in parallel. ]
|
||||
|
||||
\section ExampleN04InParN04_s1 PYTHIA primary events.
|
||||
|
||||
ExN04PrimaryGeneratorAction has G4HEPEvtInterface as the generator.
|
||||
G4HEPEvtInterface accesses to "pythia_event.data", which contains three
|
||||
events of Higgs generation produced by PYTHIA. "pythia_main.f" is an
|
||||
example FORTRAN code of PYTHIA for generating this event sample.
|
||||
|
||||
\section ExampleN04InParN04_s2 Readout geometry
|
||||
|
||||
ExN04DetectorConstruction defines a simplified collider detecor
|
||||
geometry, tracker made of cylindrical tubes, calorimeter made of
|
||||
cylindrical tubes, and muon trackers made of planes.
|
||||
Cylindrical calorimeter is made of tubes of lead and scintirator
|
||||
without cut in phi nor z direction. Energy deposition in scintirator
|
||||
is accumurated by ExN04CalorimeterSD sensitive detector, which has
|
||||
a readout geometry to find the phi-z cell.
|
||||
|
||||
\section ExampleN04InParN04_s3 Full set of "ordinary" physics processes
|
||||
|
||||
ExN04PhysicsList defines almost all of leptons and hadrons which
|
||||
Geant4 has dedicated classes for. Also almost all physics processes
|
||||
Geant4 has are defined.
|
||||
|
||||
\section ExampleN04InParN04_s4 Event filtering by the stacking mechanism.
|
||||
|
||||
Higgs events in "pythia_event.data" have two lepton pairs produced
|
||||
by the Higgs decay via Z0. At the first stage of each event, only the
|
||||
primary muons are tracked without tracking secondaries. then the number
|
||||
of hits on the muon trackers are examined. At the next stage, only
|
||||
the primary charged particles are tracked only inside the barrel
|
||||
tracking area and the isolation of the primary muons are examined.
|
||||
At the third stage, all particles in the RoI (Region of Interest) along
|
||||
the isolated muons are tracked. All these examinations are applied in
|
||||
ExN04StackingAction.
|
||||
|
||||
*/
|
||||
@@ -0,0 +1,200 @@
|
||||
|
||||
///\file "parallel/TopC/ParN04/.README.txt"
|
||||
///\brief Example ParN04 README page
|
||||
|
||||
/*! \page ExampleParN04 Example ParN04
|
||||
|
||||
ParGeant4: Geant4/TOP-C, a parallelization of Geant4
|
||||
(event-level parallelism)
|
||||
|
||||
\author Gene Cooperman,
|
||||
Northeastern University,
|
||||
gene@ccs.neu.edu
|
||||
|
||||
For the latest information on ParGeant4, see: \n
|
||||
http://www.ccs.neu.edu/home/gene/pargeant4.html
|
||||
|
||||
Note that a version now exists that runs Geant4 over the Grid.
|
||||
Please write to gene@ccs.neu.edu for further information.
|
||||
To port other applications to a parallel version, read the
|
||||
files ../../info/PAR_INSTALL and ../../info/PAR_README.
|
||||
|
||||
<hr>
|
||||
|
||||
See the beginning of GNUmakefile for reasonable `make' targets to run it.
|
||||
To run it: \n
|
||||
-#
|
||||
- a. Follow the standard Geant4 installation procedure.
|
||||
- b. Download and install TOP-C. \n
|
||||
The TOP-C home page is at http://www.ccs.neu.edu/home/gene/topc.html
|
||||
\verbatim
|
||||
cd <TOPC_INSTALL_DIR>
|
||||
gzip -dc topc.tar.gz | tar -xvf -
|
||||
cd topc
|
||||
./configure
|
||||
make
|
||||
make check
|
||||
[ Copy bin/topc-config to your path ]
|
||||
\endverbatim
|
||||
- c. Verify that the Geant4 example installs:
|
||||
\verbatim
|
||||
cd $G4INSTALL/examples/extended/parallel/ParN04
|
||||
make
|
||||
$G4WORKDIR/bin/$G4SYSTEM/ParN04 ParN04.in
|
||||
\endverbatim
|
||||
\n
|
||||
-#
|
||||
\verbatim
|
||||
make run
|
||||
\endverbatim
|
||||
[ By default, the included `procgroup' file creates two slave processes
|
||||
on localhost. ] \n
|
||||
[ Note that in addition to output on master,
|
||||
$G4WORKDIR/bin/$G4SYSTEM/slave*.out contains slave output. ]\n
|
||||
[ To remove intermediate files and start over: make parclean ]
|
||||
\n
|
||||
-#
|
||||
\n
|
||||
Try running it with slave processes on remote processes.
|
||||
First, test that your local environment is set up correctly.
|
||||
Try:
|
||||
\verbatim
|
||||
ssh <REMOTE_HOSTNAME> $G4WORKDIR/bin/$G4SYSTEM/ParN04 `pwd`/ParN04.in
|
||||
\endverbatim
|
||||
The above command needs to work without asking for a password.
|
||||
[ If you use dynamic libraries (*.so), make sure the LD_LIBRARY_PATH
|
||||
in your shell startup file (e.g. ~.tcshrc) includes both:
|
||||
$G4INSTALL/lib/$G4SYSTEM and $CLHEP_BASE_DIR/lib
|
||||
If you use AFS, you may need to type 'klog' to renew your AFS token. ]
|
||||
In `procgroup' file, replace `localhost' by desired remote hosts;
|
||||
Add additional remote hosts (additional slaves) if you like.
|
||||
Then:
|
||||
\verbatim
|
||||
make run
|
||||
\endverbatim
|
||||
|
||||
<hr>
|
||||
|
||||
If you read ParGNUmakefile, you'll find other things that you can
|
||||
modify.
|
||||
- For example, all TOP-C additions are in conditionals:
|
||||
remove -DG4USE_TOPC from ParGNUmakefile and:
|
||||
\verbatim
|
||||
make parclean; make run
|
||||
\endverbatim
|
||||
in order to re-compile and rerun without TOP-C.
|
||||
|
||||
- Define REMOTE_SHELL differently if you don't use `ssh' for a remote shell.
|
||||
(If undefined, ParGNUmakefile defines it to be `ssh')
|
||||
|
||||
- Define MACROFILE diferently to use a different set of input commands.
|
||||
|
||||
- Define MEM_MODEL=--seq
|
||||
to run with TOP-C, but using a single (sequential) process, suitable
|
||||
for easy debugging (via gdb, for example).
|
||||
|
||||
- Try: pushd $G4WORKDIR/bin/$G4SYSTEM/; ./ParN04 --TOPC-help
|
||||
to see TOP-C run-time options that can be invoked, such as
|
||||
pushd $G4WORKDIR/bin/$G4SYSTEM/; ./ParN04 --TOPC-num-slaves=5 ParN04.in
|
||||
|
||||
- Alternatively, modify TOPC_OPTIONS in ParGNUmakefile for the same effect.
|
||||
|
||||
You can also try other targets: make run-debug
|
||||
This will run it under gdb, so you can single step to see what happens.
|
||||
make parclean - Start over with clean set of files.
|
||||
|
||||
<hr>
|
||||
|
||||
New or modified files:
|
||||
\verbatim
|
||||
- ParN04.cc - Adds one line: #include "ParN04.icc"
|
||||
ParExample.icc inserts: #include "topc.h"
|
||||
and causes main to calls TOPC_init, TOPC_finalize,
|
||||
and to use: `new ParRunManager' instead of `new G4RunManager'
|
||||
|
||||
- GNUmakefile - Adds one line at beginning: include ParGNUmakefile
|
||||
ParGNUmakefile defines EXTRALIBS and CPPFLAGS so as to
|
||||
modify behavior of config/binmake.gmk
|
||||
in order to use TOP-C libraries and includes
|
||||
|
||||
- procgroup - Specifies which slave hosts to use, and where to put output
|
||||
For example: localhost 1 - > slave1.out
|
||||
host=`localhost', executable=`same as master',
|
||||
params of slave=`> slave1.out' (redirect output)
|
||||
If output not redirected, it goes to stdout on master.
|
||||
|
||||
- src/ParRunManager.cc - ParRunManger derived from G4RunManager
|
||||
replaces Gr4RunManager::DoEventLoop w/ TOP-C parallel loop,
|
||||
Adds certain local vars of DoEventLoop as ParRunManager members
|
||||
|
||||
- include/MarshaledObj.h - run-time utilities for marshalling
|
||||
- include/MarshaledEx*Hit.h - marshals N04 hits (calorimeter hits)
|
||||
- include/MarshaledG4*.h - marshaling routines for Geant4 data structures
|
||||
|
||||
- ~/slave*.out - Contains outputs of slave1, slave2, etc.
|
||||
Generated each time parallel ParN04 is executed.
|
||||
These files are specified in the file procgroup.
|
||||
|
||||
\endverbatim
|
||||
|
||||
This version passes an event number to the slave and lets the
|
||||
slave generate the event. The slave passes back marshaled hits to
|
||||
the master.
|
||||
|
||||
I will integrate the track level parallelism into this scenario at
|
||||
a later date. For the track level, I will generate several
|
||||
secondary tracks on the master, and then convert the secondary tracks
|
||||
to new events that can be passed to slaves. I will do this only if
|
||||
I detect that there are not enough initial events to fully occupy all
|
||||
the slaves. This scheme has the drawback that we are splitting an event
|
||||
into many events, which may make the summarization, histogram, and so
|
||||
on more difficult. However, track level parallelism will be triggered
|
||||
only when a very small number of events are generated.
|
||||
|
||||
I also want to support postponing
|
||||
a track to the next event ( G4ClassificationOfNewTrack::fPostpone .
|
||||
To do this, each slave will wait to retire an event until it knows that
|
||||
the previous event has been retired.
|
||||
|
||||
In addition, I plan to have only the master read commands and pass
|
||||
them to the slaves. Currently, the master and slaves each read
|
||||
identical commands.
|
||||
|
||||
<hr>
|
||||
|
||||
If you are curious about some of the layers, the following
|
||||
stack trace [somewhat out of date now] gives some idea.
|
||||
- G4RunManager::BeamOn calls ParRunManager::DoEventLoop
|
||||
(since G4RunManager::DoEventLoop is virtual)
|
||||
- ParRunManager::DoEventLoop calls TOPC_master_slave
|
||||
- TOPC_master_slave calls submit_task_input
|
||||
- submit_task_input eventually calls COMM_send_msg which calls MPI_Send
|
||||
(COMM_send_msg is the communication layer of TOPC;
|
||||
ParN04.cc was linked with the TOP-C MPI communication layer.
|
||||
The same source could have been linked with a POSIX threads layer,
|
||||
a communication layer, or some other communication layer.
|
||||
)
|
||||
- MPI_send calls send
|
||||
(where send is the socket system call of libc.so)
|
||||
<pre>
|
||||
(gdb) where
|
||||
#0 0x41946c62 in send () from /lib/libc.so.6
|
||||
#1 0x400839c1 in send () at wrapsyscall.c:186
|
||||
#2 0x805c547 in MPI_Send (buf=0x82690fc, count=4, datatype=3, dest=2, tag=1, comm=0) at sendrecv.c:236
|
||||
#3 0x805a0b5 in COMM_send_msg (msg=0x82690fc, msg_size=4, dst=2, tag=TASK_INPUT_TAG) at comm-mpi.c:224
|
||||
#4 0x805774e in send_task_input (slave=2, input={data = 0x82690fc, data_size = 4}, tag=TASK_INPUT_TAG) at topc.c:560
|
||||
#5 0x8057aa8 in submit_task_input (input={data = 0x82690fc, data_size = 4}) at topc.c:659
|
||||
#6 0x805813c in TOPC_master_slave (generate_task_input_=0x4003d2e4 <ParRunManager::GenerateEventInput(void)>,
|
||||
do_task_=0x4003d350 <ParRunManager::DoEvent(int *)>, check_task_result_=0x4003d420 <ParRunManager::CheckEventResult(int *, void *)>,
|
||||
update_shared_data_=0) at topc.c:922
|
||||
#7 0x4003d18c in ParRunManager::DoEventLoop (this=0x80c0bf0, n_event=1, macroFile=0x0, n_select=-1) at src/ParRunManager.cc:51
|
||||
#8 0x400b14d1 in G4RunManager::BeamOn () from /afs/cern.ch/user/c/cooperma/scratch-pcitapi07/geant4/lib/libG4run.so
|
||||
#9 0x400b870a in G4RunMessenger::SetNewValue () from /afs/cern.ch/user/c/cooperma/scratch-pcitapi07/geant4/lib/libG4run.so
|
||||
#10 0x4167157b in G4UIcommand::DoIt () from /afs/cern.ch/user/c/cooperma/scratch-pcitapi07/geant4/lib/libG4intercoms.so
|
||||
#11 0x416810a3 in G4UImanager::ApplyCommand () from /afs/cern.ch/user/c/cooperma/scratch-pcitapi07/geant4/lib/libG4intercoms.so
|
||||
#12 0x805db7e in G4UIterminal::ExecuteCommand () at /afs/cern.ch/sw/lhcxx/specific/redhat61/3.2.0/include/CLHEP/Random/Randomize.h:64
|
||||
#13 0x805d42d in G4UIterminal::SessionStart () at /afs/cern.ch/sw/lhcxx/specific/redhat61/3.2.0/include/CLHEP/Random/Randomize.h:64
|
||||
#14 0x8056a3d in main (argc=1, argv=0x80bfa00) at ParN02.cc:98
|
||||
</pre>
|
||||
|
||||
*/
|
||||
@@ -0,0 +1,47 @@
|
||||
The files in this directory are the files that were used to build
|
||||
../include/Marshaled*.hh . The files ../include/Marshaled*.hh contain
|
||||
marshaling (serialization) routines that are used by Parallel Geantr4.
|
||||
|
||||
Marshalgen is a package that allows one to add a small number of annotations
|
||||
to the original sequential code, in order to create marshalling or
|
||||
serialization routines. Those marshalling routines are then used
|
||||
by ParGeant4 to pass data between slave processes and the master process.
|
||||
|
||||
INPUT FILES:
|
||||
1. G4*.hh : The files *.hh are taken from Geant4. They include annotations
|
||||
(comments) that describe how to marshal individual fields of the
|
||||
given classes. These files have additional annotations that describe
|
||||
how to marshal individual fields of the given classes. For a new
|
||||
application, you have to change any references to Ex* to your
|
||||
own example/application include files. These files are then reused
|
||||
in the new Geant4 parallel application.
|
||||
2. ../include/Ex*Hit.hh : These files are the original sequential Geant4
|
||||
application files that describe the application-defined hits.
|
||||
They remain in the application include directory because they may
|
||||
depend on other files in the include directory.
|
||||
For most new parallel Geant4 applications, these files are sufficiently
|
||||
simple, that one needs only to add Marshalgen begin and end comments
|
||||
bracketing the class that needs to be marshalled, and a small number
|
||||
of annotations specifying accessor functions to get and set fields
|
||||
in the class. It may also be necessary to include marshalling
|
||||
include files, if it is necessary to marshal other Geant4 data
|
||||
structures.
|
||||
Information on how to marshal the fields, etc., is in the Marshalgen
|
||||
manual, along with the Marshalgen package:
|
||||
http://www.ccs.neu.edu/home/gene/marshalgen
|
||||
|
||||
|
||||
Marshalgen was then called on the above input files. One calls:
|
||||
./marshalgen *.hh
|
||||
./marshalgen ../include/Ex*Hit.hh
|
||||
|
||||
OUTPUT FILES:
|
||||
3. *.msh : These files are intermediate files generated automatically
|
||||
by marshalgen ( http://www.ccs.neu.edu/home/gene/marshalgen )
|
||||
These files can be deleted if desired.
|
||||
4. Marshaled*.h : These files are generated by Marshalgen. They
|
||||
include of type both MarshaledEx*.h and MarshaledG4*.h .
|
||||
(In addition, the file MarshaledObj.h is copied directly from
|
||||
the Marshalgen distribution.) These files are all copied to
|
||||
the include directory of the Geant4 parallel application.
|
||||
They provide the marshalling functions that are then used by Geant4.
|
||||
@@ -0,0 +1,157 @@
|
||||
|
||||
ParGeant4: Geant4/TOP-C, a parallelization of Geant4
|
||||
(event-level parallelism)
|
||||
|
||||
Gene Cooperman
|
||||
Northeastern University
|
||||
gene@ccs.neu.edu,
|
||||
|
||||
For the latest information on ParGeant4, see:
|
||||
http://www.ccs.neu.edu/home/gene/pargeant4.html
|
||||
Note that a version now exists that runs Geant4 over the Grid.
|
||||
Please write to gene@ccs.neu.edu for further information.
|
||||
To port other applications to a parallel version, read the
|
||||
files ../../info/PAR_INSTALL and ../../info/PAR_README.
|
||||
|
||||
See the beginning of GNUmakefile for reasonable `make' targets to run it.
|
||||
To run it:
|
||||
0. a. Follow the standard Geant4 installation procedure.
|
||||
b. Download and install TOP-C
|
||||
The TOP-C home page is at http://www.ccs.neu.edu/home/gene/topc.html
|
||||
cd <TOPC_INSTALL_DIR>
|
||||
gzip -dc topc.tar.gz | tar -xvf -
|
||||
cd topc
|
||||
./configure
|
||||
make
|
||||
make check
|
||||
[ Copy bin/topc-config to your path ]
|
||||
c. Verify that the Geant4 example installs:
|
||||
cd $G4INSTALL/examples/extended/parallel/ParN04
|
||||
make
|
||||
$G4WORKDIR/bin/$G4SYSTEM/ParN04 ParN04.in
|
||||
2. make run
|
||||
[ By default, the included `procgroup' file creates two slave processes
|
||||
on localhost. ]
|
||||
[ Note that in addition to output on master,
|
||||
$G4WORKDIR/bin/$G4SYSTEM/slave*.out contains slave output. ]
|
||||
[ To remove intermediate files and start over: make parclean ]
|
||||
3. Try running it with slave processes on remote processes.
|
||||
First, test that your local environment is set up correctly.
|
||||
Try:
|
||||
ssh <REMOTE_HOSTNAME> $G4WORKDIR/bin/$G4SYSTEM/ParN04 `pwd`/ParN04.in
|
||||
The above command needs to work without asking for a password.
|
||||
[ If you use dynamic libraries (*.so), make sure the LD_LIBRARY_PATH
|
||||
in your shell startup file (e.g. ~.tcshrc) includes both:
|
||||
$G4INSTALL/lib/$G4SYSTEM and $CLHEP_BASE_DIR/lib
|
||||
If you use AFS, you may need to type 'klog' to renew your AFS token. ]
|
||||
In `procgroup' file, replace `localhost' by desired remote hosts;
|
||||
Add additional remote hosts (additional slaves) if you like.
|
||||
Then: make run
|
||||
|
||||
============================================================================
|
||||
If you read ParGNUmakefile, you'll find other things that you can
|
||||
modify. For example, all TOP-C additions are in conditionals:
|
||||
remove -DG4USE_TOPC from ParGNUmakefile and:
|
||||
make parclean; make run
|
||||
in order to re-compile and rerun without TOP-C.
|
||||
Define REMOTE_SHELL differently if you don't use `ssh' for a remote shell.
|
||||
(If undefined, ParGNUmakefile defines it to be `ssh')
|
||||
Define MACROFILE diferently to use a different set of input commands.
|
||||
Define MEM_MODEL=--seq
|
||||
to run with TOP-C, but using a single (sequential) process, suitable
|
||||
for easy debugging (via gdb, for example).
|
||||
Try: pushd $G4WORKDIR/bin/$G4SYSTEM/; ./ParN04 --TOPC-help
|
||||
to see TOP-C run-time options that can be invoked, such as
|
||||
pushd $G4WORKDIR/bin/$G4SYSTEM/; ./ParN04 --TOPC-num-slaves=5 ParN04.in
|
||||
Alternatively, modify TOPC_OPTIONS in ParGNUmakefile for the same effect.
|
||||
|
||||
You can also try other targets: make run-debug
|
||||
This will run it under gdb, so you can single step to see what happens.
|
||||
make parclean - Start over with clean set of files.
|
||||
|
||||
============================================================================
|
||||
New or modified files:
|
||||
ParN04.cc - Adds one line: #include "ParN04.icc"
|
||||
ParExample.icc inserts: #include "topc.h"
|
||||
and causes main to calls TOPC_init, TOPC_finalize,
|
||||
and to use: `new ParRunManager' instead of `new G4RunManager'
|
||||
GNUmakefile - Adds one line at beginning: include ParGNUmakefile
|
||||
ParGNUmakefile defines EXTRALIBS and CPPFLAGS so as to
|
||||
modify behavior of config/binmake.gmk
|
||||
in order to use TOP-C libraries and includes
|
||||
procgroup - Specifies which slave hosts to use, and where to put output
|
||||
For example: localhost 1 - > slave1.out
|
||||
host=`localhost', executable=`same as master',
|
||||
params of slave=`> slave1.out' (redirect output)
|
||||
If output not redirected, it goes to stdout on master.
|
||||
src/ParRunManager.cc - ParRunManger derived from G4RunManager
|
||||
replaces Gr4RunManager::DoEventLoop w/ TOP-C parallel loop,
|
||||
Adds certain local vars of DoEventLoop as ParRunManager members
|
||||
include/MarshaledObj.h - run-time utilities for marshalling
|
||||
include/MarshaledEx*Hit.h - marshals N04 hits (calorimeter hits)
|
||||
include/MarshaledG4*.h - marshaling routines for Geant4 data structures
|
||||
|
||||
~/slave*.out - Contains outputs of slave1, slave2, etc.
|
||||
Generated each time parallel ParN04 is executed.
|
||||
These files are specified in the file procgroup.
|
||||
|
||||
====================================================================
|
||||
This version passes an event number to the slave and lets the
|
||||
slave generate the event. The slave passes back marshaled hits to
|
||||
the master.
|
||||
|
||||
I will integrate the track level parallelism into this scenario at
|
||||
a later date. For the track level, I will generate several
|
||||
secondary tracks on the master, and then convert the secondary tracks
|
||||
to new events that can be passed to slaves. I will do this only if
|
||||
I detect that there are not enough initial events to fully occupy all
|
||||
the slaves. This scheme has the drawback that we are splitting an event
|
||||
into many events, which may make the summarization, histogram, and so
|
||||
on more difficult. However, track level parallelism will be triggered
|
||||
only when a very small number of events are generated.
|
||||
|
||||
I also want to support postponing
|
||||
a track to the next event ( G4ClassificationOfNewTrack::fPostpone .
|
||||
To do this, each slave will wait to retire an event until it knows that
|
||||
the previous event has been retired.
|
||||
|
||||
In addition, I plan to have only the master read commands and pass
|
||||
them to the slaves. Currently, the master and slaves each read
|
||||
identical commands.
|
||||
|
||||
====================================================================
|
||||
If you are curious about some of the layers, the following
|
||||
stack trace [somewhat out of date now] gives some idea.
|
||||
[ This stack trace is from a run based on ParN02.]
|
||||
|
||||
G4RunManager::BeamOn calls ParRunManager::DoEventLoop
|
||||
(since G4RunManager::DoEventLoop is virtual)
|
||||
ParRunManager::DoEventLoop calls TOPC_master_slave
|
||||
TOPC_master_slave calls submit_task_input
|
||||
submit_task_input eventually calls COMM_send_msg which calls MPI_Send
|
||||
(COMM_send_msg is the communication layer of TOPC;
|
||||
ParN04.cc was linked with the TOP-C MPI communication layer.
|
||||
The same source could have been linked with a POSIX threads layer,
|
||||
a communication layer, or some other communication layer.
|
||||
)
|
||||
MPI_send calls send
|
||||
(where send is the socket system call of libc.so)
|
||||
|
||||
(gdb) where
|
||||
#0 0x41946c62 in send () from /lib/libc.so.6
|
||||
#1 0x400839c1 in send () at wrapsyscall.c:186
|
||||
#2 0x805c547 in MPI_Send (buf=0x82690fc, count=4, datatype=3, dest=2, tag=1, comm=0) at sendrecv.c:236
|
||||
#3 0x805a0b5 in COMM_send_msg (msg=0x82690fc, msg_size=4, dst=2, tag=TASK_INPUT_TAG) at comm-mpi.c:224
|
||||
#4 0x805774e in send_task_input (slave=2, input={data = 0x82690fc, data_size = 4}, tag=TASK_INPUT_TAG) at topc.c:560
|
||||
#5 0x8057aa8 in submit_task_input (input={data = 0x82690fc, data_size = 4}) at topc.c:659
|
||||
#6 0x805813c in TOPC_master_slave (generate_task_input_=0x4003d2e4 <ParRunManager::GenerateEventInput(void)>,
|
||||
do_task_=0x4003d350 <ParRunManager::DoEvent(int *)>, check_task_result_=0x4003d420 <ParRunManager::CheckEventResult(int *, void *)>,
|
||||
update_shared_data_=0) at topc.c:922
|
||||
#7 0x4003d18c in ParRunManager::DoEventLoop (this=0x80c0bf0, n_event=1, macroFile=0x0, n_select=-1) at src/ParRunManager.cc:51
|
||||
#8 0x400b14d1 in G4RunManager::BeamOn () from /afs/cern.ch/user/c/cooperma/scratch-pcitapi07/geant4/lib/libG4run.so
|
||||
#9 0x400b870a in G4RunMessenger::SetNewValue () from /afs/cern.ch/user/c/cooperma/scratch-pcitapi07/geant4/lib/libG4run.so
|
||||
#10 0x4167157b in G4UIcommand::DoIt () from /afs/cern.ch/user/c/cooperma/scratch-pcitapi07/geant4/lib/libG4intercoms.so
|
||||
#11 0x416810a3 in G4UImanager::ApplyCommand () from /afs/cern.ch/user/c/cooperma/scratch-pcitapi07/geant4/lib/libG4intercoms.so
|
||||
#12 0x805db7e in G4UIterminal::ExecuteCommand () at /afs/cern.ch/sw/lhcxx/specific/redhat61/3.2.0/include/CLHEP/Random/Randomize.h:64
|
||||
#13 0x805d42d in G4UIterminal::SessionStart () at /afs/cern.ch/sw/lhcxx/specific/redhat61/3.2.0/include/CLHEP/Random/Randomize.h:64
|
||||
#14 0x8056a3d in main (argc=1, argv=0x80bfa00) at ParN02.cc:98
|
||||
@@ -0,0 +1,47 @@
|
||||
-------------------------------------------------------------------
|
||||
|
||||
=========================================================
|
||||
Geant4 - an Object-Oriented Toolkit for Simulation in HEP
|
||||
=========================================================
|
||||
|
||||
ParN04
|
||||
------
|
||||
|
||||
ParN04 has a simplified collider detector geometry.
|
||||
This example demonstrates the following features.
|
||||
[ Read README for a description of how to run it in parallel. ]
|
||||
|
||||
1. PYTHIA primary events.
|
||||
|
||||
ExN04PrimaryGeneratorAction has G4HEPEvtInterface as the generator.
|
||||
G4HEPEvtInterface accesses to "pythia_event.data", which contains three
|
||||
events of Higgs generation produced by PYTHIA. "pythia_main.f" is an
|
||||
example FORTRAN code of PYTHIA for generating this event sample.
|
||||
|
||||
2. Readout geometry
|
||||
|
||||
ExN04DetectorConstruction defines a simplified collider detecor
|
||||
geometry, tracker made of cylindrical tubes, calorimeter made of
|
||||
cylindrical tubes, and muon trackers made of planes.
|
||||
Cylindrical calorimeter is made of tubes of lead and scintirator
|
||||
without cut in phi nor z direction. Energy deposition in scintirator
|
||||
is accumurated by ExN04CalorimeterSD sensitive detector, which has
|
||||
a readout geometry to find the phi-z cell.
|
||||
|
||||
3. Full set of "ordinary" physics processes
|
||||
|
||||
ExN04PhysicsList defines almost all of leptons and hadrons which
|
||||
Geant4 has dedicated classes for. Also almost all physics processes
|
||||
Geant4 has are defined.
|
||||
|
||||
4. Event filtering by the stacking mechanism.
|
||||
|
||||
Higgs events in "pythia_event.data" have two lepton pairs produced
|
||||
by the Higgs decay via Z0. At the first stage of each event, only the
|
||||
primary muons are tracked without tracking secondaries. then the number
|
||||
of hits on the muon trackers are examined. At the next stage, only
|
||||
the primary charged particles are tracked only inside the barrel
|
||||
tracking area and the isolation of the primary muons are examined.
|
||||
At the third stage, all particles in the RoI (Region of Interest) along
|
||||
the isolated muons are tracked. All these examinations are applied in
|
||||
ExN04StackingAction.
|
||||
@@ -0,0 +1,46 @@
|
||||
------------------------------------------------------------------------------
|
||||
This file and NEW_APP_INSTALL describe important information if you are
|
||||
planning to create a new parallel Geant4 application, similarly
|
||||
to the examples ParN02 and ParN04.
|
||||
This package implements event-level parallelism for Geant4
|
||||
using the free TOP-C distribution. (TOP-C = Task Oriented Parallel C/C++)
|
||||
|
||||
To parallelize an existing sequential Geant4 application, read NEW_APP_INSTALL.
|
||||
|
||||
The design goals are:
|
||||
1. No modification of the Geant4 kernel libraries.
|
||||
2. Application sessions should be efficient and reproducible
|
||||
(independently of the number of slave processes and the
|
||||
order in which they are called by TOP-C)
|
||||
3. Setting deterministic random seeds on each slave
|
||||
[ This code assumes that each slave will initialize its random state
|
||||
based on either the event number or a random seed selected
|
||||
on the master.]
|
||||
|
||||
This assumes that you have already installed CLHEP, Geant4, and TOP-C
|
||||
TOP-C can be downloaded via its home page:
|
||||
http://www.ccs.neu.edu/home/gene/topc.html
|
||||
|
||||
Other qualifications are:
|
||||
1. It has only been tested under Linux, although it should be
|
||||
portable to any other UNIX supported by Geant4.
|
||||
2. Since Geant4 does not currently support a uniform format for
|
||||
tracker hits, calorimeter hits, etc., on new applications,
|
||||
you may have to write marshalling code for your hits, based
|
||||
on the examples in ParN02 and ParN04.
|
||||
3. This should support histograms as long as AnalysisManager::Step() is
|
||||
is trivial. However, I have not had the opportunity
|
||||
for careful testing. Please write to gene@ccs.neu.edu if you
|
||||
are including histograms, and especially if you define
|
||||
AnalysisManager::Step().
|
||||
4. Currently, this distribution does not support:
|
||||
trajectory stores, n-tuples, track parallelism,
|
||||
interactive input commands (no macrofile)
|
||||
I plan to add those features according to demand.
|
||||
Probably: a. allowing interactive input
|
||||
b. track-level parallelism
|
||||
c. other (trajectory store, PostponeOneEvent(), ...)
|
||||
|
||||
Some design assumptions in this ParGeant4 implementation:
|
||||
1. A collectionName for a G4HitsCollection uniquely identifies the
|
||||
G4HitsCollection within a G4HCofThisEvent.
|
||||
@@ -0,0 +1,114 @@
|
||||
<HTML lang="en">
|
||||
<HEAD> <!-- -*-text-*- -->
|
||||
<META http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
|
||||
<TITLE>ParGeant4 README</TITLE>
|
||||
<link rev="made" href="mailto:gene@ccs.neu.edu">
|
||||
<META NAME="Description" CONTENT="ParGeant4 is a parallelization
|
||||
ofGeant4 using TOP-C/C++ (Task Oriented Parallel C/C++).
|
||||
It is free, open source software.">
|
||||
<META NAME="Keywords" CONTENT="ParGeant4, Geant4, TOP-C, parallel computing,
|
||||
unix open source, free, NoW, Beowulf, SMP, DSM">
|
||||
<META NAME="Language" CONTENT="English">
|
||||
<meta name="Author" content="Gene Cooperman">
|
||||
<meta name="Modified" content="2002-02-25">
|
||||
</HEAD>
|
||||
<BODY bgcolor="#FFFFFF">
|
||||
|
||||
<H1>ParGeant4 README</H1>
|
||||
<center>
|
||||
by <a href="http://www.ccs.neu.edu/home/gene/">Gene Cooperman</a>
|
||||
(<a href="mailto:gene@ccs.neu.edu">gene@ccs.neu.edu</a>)
|
||||
</center>
|
||||
There is also a <a href="http://www.ccs.neu.edu/home/gene/pargeant4.html">
|
||||
ParGeant4 Home Page</a>, which will be updated more frequently than
|
||||
this page.
|
||||
|
||||
<P>
|
||||
This distribution implements event-level parallelism for
|
||||
<a href="http://cern.ch/geant4">Geant4</a>
|
||||
using the free TOP-C distribution.
|
||||
(<a href="http://www.ccs.neu.edu/home/gene/topc.html">TOP-C
|
||||
= Task Oriented Parallel C/C++</a>)
|
||||
|
||||
<P>
|
||||
These are example parallelizations that parallelize examples/novice/N02
|
||||
and examples/novice/N04 of the Geant4 distribution.
|
||||
The files also contain information on how to write new parallel
|
||||
Geant4 applications or parallelize existing sequential Geant4
|
||||
applications.
|
||||
|
||||
<P>
|
||||
<PRE>
|
||||
The design goals are:
|
||||
1. No modification of the Geant4 kernel libraries.
|
||||
2. Application sessions should be efficient and reproducible
|
||||
(independently of the number of slave processes and the
|
||||
order in which they are called by TOP-C)
|
||||
3. Setting deterministic random seeds on each slave
|
||||
[ This code assumes that each slave will initialize its random state
|
||||
based on either the event number or a random seed selected
|
||||
on the master.]
|
||||
|
||||
This assumes that you have already installed CLHEP, Geant4, and TOP-C
|
||||
TOP-C can be downloaded via its home page:
|
||||
http://www.ccs.neu.edu/home/gene/topc.html
|
||||
|
||||
Other qualifications are:
|
||||
1. `topc-config' is already in your path.
|
||||
2. It has only been tested under Linux, although it should be
|
||||
portable to any other UNIX supported by Geant4.
|
||||
3. Currently, this distribution does not support:
|
||||
histograms, trajectory stores, n-tuples, track parallelism,
|
||||
interactive input commands (no macrofile)
|
||||
I plan to add those features according to demand.
|
||||
Probably: a. histograms (I believe histograms work now, but I don't
|
||||
have a ready test application. If someone does, could
|
||||
you either check yourself if the parallelization works
|
||||
for histograms, or tell me, and I'll parallelize it.)
|
||||
b. track-level parallelism
|
||||
c. allowing interactive input
|
||||
d. other
|
||||
</PRE>
|
||||
<!--
|
||||
Some design assumptions of Geant4 used here:
|
||||
1. A collectionName for a G4HitsCollection uniquely identifies the
|
||||
G4HitsCollection within a G4HCofThisEvent.
|
||||
-->
|
||||
|
||||
<P>
|
||||
ParGeant4 has been demonstrated to run over the Computational Grid
|
||||
using a new TOP-C communication layer, with support for the Grid,
|
||||
based on Ampic. See related paper:
|
||||
<br>
|
||||
G. Cooperman, H. Casanova, J. Hayes and T. Witzel,
|
||||
``Using TOP-C and AMPIC to Port Large Parallel Applications
|
||||
to the Computational Grid'',
|
||||
<em>Proc. of 2nd IEEC/ACM Interantional Symposium
|
||||
on Cluster Computing and the Grid (CCGrid)</em>, 2002,
|
||||
to appear
|
||||
|
||||
|
||||
<P>
|
||||
(This material is based in part upon work supported by the National Science
|
||||
Foundation under Grants No. 9732330 and 9872114. Any opinions, findings, and
|
||||
conclusions or recommendations expressed in this material are those of
|
||||
the author(s) and do not necessarily reflect the views of the National
|
||||
Science Foundation.)
|
||||
<P>
|
||||
<ADDRESS>
|
||||
Gene Cooperman<br>
|
||||
College of Computer Science, 215CN<br>
|
||||
Northeastern University<br>
|
||||
Boston, MA 02115 <br>
|
||||
e-mail: gene@ccs.neu.edu<br>
|
||||
Phone: (617) 373-8686<br>
|
||||
Fax: (617) 373-5121
|
||||
</ADDRESS>
|
||||
|
||||
<script language="JavaScript">
|
||||
<!---//hide script from old browsers
|
||||
document.write( "<br>Last updated "+ document.lastModified );
|
||||
//end hiding contents --->
|
||||
</script>
|
||||
</BODY>
|
||||
</HTML>
|
||||
Reference in New Issue
Block a user