PETSc Tutorial Exercises
Intended for use with version 2.0.29 of PETSc
***************
Getting Started
***************
- Locate the PETSc on-line documentation, available via
http://www.mcs.anl.gov/petsc/docs
In particular, you may want to reference the on-line users manual
and manual pages.
- Set the following environmental variables:
PETSC_DIR - location of PETSc installation
PETSC_ARCH - machine on which you will use PETSc
e.g., setenv PETSC_DIR /users/gropp/petsc-2.0.29
setenv PETSC_ARCH alpha
At some sites, there may be commands such as module or soft to set these.
- Explore some of the following example exercises. First, copy
an example program and its corresponding makefile from the PETSc
distribution into a working directory. For example, to use the
"hello world" example listed below, do the following:
% mkdir petsc_examples
% cd petsc_examples
% mkdir hello_world
% cd hello_world
% cp $PETSC_DIR/src/sys/examples/tutorials/ex2.c .
% cp $PETSC_DIR/src/sys/examples/tutorials/makefile .
Note that each of the examples categories should be placed in
a separate working directory so that the makefile commands will
function properly.
- Compile an example program:
debugging version:
make BOPT=g <program_name>, e.g.,
make BOPT=g ex2
optimized version: This is the letter O, not the digit zero
make BOPT=O <program_name>, e.g.,
make BOPT=O ex2
- Run an example program:
mpirun -np <number_of_procs> <program_name>, e.g.,
mpirun -np 2 ex2
******************
Beginner Exercises
******************
- - - - - -
Hello World
- - - - - -
* Objective: To run a simple program and learn the makefile and run commands
Program: $PETSC_DIR/src/sys/examples/tutorials/ex2.c
Compilation commands:
debugging version: make BOPT=g ex2
optimized version: make BOPT=O ex2
Runtime commands:
mpirun -np 1 ex2
mpirun -np 2 ex2
mpirun -np 4 ex2
- - - - - - -
Vectors (Vec)
- - - - - - -
* Objectives: To demonstrate basic vector capabilities.
To demonstrate creation/destruction of PETSc objects.
Programs: $PETSC_DIR/src/vec/examples/tutorials/ex1.c
or $PETSC_DIR/src/vec/examples/tutorials/ex1f.F
Runtime commands:
mpirun -np 2 ex1
or
mpirun -np 2 ex1f
* Objective: To demonstrate parallel vector layout
Programs: $PETSC_DIR/src/vec/examples/tutorials/ex3.c
or $PETSC_DIR/src/vec/examples/tutorials/ex3f.F
Runtime commands:
mpirun -np 2 ex3 -draw_pause -1
or
mpirun -np 2 ex3f -draw_pause -1
where
-draw_pause -1 : click the RIGHT mouse button to
advance the program
- - - - - - - -
Matrices (Mat)
- - - - - - - -
* Objective: To demonstrate matrix assembly
Program: $PETSC_DIR/src/sles/examples/tutorials/ex3.c
Runtime commands:
mpirun -np 2 ex3 -mat_view
and
mpirun -np 2 ex3 -mat_view_draw -draw_pause -1
where
-mat_view : print matrix elements to screen
-mat_view_draw : draw the matrix sparsity structure
Comments:
Note that this matrix assembly is done by 4x4 blocks,
with each process contributing its own local parts; see
calls to MatSetValues().
In this example, you may need to move one of the windows that
pops up aside so that you can right-click the one underneath
to continue the program's execution.
- - - - - - - - - - -
Linear Solvers (SLES)
- - - - - - - - - - -
SLES provides an easy-to-use interface to the combination of a
Krylov subspace iterative solver (KSP) and a preconditioner (PC).
See the SLES, KSP, and PC manual pages as well as the "Linear
Equations Solvers" chapter of the PETSc users manual for details.
* Objective: To demonstrate basic linear solver use
Program: $PETSC_DIR/src/sles/examples/tutorials/ex23.c
Runtime command:
mpirun -np 2 ex23 -ksp_monitor
where
-ksp_monitor : print residual norm at each iteration
Comments:
Note that the program calls SLESView(), which displays
information about the particular solvers used at runtime.
* Objective: To demonstrate setting different preconditioners and
Krylov methods at runtime
Program: $PETSC_DIR/src/sles/examples/tutorials/ex23.c
Runtime command:
mpirun -np 2 ex23 -ksp_monitor -pc_type asm -ksp_type tfqmr -optionsleft
where
-ksp_type tfqmr : set Krylov method to transpose-free QMR
-pc_type asm : set preconditioner to additive Schwarz
-optionsleft : print information about the options specified at runtime
Comments:
Use the -help option for a complete list of solver options.
* Objective: To produce a summary of program performance
Program: $PETSC_DIR/src/sles/examples/tutorials/ex23.c
Runtime command:
mpirun -np 4 ex23 -n 1259 -log_summary
where
-n <dimension> : set the problem dimension
Comments:
The performance summary output is wide, so use an xterm with
120 columns; either resize one or use "xterm -geometry 120x24".
See the chapter "Hints for Performance Tuning" in the PETSc users
manual for help in interpreting this output.
Try using additional solver options as well, and note the
effects on performance. For example,
-ksp_rtol 1.e-10 : set convergence tolerance
Use the -help option for a complete list of solver options.
- - - - - - - - - - - -
Nonlinear Solvers (SNES)
- - - - - - - - - - - -
* Objective: To demonstrate basic nonlinear solver use
Programs: $PETSC_DIR/src/snes/examples/tutorials/ex5.c
or $PETSC_DIR/src/snes/examples/tutorials/ex5f.F and ex5f.h
or $PETSC_DIR/src/snes/examples/tutorials/ex5f90.F
Runtime commands:
mpirun -np 3 ex5 -mx 10 -my 12 -snes_monitor -snes_view
or
mpirun -np 3 ex5f -mx 10 -my 12 -snes_monitor -snes_view
or
mpirun -np 3 ex5f90 -mx 10 -my 12 -snes_monitor -snes_view
where
-snes_monitor : print residual norm at each iteration
-snes_view : print information about the particular
nonlinear solvers used at runtime
-mx <xdim> -my <ydim> : set mesh dimensions
Comments:
By default a Netwon line search method is used.
* Objective: To demonstrate setting different nonlinear solvers at runtime
Programs: $PETSC_DIR/src/snes/examples/tutorials/ex5.c
or $PETSC_DIR/src/snes/examples/tutorials/ex5f.F and ex5f.h
or $PETSC_DIR/src/snes/examples/tutorials/ex5f90.F
Runtime command:
mpirun -np 3 ex5 -mx 10 -my 12 -snes_monitor -snes_view \
-snes_type tr -optionsleft
(or similarly for ex5f and ex5f90)
where
-snes_type tr : set nonlinear solver to a Newton trust region method
-optionsleft : print information about the options specified at runtime
Comments:
Use the -help option for a complete list of solver options.
- - - - - - - - - - - - -
Timestepping Solvers (TS)
- - - - - - - - - - - - -
* Objective: To demonstrate basic timestepping solver use
Programs: $PETSC_DIR/src/ts/examples/tutorials/ex2.c and util2.c
or $PETSC_DIR/src/ts/examples/tutorials/ex2f.F, util2.c, and ex2f.h
Runtime command:
mpirun -np 2 ex2 -ts_view
or
mpirun -np 2 ex2f -ts_view
where
-ts_view : print information about the particular
timestepping solvers used at runtime
Comments:
The backward Euler method is set in this code by a call to
TSSetType(). This example runs for 1000 time steps.
* Objective: To demonstrate setting different timestepping solvers at runtime
Programs: $PETSC_DIR/src/ts/examples/tutorials/ex2.c and util2.c
or $PETSC_DIR/src/ts/examples/tutorials/ex2f.F, util2.c, and ex2f.h
Runtime command:
mpirun -np 2 ex2 -ts_view -ts_type euler
or
mpirun -np 2 ex2f -ts_view -ts_type euler
where
-ts_type euler : set timestepping solver to the Euler method
Comments:
Use the -help option for a complete list of solver options.
**********************
Intermediate Exercises
**********************
- - - - - - - - - - - - - - - - - -
Mesh Management (DA and VecScatter)
- - - - - - - - - - - - - - - - - -
* Objective: To demonstrate use of distributed arrays (DA) to
manage a parallel structured mesh computation.
Program: $PETSC_DIR/src/snes/examples/tutorials/ex5.c
Runtime command:
mpirun -np 4 ex5 -mx 10 -my 11 -da_view_draw -snes_monitor \
-draw_pause -1
where
-da_view_draw : draw DA configuration
* Objective: To demonstrate use of vector scatters (VecScatter)
to manage a parallel unstructured mesh computation.
Program: $PETSC_DIR/src/snes/examples/tutorials/ex10d/ex10.c and adj.in
Runtime command:
mpirun -np 2 ex10 -snes_monitor
- - - - - - - - - - - - - - - - - -
Evaluating and Tuning Performance
- - - - - - - - - - - - - - - - - -
See the chapter "Hints for Performance Tuning" in the PETSc users manual.
* Objective: To demonstrate verbose logging information
Program: $PETSC_DIR/src/sles/examples/tutorials/ex23.c
(and any other programs as well)
Runtime command:
mpirun -np 2 ex23 -log_info
where
-log_info : print verbose logging information
Comments:
Note that this command prints information about matrix assembly,
such as the number of dynamic memory allocations used, maximum
number of nonzeros per row, etc. This information can help
with tuning the matrix assembly process.
******************************************************
Advanced Exercises
******************************************************
* Objective: Demonstrate various multigrid, additive Schwarz,
and Neumann-Neumann preconditioner options for
several test problems.
- - - - - - - - - - -
Linear Solvers (SLES)
- - - - - - - - - - -
* Objective: 3D Laplacian on a structured mesh
Program: $PETSC_DIR/src/sles/examples/tutorials/ex22.c
Multigrid:
mpirun -np 1 ex22 -sles_view -ksp_monitor
(Note: the default smoother is ILU(0))
mpirun -np 1 ex22 -sles_view -ksp_monitor \
-mg_levels_pc_type sor (use sor as a smoother)
mpirun -np 4 ex22 -sles_view -ksp_monitor \
-mg_levels_pc_type sor -mg_levels_pc_sor_local_symmetric
Try some additional multigrid options:
- Use a different number of levels:
-damg_nlevels <n>
- Use a W-cycle instead of the default V-cycle:
-pc_mg_cycles 2
- Use a different scheme:
-pc_mg_type <additive,multiplicative,full,cascade>
- Use 2 pre and 2 post smoothing steps:
-pc_mg_smoothup 2 -pc_mg_smoothdown 2
Also try different combinations of these.
One-level additive Schwarz: (one domain per processor)
mpirun -np 1 ex22 -sles_view -ksp_monitor -pc_type asm
(Note: This has one subdomain, so is the same as ILU(0), the
default subdomain solver)
mpirun -np 4 ex22 -sles_view -ksp_monitor -pc_type asm
(Note: default overlap of 1)
mpirun -np 4 ex22 -sles_view -ksp_monitor -pc_type asm -sub_pc_type lu
(Solve the subproblems exactly with LU)
mpirun -np 4 ex22 -sles_view -ksp_monitor -pc_type asm -pc_asm_overlap 2
Two-level additive Schwarz: (grid ratio of 32 between levels)
mpirun -np 4 ex22 -sles_view -ksp_monitor -damg_nlevels 2 -damg_ratio 32
* Objective: 2D Laplacian on an unstructured grid
Program: $PETSC_DIR/src/contrib/oberman/laplacian_q1/* (copy
everything in this directory)
Compilation:
make BOPT=g main (debugging version)
or
make BOPT=g main (optimized version)
Default solver: block Jacobi with ILU(0) on each block
mpirun -np 4 main -f gr10 (also try using larger grids: gr22, gr44, gr99)
One-level additive Schwarz:
mpirun -np 4 main -f gr10 -pc_type asm
(Note: default is Restricted Additive Schwarz (RASM))
mpirun -np 4 main -f gr10 -pc_type asm -pc_asm_type basic
(This is "standard" additive Schwarz, where the subdomain
restriction and interpolation are the same.)
Try varying the overlap, subdomain solver, etc.
Mandel's balancing Neumann-Neumann preconditioner:
mpirun -np 4 main -ksp_monitor -f gr10 -pc_type nn -mat_type is
mpirun -np 8 main -ksp_monitor -f gr33n -pc_type nn -mat_type is
(Note: This mesh has Dirichlet boundary conditions only on the left edge)
mpirun -np 8 main -ksp_monitor -f gr33n -pc_type nn -mat_type is \
-turn_off_first_balancing -turn_off_second_balancing
(Run with no coarse problem; note that this requires many
more iterations.)
- - - - - - - - - - - -
Nonlinear Solvers (SNES)
- - - - - - - - - - - -
* Objective: Scalar, radiative transport PDE on a structured mesh
Program: $PETSC_DIR/src/snes/examples/tutorials/ex18.c
mpirun -np 2 ex18 -snes_view -snes_monitor
The problem is run twice to preload the memory.
Run the program with the different options suggested above
for multigrid and additive Schwarz.
* Objective: Multicomponent, driven cavity problem (with 4 degrees
of freedom per node) on a structured mesh
Program: $PETSC_DIR/src/snes/examples/tutorials/ex19.c
mpirun -np 2 ex19 -snes_view -snes_monitor
Run the program with the different options suggested above
for multigrid and additive Schwarz.
******************************************************
Interfacing between PETSc and other software packages
******************************************************
See slide #7 for web page pointers to these software packages.
To use one of these external tools with PETSc, the tool must have been
enabled when the local PETSc installation was built (see the file
$PETSC_DIR/bmake/<$PETSC_ARCH>/base.site). The following tools
[may/may not] be available on your machine.
- - - - - - - - - - - - - - - - - - - - - - - - -
SPAI - sparse approximate inverse preconditioner
- - - - - - - - - - - - - - - - - - - - - - - - -
Program: $PETSC_DIR/src/sles/examples/tutorials/ex23.c
Runtime commands:
mpirun -np 1 ex23 -ksp_monitor -pc_type spai
Comments:
Use the -help option for a complete list of SPAI options.
- - - - - - - - - - - - - - - - - -
ParMETIS - parallel partitioner
- - - - - - - - - - - - - - - - - -
Program: $PETSC_DIR/src/dm/ao/examples/tutorials/ex2.c
Runtime command:
mpirun -np 2 ex2 -mat_partitioning_type parmetis
- - - - - - - - - - - -
PVODE - ODE integrator
- - - - - - - - - - - -
Programs: $PETSC_DIR/src/ts/examples/tutorials/ex2.c
or $PETSC_DIR/src/ts/examples/tutorials/ex2f.F
Runtime commands:
mpirun -np 2 ex2 -ts_type pvode
or
mpirun -np 2 ex2f -ts_type pvode
- - - - - - - - - - - - - - - - - - - -
Overture - composite mesh PDE package
- - - - - - - - - - - - - - - - - - - -
Please speak with us if you are interested in the Overture/PETSc interface.
- - - - - - - - - - -
SAMRAI - AMR package
- - - - - - - - - - -
Please speak with us if you are interested in the SAMRAI/PETSc interface.
- - - - - - - - - - - - - - - - - - - - - -
TAO - large-scale optimization software
- - - - - - - - - - - - - - - - - - - - - -
See the TAO installation (TAO_DIR = /u3/sbenson/tao-0.0.2)
and tutorial, including programs:
unconstrained minimization:
$TAO_DIR/src/unconstrained/examples/tutorials/eptorsion2.c
$TAO_DIR/src/unconstrained/examples/tutorials/eptorsion2f.F
bound constrained minimization:
$TAO_DIR/src/bound/examples/tutorials/plate2.c
$TAO_DIR/src/bound/examples/tutorials/plate2f.F