Hide Notes 

Slide 1: The Center for Astrophysical Thermonuclear Flashes What’s FLASH? How does FLASH do it? Katherine M Riley Code Group An Advanced Simulation & Computing (ASC) Academic Strategic Alliances Program (ASAP) Center at The University of Chicago
Slide 2: Overview u What is FLASH u Basics of FLASH2 Architecture u Basics behind a problem setup u Walk away with: u Enough of an idea of these concepts to be able to look at a sample setup and understand what is happening. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago
Slide 3: FLASH: The Application u To simulate matter accreted onto the surfaces of compact stars, nuclear ignition of the accumulated (and possibly stellar) material, and subsequent evolution of the star’s interior, surface, and exterior. u Novae (on white dwarf surfaces) u Type 1a supernovae (in white dwarf interiors) u X-ray bursts (on neutron star surfaces) The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago
Slide 4: FLASH Results u u u u u Compressible reactive flow Wide range of length of time scales Many interacting physical processes Only indirect validation possible for the astrophysics Many people in collaboration Compressible turbulence Flame-vortex interactions Shocked cylinder Nova outbursts on white dwarfs Intracluster interactions Cellular detonations White Dwarf deflagration Helium burning on neutron stars Rayleigh-Taylor instability The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago
Slide 5: What FLASH Provides u Physics u Hydrodynamics PPM u MHD u Relativistic PPM u u Infrastructure u Setup u AMR: Paramesh u Regular testing u Parallel I/O u u Nuclear Physics u Gravity u Cosmology u Particles hdf5, pnetcdf, u Profiling u Runtime and post- processing visualization The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago
Slide 6: A Little FLASH History u FLASH0 u Paramesh2, Prometheus and EOS/Burn BAM u FLASH1 u Smoothing out the smash u First form of module architecture & inheritance u FLASH2 u Untangle modules from each other (Grid) u dBase u Concept of levels of users u FLASH3 u Stricter interface control & module architecture u Taming the database The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago
Slide 7: FLASH2 Audiences FLASH Application Developer • Works on just about everything • Grid development • Data access • Architecture Application Programmer • Develop physics modules • Talk to grid • Module communication End User • Initialize setup • Boundary conditions • Basic data structure • (Might add a kernel) The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago
Slide 8: FLASH2 Code Basics u An application code, composed of units/modules. Particular modules are set up together to run different physics problems. u Performance, Testing, Usability, Portability u Fortran, C, Python, … u 560,000* lines of code u 75% code, 25% comment u Very portable u Scaling to 1000’s of procs The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago * Internal Release
Slide 9: Basic Computational Unit : Block u The adaptive grid is composed of blocks u All blocks: same dimensions u Cover different fraction of the physical domain. u Kevin Olson will talk about this more. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago
Slide 10: First Look at FLASH u ‘The tar-ball’ u ‘source’ directories u modules or groups of modules u Post-processing tools, docs, setups u Setup u FLASH architecture tool u Selects and sets up these modules u Collecting variables, runtime parameters, etc The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago
Slide 11: Structure of FLASH Modules (not exact!) Database dBase_init() dBaseGetData() dBasePutData() dBaseProperty() Mesh mesh_init() mesh_guardcells() mesh_updateRef() mesh_fluxConserve() IO init() Checkpoint_write() Checkpoint_read() MHD init() Driver init() dBaseGetData() dBasePutData() dBaseProperty() Visualization init() render() Particles init() advance() Cosmology init() Hydro init() tstep() hydro3d() Gravity init() tstep() grav3d() Source_terms init() tstep() src_terms() Materials eos3d() eos1d() eos() Explicit mhd constant point_mass burn iso13 ... cool Gamma Helmholtz ... PPM Unsplit Poisson heat Diffuse WENO PPM Multigrid Multipole
Slide 12: What’s a FLASH Module? u FLASH basic architecture unit: Modules u Component of the FLASH code providing a particular functionality u Different combinations of modules are used for particular problem setups u Ex: driver, hydro, mesh, dBase, I/O u Fake inheritance by use of directory structure u Modules communicate u Driver u Variable Database The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago
Slide 13: Abstract FLASH2 Module 1. Meta-data (Configuration Info) • Interface with driver and setup • Variable/parameter registration • Variable attributes • Module Requirements 2. Interface Wrapper • Exchange with variable database • Prep data for kernels FLASH Application 3. Physics Kernel(s) • Single patch, single proc functions written in any language • Can be sub-classed Collection of Flash2 Modules The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Component driver Database mesh
Slide 14: Module Implementations u FLASH2 Modules are directory trees u source/hydro/explicit/split/ppm u Each level might have source u Source relevant for all directories/implementations below u Preserves interfaces u Allows flexible implementations The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago
Slide 15: Inheritance Through Directories: Hydro An ‘empty’ hydro init, hydro, tstep are defaults on top of the directory tree. Hydro/Explicit Replaces tstep Introduces ‘shock’ No hydro Implemented yet! Hydro/Explicit/Split hydro implemented Uses general explicit tstep Uses general shock Replaces init init hydro tstep Explicit tstep shock Split hydro implemtation DeltaForm init The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago
Slide 16: The Module Config File u Declare solution variables, fluxes u Declare runtime parameters u Sets defaults u Lists required, exclusive modules u Config files are additive down the directory tree - no replacements The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago
Slide 17: Setup Building an Application Configuration Tool (Setup) Source Terms Materials Gravity Database Mesh Driver Particles I/O Vis MHD Hydro The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago
Slide 18: FLASH Setup: Implements Architecture u Python code links together needed physics and tools for a problem u object u Traverses modules to get implementations u Determines solution data storage list u Creates list of parameters from modules u Configures Makefiles properly The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago
Slide 19: Accessing the Mesh u Physics and tools access some scope of mesh u Database u Unifies method for module to access data out of its scope u Mesh data u Solution Data, Grid information u Runtime parameters u Mesh Interface u Wrappers to the public grid routines u initialize, guardcell, fluxconserve, updateRefinment The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago
Slide 20: FLASH Audiences FLASH Application Solidify information so far • Modules and setup • How do they combine to make create an application? That runs? • Go through a little of that. End User • Initialize setup • Boundary conditions • Basic data structure • (Might add a kernel) The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago
Slide 21: Simple FLASH Flow Driver Init Evolve TimeStep Variable Database dBase Mesh Function Data Evolve Hydro SourceTerms … UpdateGrid Paramesh updateGrid ghostCellFill fluxConserve Hydro Prepare data for kernels SourceTerms Burn Heat Cool Ioniz… hydroBlock Burn burnBlock The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago
Slide 22: Pulling it All Together u Choose a problem setup u Run setup to configure that problem u Everything is in a new top-level directory u ‘object’ u Make u Run u Flash.par for runtime parameters u Defaults already set from particular modules The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago
Slide 23: Setups u A basic problem setup u Config file u Required physics modules Default list runtime parameter configuration Initial conditions for the problem set block by block u u Flash.par u u Init_block u u Many other possible files: Driver, Refinement algorithms, User defined boundary conditions u Any files in setup take precedence The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago
Slide 24: Provided Driver u Provided: u Second order, u New drivers u Put in setups u Welcome state form, strang split flash.F90 Initialize() Loop over timesteps evolvePhysics() timestep() output() visualize() End loop Finalize() contributions evolve.F90 set time step hydro sourceTerms cosmology radiation particles gravity set time step (repeat physics) Mesh_updateGrid The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago
Slide 25: What’s Next u Covered the basics of what FLASH can do u Now - How does it do it? u Alan Calder The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago