The set of FUN3D design tools has been under heavy development since 1995. Much progress has been made, however much work also remains to be done. Bear in mind as you get familiar with the components that they are very much in flux and are primarily used as research tools, rather than production tools.

A gradient-based approach to the design methodology has been taken, as the cost of computing high number of functions for other techniques is typically prohibitive for CFD simulation-based design. Work within the FUN3D research group has been aimed primarily at robust and efficient analysis and sensitivity analysis capabilities, rather than the optimization schemes themselves. However, others at Langley have done numerous optimization studies using FUN3D as the simulation package.

At this point, all design work must be done on fully tetrahedral grids; mixed element capabilities have not yet been propagated through all of the design tools. _It is assumed that the user has a very good knowledge of using FUN3D for analysis purposes; beginners should get plenty of experience running the flow solver as a stand-alone tool before jumping into design work._ Moreover, please read this documentation before contacting the FUN3D Software Development Team.

One last note before we get started—don’t even think about trying optimization without at least 32 high-end processors available for 24/7 use. The algorithms are very efficient, but a typical optimization will still usually cost you the equivalent of at least a few dozen flow solves.

Setting Up The Directory Structure

The FUN3D design tools rely on a specific directory structure that must be present exactly as described here. The optimization driver can set this tree up for you if you want. Simply execute the following command from the Design directory of the repository (this assumes your repository resides on the filesystem where you will be running):

 ./opt_driver --setup_design

The program will prompt you for the absolute directory name in which your FUN3D source code resides. Enter it in this format, with no trailing slash, and with the single quotation marks:

 '/path/to/my/FUN3D/installation'

The code will also ask you for the directory in which you wish to set up the design codes. This directory must exist, must also be an absolute path with no trailing slash, and don’t forget the single quotation marks.

 '/path/to/my/design/case'

After entering the information, the code will populate the latter directory with the structure of files required to do a FUN3D-based design. As the code terminates, it will print to the screen the remaining steps necessary to set up your design case. _Pay attention to this information: It will specify exactly which files need to be where and what parameters to set._

The directory structure is primarily divided into three parts. The first of the three is called description.1. This directory contains the baseline files for the model, none of which are modified during the course of the design. It is simply a safe place where the code always knows that the original model resides. Prior to executing the design, the user will set up the baseline files in this directory. During execution, the optimization driver will pull files as needed. (The trailing 1 on the directory name is used as a model ID for optimizations using model management algorithms being developed at Langley. Model management is not distributed to users at this time, so your directories will always have a 1 at the end.)

The second directory created in the location you provided will be called model.1. The CFD codes will perform all of their work in these subdirectories during the design. Beneath the model.1 directory, you will also see subdirectories called Adjoint, Flow, Grid, and Rubberize. These directories are populated during the initialization phase with softlinks to each of the executables in the FUN3D installation on your machine. Details of each of the components are given on subsequent pages of this manual.

The third directory created will be called ammo. This directory will contain files related to the optimization procedure itself.

Setting Up The Files Required to Do Design

The user must provide a set of files in the description.1 directory to begin the design. These files are related to the CFD model, the geometric parameterization, the optimization, and the environment in which you are running. The files described below must be present, unless noted otherwise.

CFD Grid Files

The user must provide the baseline grid (and associated boundary condition files) to be used for the optimization. This can be in any of the grid formats currently supported by Party.

CFD Input Deck

The user must provide the baseline input deck fun3d.nml (or ginput.faces prior to release 10.5.0) for the flow solve. The only parameters in this file that will be modified during the design will be the Mach number and angle of attack, if they are specified as design variables (details on that later). Additional quantities such as RMSTOL and IREST may be overridden via command line options (details on that later too). Finally, enough timesteps (NCYC) must be given to converge the flow field to a reasonably steady-state. This is to ensure a stable adjoint solution. It is strongly recommend that you manually running a flow solve and an adjoint solve on your baseline configuration first, prior to doing any design, to get a feel for how RMSTOL, NCYC, and so forth, should be set. More details on how to execute the adjoint solver will come later.

Geometry Parameterization Files

If doing shape optimization, the user must provide a MASSOUD or bandaid parameterization for each body in the mesh to be modified. Currently, the set of desired bodies must either be entirely parameterized with MASSOUD or with bandaids, but not both. Future versions of the codes will allow arbitrary combinations. The documentation here is pretty sparse; it is assumed that the user has obtained MASSOUD or the bandaid package from Jamshid Samareh and has already become familiar with their inputs and outputs.

For MASSOUD parameterizations, the MASSOUD parameter files should be named design.gp.1, design.gp.i, ..., design.gp.n for each of the n bodies to be designed. The files specifying the raw MASSOUD variables should be called design.1, design.i, ..., design.n for each of the n bodies to be designed. _Note, however, that in the current implementation, you must use the custom design variable linking feature of MASSOUD. If you wish to use the raw MASSOUD variables as is, simply define the linking matrix as the identity matrix._ These files specifying the design variable linking for each body should be named design.usd.1, design.usd.i, ..., design.usd.n. The user must also provide the files containing the baseline custom design variable values in customDV.1, customDV.i, ..., customDV.n.

Finally, if running MASSOUD, the MASSOUD input file which specifies the filenames must be provided as massoud.1, massoud.i, ..., massoud.n. These files must reflect the names given in the above paragraph. In addition, the first line of these files must have a positive integer in them equal to the number of user-custom design variables. If you just want to use the raw MASSOUD variables and have specified the identity matrix in the linking file, this number is simply the number of raw MASSOUD variables for that body. For the in/out-of-core parameter, just use in-core (0). The filename for Tecplot output viewing must be named model.tec.i for the @i@th body. The remaining FAST output filename can be named to anything the user wishes; the FUN3D tools do not use this file. A massoud.i file should look like:

 #MASSOUD INPUT FILE
 # runOption 0-analysis, >0-sd users dvs, -1-sd massouds    dvs
 52
 # core 0-incore solution, 1-out of core solution
 0
 # input parameterized file
 design.gp.1
 # design variable input file
 design.1
 # input sensitivity file - used for runOption > 0
 design.usd.1
 # output file grid file
 newframe.fast.1
 # output tecplot file for viewing
 model.tec.1
 # file containing the design variables group
 designVariableGroups.1
 # user design variable file
 customDV.1

For bandaid parameterizations, the files should be named bandaid.data.1, bandaid.data.i, ..., bandaid.data.n. Because bandaids are linear, this inputs is all that is required.

Machine File

This file is optional, and provides a list of machines to run the MPI applications across. It can be named anything you wish, but must be referenced accordingly in the command line options file, described below.

Design Control File

The design is ultimately controlled by the data contained in a file called rubber.data. See the section describing this file for information on its details.

Command Line Options File

This file specifies the command line options to be used with each code in the suite, as well as with mpirun. A template is provided in the Design directory of the source code distribution. The first line of the file specifies the number of codes for which you are specifying command line options. The subsequent line must contain an integer followed by a keyword. The integer specifies how many command line options you are providing for the code identified by the keyword. The valid keywords are flow, adjoint, gridmove, getgrad, party, and mpirun. This line is followed by a line for each of the command line options you wish to provide for the code identified by the keyword. Each command line option should appear in single quotation marks on its own line. The optimization driver will append each of the options for the relevant code to the command line it uses to invoke the code. Note that this mechanism is also used for mpirun, so that options such as -nolocal and -machinefile ../machinefile can be specified. Note that the path preceding the machinefile must be ../ due to directory structure.

Target Pressure Data Files

These files are optional and need only be present for target pressure designs. The files must be named cpstar.data.1, cpstar.data.i, ..., cpstar.data.n. Note this option is relatively new; more thorough documentation is forthcoming. Contact FUN3D Support if you want to pursue inverse design (target pressures) in the near future.

In summary, you should have a set of files in your description.1 directory similar to:

[project].fgrid	FAST CFD grid file (could also be VGRID, etc)
[project].fastbc	FAST CFD boundary conditions file (could also be VGRID, etc)
fun3d.nml (or ginput.faces prior to release 10.5.0)	Solver input deck
bandaid.data.i	Bandaid parameterization for ith body (optional)
design.gp.i	MASSOUD parameterization for ith body (optional)
design.i	MASSOUD raw variables file for ith body (optional)
design.usd.i	MASSOUD linking matrix file for ith body (optional)
massoud.i	MASSOUD filenames file for ith body (optional)
machinefile	List of cluster nodes on which to run the codes (optional)
rubber.data	Main control file for the design
command_line.options	The set of command line parameters to be used for every code, as well as mpirun
cpstar.data.i	Target pressure distribution for slice i (optional)

Optimizer Input: ammo.input

Now that the description.1 directory has been populated and all of the necessary parameters have been set, head over to the ammo subdirectory. You will need to specify the parameters in the ammo.input file. This will control the actual optimization. Set the number of processors to be used for the CFD codes on the top line. For the optimization package, use either a 4 (PORT—unconstrained optimization) or a 5 (NPSOL—constrained optimization). Specify the base directory using an absolute path for the optimization case in single quotation marks on the next line—no trailing slash. This should be identical to the path you specified earlier that pointed to where you wanted to run the design. Next, just put a 1 for the number of models in the suite. The next input is the grid type for your case. This corresponds to the party input parameter for the grid type (1 for FAST, 2 for VGRID, and so on).

On the next line, you can choose the operation to perform. Simply putting a 1 here will just do an analysis on your configuration, i.e., place the surface grid according to the current design variables, move the mesh into place, then run a flow solve. Setting this parameter to a 2 will perform an analysis as just described, followed by a sensitivity analysis that uses a flow adjoint solution and a grid adjoint solution. Finally, setting this value to a 3 will perform the actual optimization.

The following line allows for restarting the optimization from the last design point. If this is a 0, the optimization will be started fresh and use the baseline model files from the description.1 directory. If this is a 1, the driver will just forge ahead with a new optimization, using the most recent set of data files that have been placed throughout the model.1 directory by a previous execution. For the target model to run on, just put a 1.

Next, you need to specify whether using MASSOUD (1) or bandaid parameterizations (2). The appropriate files should have already been placed in the description.1 directory as described above. For the diagnostics flag, just leave this as a 0. The max number of high-fidelity design cycles can be anything; it isn’t used currently. For the max low-fidelity functions input, this is relevant when running PORT, and should be set to the maximum number of flow solves that you are willing to execute. The max number of low-fidelity iterations is also for PORT, and should be set to the maximum number of PORT cycles you are willing to execute.

The relative gradient convergence criterion, tau, and the typical value of the high-fidelity objective function is currently not used. Next, if using PORT, set the relative convergence criterion for the subproblem. If running a lift-constrained case (NPSOL), the next line must contain the lower bound value for the lift constraint. I’ve only dabbled with this constraint, get in touch if you want to play with it further. Finally, for inverse design, the number of slices and intersecting patches at which you are specifying the pressures must be listed. Again, this is pretty new, just put a 0 for the number of slices on the first line, then you don’t need to enter any boundary pairs on the following line. Get in touch if you want to do design for target pressures.

Having fun yet? That should do it for the majority of the problem setup. We still need to deal with rubber.data and discuss some command line options for each code that you may want to list in command_line.options. Read on for more information on how to get your design up and running…

This section describes how to set up each section of the main design control file, rubber.data. A template is provided in the Adjoint directory of the source code distribution.

Code Status Section

This section of the file is currently not in use and should not be altered by the user.

Design Variable Section

This section of rubber.data lays out the design variables for the computation. The section is divided into global variables such as Mach number and angle of attack, as well as shape optimization variables. Each design variable appears on its own row in the file, and has several attributes that must be set by the user. The first column is just a dummy index and is merely to assist the user in quickly navigating through the file. The second column is a toggle to activate the design variable. If this value is a 1, the variable will be allowed to change during the design. If the value is assigned a 0, this variable will be held constant. The third column is the initial value for the current design variable. Columns four and five specify the upper and lower bounds for the current design variable.

Global Design Variables

The freestream Mach number and angle of attack are available for use as design variables. Mach number is in the first row; alpha is in the second. The angle of attack is in degrees. Both the Mach number and alpha specified here will override whatever is present in the initial fun3d.nml (or ginput.faces prior to release 10.5.0) file provided by the user. The freestream Mach number may not be activated for incompressible design calculations.

Shape Variables

The first row following the Mach number and angle of attack entries specifies the number of bodies that the user has parameterized separately using MASSOUD, or alternatively, the number of bandaids present in the problem.

Following the number of bodies, there should be a section of inputs for each body. The first line specifies the number of variables on the current body, and the subsequent lines lay out the design variable information for that body. _The bodies present in the computation may be listed in any order, but the order of their appearance in this control file must match the integer suffix on their parameterization files that you provide in the description.1 directory._ The first body listed here must match up with design.gp.1 (or bandaid.data.1) series. A row of data must be provided for every variable in the parameterizations, whether you are using them or not. If you have 25 variables parameterized in a bandaid, then 25 rows must appear in the corresponding data block of rubber.data, even if only a subset is active. The same applied to design variables for MASSOUD.

Cost Function/Constraint Specification

The first line following the design variable sections specifies the number of functions and constraints to be used. For a single unconstrained cost function, this value should be 1. For a single-objective, single-constraint problem, this value should be a 2. And so forth.

Following the value specifying the total number of functions and constraints, each function and/or constraint will have a block of data associated with it. The cost functions and constraints may be specified in any order.

The first line in the block specifies the composition rule for the current function/constraint. Leave this value as a 1: there is currently only a single composition rule implemented, and it takes the form,

f = SUM [ omega x (C - Cstar) ^ power ]

Here, f_ is the current cost (or constraint), _omega is a weighting factor that may be assigned any value the user desires, C is a generic variable representing an aerodynamic quantity (to be described below), C_star is the target value for the aero quantity, and power is an exponent to be applied to the difference C-C_star. The summation indicated by SUM is taken over the number of components, to be specified below.

Following the composition rule, which should be a 1, the following line specifies whether the current function block is actually a cost/objective (represented by a 1), or a constraint (represented by a 2). The CFD codes themselves do not care if a function is an objective or a constraint—they will provide values and derivatives for them regardless. The optimization driver is the only thing that cares for what the function is actually being used.

The next line states how many components of the current function/constraint is composed. This is the number over which the summation above will be summed. Following the number of components, each component has a line in the file containing several pieces of data. The first column is the boundary condition ID over which to apply the current component. These correspond directly to the boundary conditions in your baseline grid. If you wish to apply a component over the whole grid (total drag, for example), simply put a 0 in this column. Alternatively, if you know you have a strong shock sitting on a flap in a drag minimization problem, you might specify your cost function as the pressure drag acting on just that boundary group, as opposed to the entire vehicle, so that the optimizer will really hone in on the flap. The next column is the keyword for the aero quantity to be used for the current function component. For a list of available keywords, see the module header of the file force_module.f90 in the LibF90 directory. Several of the quantities have not been differentiated (pretty much the more obscure ones such as C_x~, Cy, or Cz~ and their pressure/viscous components, and so forth) The relevant codes (the adjoint solver and GetGrad) will check your input to make sure the components you’ve requested are available. Contact FUN3D Support if a component you need is not available. All of the derivatives for the pressure and shear stresses are there, but all the coefficients have not be differentiated. The next column contains the current value of the current function component. This is an output and need not be set by the user. The final three columns in the row correspond to the weight, target value, and power shown in the summation function above.

The next line in the file lists the current value of the composite function built up of its components. This is an output and need not be set by the user.

The remaining lines in the current function block contain the sensitivity derivatives with respect to all of the design variables listed in the top half of the file. These are outputs that need not be set by the user, however, _the user must provide a line for each design variable in each function block_. The values do not matter, but the codes will need positions to place the latest values. Note that this section is divided into derivatives with respect to the global design variables, as well as the design variables on each of the bodies laid out in the top of the file.

The FUN3D suite is currently set up to interface directly with geometry parameterizations processed by MASSOUD or bandaids. Both capabilities have been developed by Jamshid Samareh of NASA Langley. Users are encouraged to contact him for copies of the software and detailed instructions on how to use them. His packages allow the user to parameterize completely arbitrary shapes using a free-form deformation technique. The packages are extremely efficient and robust, and also provide analytic derivatives for the parameterizations, necessary for FUN3D-based design.

To parameterize your surface grids using MASSOUD or bandaids, you will need to extract them using Party. The easiest way to do this is to pre-process your grid as usual, run a single iteration of the flow solver (the party operation coming next will look for solution files even if you won’t be using them here), then run Party again. Choose the post-processing option, and then option 3 from the Tecplot output menu. This will dump out the surface grid boundary groups in a series of files called [project]_massoudin.tec.i, where i refers to the boundary group number. These files contain the information necessary to parameterize the surface grid using MASSOUD or bandaids—see the documentation for those codes for further instructions on how to proceed from there.

If you use MASSOUD for your parameterizations, the MASSOUD executable must be visible in your environment’s path, and must be named massoud. The optimization driver supplied with FUN3D will attempt to call this executable if MASSOUD parameterizations are present. If you feel the optimization driver is not seeing the MASSOUD executable in your path, try placing a copy of the executable in the model.1/Rubberize subdirectory. This is where MASSOUD is executed by the driver. If you are using bandaids, no additional executables need be supplied— all parameterization manipulations are handled internally by the FUN3D optimization driver. (All of the relationships for bandaids are linear, so that the initial sensitivities remain constant and need only be read in to determine new coordinates for the surfaces.)

Although the two packages described above are the predominant choices to use, the interface to the parameterization scheme has been coded in a modular fashion, and the user may choose to use his or her own package. See the Customization section for additional details.

This section describes how to execute the adjoint solver manually. Normally this is handled for the user by the perform_sensitivity_analysis wrapper, however, it is highly recommended that the user perform an adjoint solution on the baseline configuration to get a feel for input parameters, kick-out strategies, and so forth, to use during the actual design optimization.

Running An Adjoint Solution

To perform an adjoint solution, the user must first perform a flow solution in the model.1/Flow directory. Once the flow solve is complete, a copy of the desired rubber.data file must be placed in the main model.1 directory (one level above where the flow solve was run). The adjoint solver will read rubber.data to get objective/constraint function information for its right-hand side. The file fun3d.nml (or ginput.faces prior to release 10.5.0) must reside in the Flow directory, right where it was for the flow solution. THE FOLLOWING IS PROBABLY NO LONGER CORRECT: (The parameter IREST in this file must be 3, which represents a cold-start for the adjoint field. During designs, the wrapper will automatically do warm restarts, IREST=4, once an adjoint field exists. The rest of the parameters in fun3d.nml (or ginput.faces prior to release 10.5.0) used for the flow solve should remain the same.)

To execute the adjoint solver, change over to the model.1/Adjoint directory and enter the following command (mpirun arguments may be modified for your environment):

 mpirun -np ## -nolocal -machinefile ../machinefile ./dual_mpi

This will run simultaneous adjoint solutions for each objective/constraint outlined in rubber.data for the current flow field, using a solution scheme exactly dual to the flow solve algorithm. When complete, you should see a set of [project]_adj.i files in the model.1/Adjoint directory. These are the adjoint solutions on each partition of the grid, and may be repartitioned and/or post-processed using party, in the same manner as regular flow solution files. Another file that will come out of the adjoint solver will be [project]_hist.tec, a file similar to that of the same name that comes out of the flow solver. This file is a Tecplot file that contains the residual convergence histories for the adjoint solution. You will want to keep track of how far you wish to converge the density adjoint residual—loosely speaking, your sensitivities will converge at the same rate as your functions (i.e., flow solution), but maybe you don’t need your derivatives to be super-accurate for optimization purposes. In this case, you may want to specify a looser RMSTOL for the adjoint solver by providing a command-line override (see below) for the adjoint solver. Because the magnitudes of the adjoint residuals scale with the cost function/constraint definitions, you will probably want to specify a different RMSTOL for the adjoint solution anyway.

Another thing to be aware of, particularly for turbulent flows, is the degree to which you converge your flow solutions. You may find that your forces converge relatively quickly during the flow solve, at which point you may be tempted to terminate the flow solution and fire up an adjoint. However, if the flow field has not reached a sufficiently steady-state, the subsequent adjoint solution may diverge. This is the single most important reason for running an adjoint solution by hand a priori—to get a feel for the convergence levels you need in the flow field to obtain a stable adjoint solution. This is not a major stumbling block for inviscid or laminar flows, but this can be troublesome for RANS problems. This a burden on the user, and we’re working on automated ways to detect stability requirements.

Common Command Line Options

The adjoint solver can receive command line options, just like any other code in the FUN3D suite. Here are some of the more common ones you may be interested in:

--help	Print all available command line options. Note we haven’t had time to actually separate the options for each code; you will see a list of all of the command line options available across the entire FUN3D suite. Currently, you have to know which ones are relevant to the code you are running.
--tightly_couple	(For turbulent flows) Tightly couple the adjoint equations for the mean flow and the turbulence model.
--monitor_linear	Monitor linear system convergence at each time step.
--rmstol #	Override stopping_tolerance in fun3d.nml (or RMSTOL in ginput.faces) (requires a real number).
--irest #	Override restart_read in fun3d.nml (or IREST in ginput.faces) (requires an integer).
--cpstar #	Specify the number of slices for which target pressures are provided (requires an integer).

This section describes how to execute getgrad, which computes the sensitivities with respect to angle-of-attack and the Mach number, and also linearizes the residual and cost function(s) with respect to the grid in order to form the right-hand side of the mesh adjoint equation for shape sensitivities. If using the optimization driver supplied with the source code distribution, the user does not need to know the details described here. This section is provided only to the advanced user who wishes to execute getgrad interactively.

Running Getgrad

To execute getgrad, an adjoint solution for the flow field must have been previously computed and reside in the model.1/Adjoint directory. Getgrad will need these adjoint variables during execution. The control file rubber.data must reside in the model.1 directory, and fun3d.nml (or ginput.faces prior to release 10.5.0) must reside in the model.1/Flow directory. To execute getgrad, go into the model.1/Adjoint directory and issue the following command line (mpirun arguments may be modified for your environment):

 mpirun -np [#] -nolocal -machinefile ../machinefile ./getgrad_mpi --grid_adjoint

Upon completion, if Mach number and/or alpha have been specified as active design variables, their derivatives for each cost function/constraint will now appear in rubber.data. In addition, you should see a new set of files in the model.1/Grid directory called [project]_gridadj_rhs.i. These files contain the right-hand side entries for the mesh adjoint problem on each partition. The mesh movement code must then be run in adjoint mode to obtain the final sensitivities with respect to the shape variables.

Common Command Line Options

Getgrad can receive command line options, just like any other code in the FUN3D suite. Here are some of the more common ones you may be interested in:

--help

Print all available command line options. Note we haven’t had time to actually separate the options for each code; you will see a list of all of the command line options available across the entire FUN3D suite. Currently, you have to know which ones are relevant to the code you are running.

This section describes how to execute the mesh movement. This code also can operate in an adjoint mode, which is used to determine the shape variable sensitivities. If using the optimization driver supplied with the source code distribution, the user does not need to know the details described here. This section is provided only to the advanced user who wishes to execute the mesh movement code interactively.

Running in Mesh Movement Mode

To move your mesh, you must have already executed MASSOUD for every body if you are using MASSOUD parameterizations. The Tecplot-formatted sensitivity files that are produced by MASSOUD must be in the model.1/Rubberize directory, and must be named model.tec.i.sd1 for the @i@th body. Alternatively, if you are using band-aid parameterizations, the bandaid.data.i files must be present in the model.1/Grid directory. The control file rubber.data must also be present in the model.1 directory.

To actually move the grid, enter the following command in the model.1/Grid directory (mpirun arguments may be modified for your environment):

 mpirun -np ## -nolocal -machinefile ../machinefile ./gridmove_mpi --dv_index 1 -1

At this point, you should see some convergence criteria go by on the screen. One number to watch for is a thing called BNORM near the top of the output. This is basically a norm on the amount of surface movement the grid is seeing as input. This can be helpful for example in the case where you have parameterized your baseline surface grids, and you want to move the volume grid to correspond to the surface definition. If the parameterization has been done correctly, this initial BNORM should be very small. (We call this “snapping” the grid.) By default, the code will attempt to move the mesh in two increments of the total change required by the surface deformations. If a movement fails for the current increment (you will see some warnings to the screen about negative volumes), it will “rewind” the mesh back to its previous valid state and cut the increment in half and try again. This will continue up until the number of increments has reached 20. At that point, if the mesh cannot be moved successfully, the code will give up (and tell you so). If the code is successful in moving the grid completely to the new orientation, it will construct new [project]_part.i files in the model.1/Flow directory.

Mesh movement is always an adventure, particularly for viscous grids. You would be wise to play with some typical deformations by hand prior to running a hands-off optimization case. This will give you a feel for the upper and lower bounds to use on the design variables. Some general advice: holding trailing edges fixed can be very helpful as most grid generation packages leave a lot to be desired with grid quality in those regions, especially for viscous grids. Translation and rotation of bodies can also be difficult.

Common Command Line Options for Mesh Movement

The mesh movement code can receive command line options, just like any other code in the FUN3D suite. Here are some of the more common ones you may be interested in:

--help

Print all available command line options. Note we haven’t had time to actually separate the options for each code; you will see a list of all of the command line options available across the entire FUN3D suite. Currently, you have to know which ones are relevant to the code you are running.

--bandaid

This option is necessary if you are running with bandaid parameterizations.

Running in Mesh Adjoint Mode

To compute a mesh adjoint and get the final shape sensitivity derivatives, you must have already executed MASSOUD for every body if you are using MASSOUD parameterizations. The Tecplot-formatted sensitivity files that are produced by MASSOUD must be in the model.1/Rubberize directory, and must be named model.tec.i.sd1 for the ith body. Alternatively, if you are using bandaid parameterizations, the bandaid.data.i files must be present in the model.1/Grid directory. The control file rubber.data must also be present in the model.1 directory. In addition, you must also have executed getgrad to generate [project]_gridadj_rhs.i files in the model.1/Grid directory. These are the right-hand sides for each partition for the mesh adjoint problem.

To actually perform the grid adjoint computation, enter the following command in the model.1/Grid directory (mpirun arguments may be modified for your environment):

  mpirun -np ## -nolocal -machinefile ../machinefile ./gridmove_mpi --grid_adjoint

As the code completes, it will populate the final shape derivatives in the rubber.data file in the model.1 directory. These will be the derivatives that the optimizer will see.

Common Command Line Options for Mesh Adjoints

The mesh movement code can receive command line options, just like any other code in the FUN3D suite. Here are some of the more common ones you may be interested in:

--help

Print all available command line options. Note we haven’t had time to actually separate the options for each code; you will see a list of all of the command line options available across the entire FUN3D suite. Currently, you have to know which ones are relevant to the code you are running.

--bandaid

This option is necessary if you are running with bandaid parameterizations.

The design driver that is supplied in the Design directory of the source code distribution is set up to run with either PORT (unconstrained optimization) or NPSOL (constrained optimization). This section will give a brief overview of the interface with each package.

Using PORT

The interface to the PORT library is contained in port_module.F90 in the Design directory of the source code distribution. To set PORT-specific parameters as outlined in its documentation, you can set various entries in the port_int_workspace and port_real_workspace arrays. (You will see several such assignments in the source code.) You will of course have to recompile the source code if you make any such changes. PORT is connected via a reverse-communication setup: Based on the value of port_int_workspace(1) returned by PORT, the module will call either perform_analysis or perform_sensitivity_analysis before calling PORT again.

When running with PORT, you should receive its output in a file called port.output in the model.1 directory.

Using NPSOL

The interface to the NPSOL library is contained in npsol_module.F90 in the Design directory of the source code distribution. To set NPSOL-specific parameters as outlined in its documentation, you can make various calls to the NPSOL routine npoptn. (You will see several such calls in the source code.) You will of course have to recompile the source code if you make any such changes. Because NPSOL does not allow for reverse communication, it relies on external functions called “funcon” and “funobj”, which provide the current value and derivatives of any constraints and objective functions, respectively. These routines are also contained in the same file (though not in the module!), and will make the subsequent calls to the FUN3D wrappers perform_analysis and perform_sensitivity_analysis.

When running with NPSOL, you should receive its output in files called npsol.printfile and npsol.summaryfile in the model.1 directory.

Running the Optimization

When you have finished populating all of the necessary files and setting all of the various input parameters, you are ready to run the design case. When you are running on a cluster system with a remote filesystem, I have found that it is most robust to execute from the first node in the cluster, rather than a “head” node which controls the worker nodes. I have found that the rapid execution of the various codes causes the filesystem to have a hard time keeping up. By running on the first node, it seems to allow the filesystem to keep up. This is just my experience, you may have to fiddle with things, depending on how your hardware is set up. To execute the optimization, go into the ammo directory and enter the following command:

 ./opt_driver > screen.output &

Redirecting the output into a file will provide a record of what flies by on the screen from each code as the design progresses. It also lets one see the latest info if you are logged in remotely (from home in the middle of the night, etc). _I would highly suggest watching the output during the first design cycle and from time-to-time during the run._

Once the run completes, you should have some sort of design history stored in one of the files mentioned above. Another file that may be of interest is movie.tec in the model.1/Flow directory. This file is a Tecplot file that is appended with the latest grid and flow solve information after each function evaluation. By using Tecplot’s “Animate Zones” option, you can animate the design and see what the grid was doing, how solution contours changed, and so on. Finally, the final set of design variables determined by the optimizer will be available in model.1/rubber.data.

Hooking In Your Own Optimizer

In the design context, the term “function” for CFD computations includes a mesh movement (both surface and volume), a flow solution, and an evaluation of the cost function (and possibly any constraints) at a given set of design variables. For those interested in using the tools at a high-level and do not necessarily need to know what’s “under the hood”, a wrapper has been provided in the LibF90 directory of the distribution named analysis.f90. This module contains a subroutine called perform_analysis which will perform the above operations.

To obtain sensitivities, the FUN3D package relies on a discrete adjoint formulation. This approach yields discretely consistent sensitivity derivatives at the same cost as the baseline analysis described above. A call to the perform_sensitivity_analysis routine in the sensitivity.f90 module will perform an adjoint solution for the flow field, an adjoint solution for the mesh movement scheme, and obtain the final sensitivity derivatives that are requested.

Using the two wrappers provided, users should hopefully be able to hook up to their optimization package of choice with little trouble. Feel free to get in touch for guidance in hooking the wrappers up to your framework.

Using Your Own Parameterization Scheme

Users may use their own parameterization schemes, as long as the file formats match those of MASSOUD or band-aids. Contact FUN3D Support for details on hooking in your own parameterization package. You will need to be able to provide xyz-coordinates for the bodies of interest as well as derivatives of these coordinates with respect to the parameterization variables.

Implementing New Cost Functions / Constraints

Implementing new cost functions or constraints will obviously require some low-level coding. Although the codes are continuously being refactored for modularity and ease of maintenance, there is still a pretty steep learning curve for adding new capabilities. The user should be well versed in F95, unstructured grids, programming in a domain-decomposed environment, and CFD in general. The user will need to provide a basic routine to evaluate the function, and routines to evaluate the linearizations of the function with respect to both the flow-field variables (for the adjoint solver) and with respect to the grid (for sensitivities). Users wanting to venture down this path should probably get in touch with FUN3D support ahead of time to get advice on implementation issues they may end up facing. To check your work, we will highly recommend that you verify your linearizations against the complex-variable form of FUN3D—see the accompanying section on this topic.

Other Customizations

Feel free to get in touch for advice on implementing any other modifications. Our team are more CFDer than optimization or aircraft design gurus, so we are interested in hearing what outside groups actually need in order to get the job done. If your application is of mutual interest we will lobby for the ability to support a joint effort.

Although a reverse, or adjoint, mode of differentiation is primarily used for design with FUN3D, a forward-mode of differentiation is also provided. This capability is useful for design problems containing few design variables and many cost functions or constraints. It is also useful for aero-structural optimization, where derivatives of the dependent variables may be needed at every grid point on the surface. Finally, it is invaluable during code development of new linearizations. For a description of the complex-variable “trick” (it’s super easy), see some of the FUN3D publications.

To generate a complex-variable formulation of FUN3D, go into the main directory of your source code distribution and enter the following command:

 ./Ruby/complexify.rb

This command will execute a Ruby code which will create copies of the current source code for the flow solver and complexify them. This allows us to immediately obtain a forward mode of differentiation for the current flow solver source code. You will see the new source code in a directory called Complex_FUN3D_90. At this point, you can change into this directory and compile the complex form of the solver in the same manner used for the real-valued solver (make seq or make mp“). This solver will read the usual real-valued grid files and can compute derivatives of every variable with respect to Mach number, angle of attack, non-inertial rotation rates, or the x_, _y, or z coordinate of a single grid point. This choice is controlled by the file perturb.dat, a template of which is provided in the FUN3D_90 directory. This file also specifies the step size to be used for the complex perturbation—a value of 1.e-30 or smaller is recommended. The smallest perturbation size supported by most machines/compilers is on the order of 1.e-308.

To compute derivatives with respect to a parameterized variable (i.e., MASSOUD or band-aid variables), a complex-valued grid must first be generated. To do this, go into the top-level directory of the source code distribution, and enter this command:

 ./Ruby/complexify.rb GridMove LibF90

This command will copy and complexify the grid movement code in a new subdirectory called Complex_GridMove. Simply go down into this directory and enter either make seq or @make mpi@ to compile the new source code. Given a set of sensitivities for the surface mesh, this code will propagate the sensitivity information out into the field, and dump out a complex-variable form of the grid partition files. At this point, the user can then run the complex flow solver to obtain derivatives of flow-field quantities with respect to the original design variable for which sensitivities were provided. For more help in executing the complex form of the codes, contact FUN3D Support.