Tools

`fre`

The click based entry point to the rest of the package at the command line for fre-cli. fre has a command structure like, fre TOOL COMMAND, akin to the CLI’s provided by popular packages like git and cylc. This enables discovery of the tooling capability, useful for complex tools with multiple options and detailed configurations.

arguments

(optional) --help, help flag
- print information on TOOL options
(optional) -v[v], verbosity flag, up to two
- increments fre’s verbosity from the default logging.WARNING to logging.INFO with one -v, and again to logging.DEBUG with -vv
(optional) -q, quiet flag, up to one
- sets fre’s verbosity from the default logging.WARNING to logging.ERROR.
(optional) -l, --log_file PATH, argument specifying a log file in which to store logging messages
- the given (or default) verbosity will be used as-specified
- the text will still be seen in the terminal, and the format within the log_file is the same as what is printed to screen.

Tools that require arguments to run will alert user about missing arguments, and will also list the rest of the optional parameters if --help is executed. e.g.:

fre TOOL COMMAND --help

Argument flags are not positional, can be specified in any order. Some arguments expect sub-arguments.

`fre app`

fre app tools are intended to be a collection of single-purpose tools.

`fre catalog`

For more complete information on the catalogbuilder tool, please see its official documentation.

`build`

Generate a catalog.

Builds json and csv format catalogs from user input directory path
Minimal Syntax: fre catalog build -i [input path] -o [output path]
Module(s) needed: n/a
Example: fre catalog build -i /archive/am5/am5/am5f3b1r0/c96L65_am5f3b1r0_pdclim1850F/gfdl.ncrc5-deploy-prod-openmp/pp -o ~/output --overwrite

`validate`

Validate a catalog

Runs the comprehensive validator tool (validates vocabulary and ensures catalogs were generated properly)
Minimal Syntax: fre catalog validate [json_path] --vocab OR
fre catalog validate [json_path] --proper_generation
Module(s) needed: n/a
Example: fre catalog validate ~/example_catalog.json --vocab

`fre cmor`

Deprecated, see fremor here

please use NOAA-GFDL/fremor

`fre make`

`checkout`

fre make checkout-script [options]

Purpose: Creates the checkout script and can check out source code (with execute option)
Options:
- -y, --yamlfile [experiment yaml] (required)
- -p, --platform [platform] (required)
- -t, --target [target] (required)
- -j, --njobs [number of jobs to run simultaneously]
- -npc, --no-parallel-checkout (for container build)
- -e, --execute

`makefile`

fre make makefile [options]

Purpose: Creates the makefile
Options:
- -y, --yamlfile [experiment yaml] (required)
- -p, --platform [platform] (required)
- -t, --target [target] (required)

`compile`

fre make compile-script [options]

Purpose: Creates the compile script and compiles the model (with execute option)
Options:
- -y, --yamlfile [experiment yaml] (required)
- -p, --platform [platform] (required)
- -t, --target [target] (required)
- -j, --njobs [number of jobs to run simultaneously]
- -n, --parallel [number of concurrent module compiles]
- -e, --execute

`dockerfile`

fre make dockerfile [options]

Purpose: Creates the dockerfile and creates the container (with execute option)
With the creation of the dockerfile, the Makefile, checkout script, and any other necessary script is copied into the container from a temporary location
Options:
- -y, --yamlfile [experiment yaml] (required)
- -p, --platform [platform] (required)
- -t, --target [target] (required)

`all`

fre make all [options]

Purpose: Create the checkout script, Makefile, compile script, and dockerfile (platform dependent) for the compilation of the model
Options:
- -y, --yamlfile [experiment yaml] (required)
- -p, --platform [platform] (required)
- -t, --target [target] (required)
- -npc, --no-parallel-checkout (for container build)
- -j, --njobs [number of jobs to run simultaneously]
- -n, --parallel [number of concurrent module compiles]

`fre pp`

The identifiers for the Cylc workflows are built from 3 pieces, separated with two underscores:

$(experiment)__$(platform)__$(target)

For the tools called by `fre pp $tool`, these usually correspond to the -e, -p. -t arguments.

The experiment is a short description of the model configuration that was run. It corresponds to one of the experiment names that are displayed by `fre list exps -y $yamlfile`, and the workflow tools with throw an error if -e does not match one of the experiment names in the yaml.

The platform is the combination of the platform upon which the model was run and the compiler with which it was built. Bronx added ‘gfdl’ to the platform for all postprocessing experiments, since those are run on the postprocessing nodes at the Geophysical Fluid Dynamics Lab.

The target is the option that was used for the model compiler. There are currently 5-ish options we expect to see:

prod - Used for production experiments. Runs most slowly, but with highest precision.
repro - Used when reproducing past results for regression tests. Slower than debug, faster than production.
debug - Used when debugging. Fastest runtime.
openmp - Used when compiled and linked with OpenMP. Can be put onto the end of another target (e.g. prod-openmp)
hdf5 - Added directives to generate HDF5-based netCDF files. Can be put onto the end of another target (e.g. repro-hdf5)

For more information see: https://sites.google.com/noaa.gov/oar-gfdl-msd-docs/fre-documentation/fre-documentation/compiling-an-experiment#h.52hq1w2v4klg

`all`

Executes the fre postprocessing steps in order (fre pp configure-yaml, fre pp checkout, fre pp validate, fre pp install, fre pp run, trigger and status)
Minimal syntax: fre pp all -e experiment_name -p platform_name -T target_name -c config_file [ -b [branch] -t [time] ]
Module(s) needed: n/a
Example: fre pp all -e c96L65_am5f4b4r0_amip -p ncrc5.intel23 -T prod -c ./am5.yaml -b main

`configure-yaml`

Postprocessing yaml configuration
Minimal Syntax: fre pp configure-yaml -y [user-edit yaml file] -e experiment_name -p platform_name -t target_name
Module(s) needed: n/a
Example: fre pp configure-yaml -y /home/$user/pp/ue2/user-edits/edits.yaml -e c96L65_am5f4b4r0_amip -p gfdl.ncrc5-deploy -t prod-openmp

`checkout`

Checkout template file and clone https://github.com/NOAA-GFDL/fre-workflows.git git repository to ~/cylc_src
Minimal Syntax: fre pp checkout -e experiment_name -p platform_name -t target_name [ -b [branch] ]
Module(s) needed: n/a
Example: fre pp checkout -e c96L65_am5f4b4r0_amip -p gfdl.ncrc5-deploy -t prod-openmp

`install`

Installs an experiment configuration into ~/cylc_run/$(experiment)_$(platform)_$(target)
Minimal Syntax: fre pp install -e experiment_name -p platform_name -t target_name
Module(s) needed: n/a
Example: fre pp install -e c96L65_am5f4b4r0_amip -p gfdl.ncrc5-deploy -t prod-openmp

`run`

Submits postprocessing job
Minimal Syntax: fre pp run -e experiment_name -p platform_name -t target_name [ --pause --no-wait ]
Module(s) needed: n/a
Example: fre pp run -e c96L65_am5f4b4r0_amip -p gfdl.ncrc5-deploy -t prod-openmp

`trigger`

Starts postprocessing history files belonging to an experiment that represent a specific chunk of time (e.g. 1979-1981)
Minimal Syntax: fre pp trigger -e experiment_name -p platform_name -T target_name -t time
Module(s) needed: n/a
Example: fre pp trigger -e c96L65_am5f4b4r0_amip -p gfdl.ncrc5-deploy -T prod-openmp -t 00010101

`split-netcdf`

Splits single netcdf file into separate netcdf files with one data variable per file
Minimal Syntax: fre pp split-netcdf -f netcdf_file -o output_directory -v var_1,var_2...var_n
Module(s) needed: n/a
Example: fre pp split-netcdf -f 19790101.atmos_tracer.tile6.nc -o output/ -v tasmax,tasmin

`split-netcdf-wrapper`

Given a directory structure with netcdf files, calls split-netcdf on individual netcdf files
If split-netcdf-wrapper is called with the the argument –split-all-vars, then -c and -y are not required.
Minimal Syntax: fre pp split-netcdf-wrapper -i input/ -o output/ -s history_source --split-all-vars [--use-subdirs]
Otherwise, -c and -y requirements are necessary.
Minimal Syntax: fre pp split-netcdf-wrapper -i input/ -o output/ -s history_source -c yaml_component -y yamlfile.yml [--use-subdirs]
Module(s) needed: n/a
Example 1: fre pp split-netcdf-wrapper -i input/ -o output/ --split-all-vars --use-subdirs
Example 2: fre pp split-netcdf-wrapper -i input/ -o output/ -c ocean_cobalt_tracers_year_z -y ./ESM4.5v06_b08_cobv3_mekep_piC.yaml --use-subdirs

`nccheck`

Verify netCDF file contains expected number of time records
Minimal Syntax: fre pp nccheck -f [netCDF file path] -n [number of expected time records]
Module(s) needed: n/a
Example: fre pp nccheck -f /home/$user/some_netcdf_file -n 10

`histval`

Run nccheck over all files found in diag manifest
Minimal Syntax: fre pp histval --history [path to directory containing history files] --date_string [date_string] [ --warn ]
Module(s) needed: n/a
Example: fre pp histval --history /some_path/dir_with_history_files/ --date_string 00010101

`ppval`

Run nccheck over postprocessed time-series files
Minimal Syntax: fre pp ppval --path [path to file]
Module(s) needed: n/a
Example: fre pp ppval --path /some_path/example.200001-200412.nc

`fre yamltools`

`combine-yamls`

fre yamltools combine-yamls [options]

Purpose: Creates a combined yaml file for either compilation or post-processing.
If --use compile, the model yaml is combined with the compile and platforms yaml. If --use pp, the model yaml is combined with post-processing and analysis yamls. If --use cmor, the model yaml is combined with cmor yamls.
Options:
- -y, --yamlfile [experiment yaml] (required)
- -e, --experiment [experiment name]
- -p, --platform [platform] (required)
- -t, --target [target] (required)
- --use [compile|pp|cmor] (required)

`fre list`

`exps`

fre list exps [options]

Purpose: List available post-processing experiments included in the yaml configurations
Options:
- -y, --yamlfile [experiment yaml]

`platforms`

fre list platforms [options]

Purpose: List available platforms included in the yaml configurations
Options:
- -y, --yamlfile [experiment yaml]

`pp-components`

fre list pp-components [options]

Purpose: List components in the postprocessing yaml that will be post-processed. All components will be post-processed unless postprocess_on: False is specified
Options:
- -y, --yamlfile [experiment yaml]
- -e, --experiment [experiment to be post-processed]

Tools

fre

arguments

fre app

fre catalog

build

validate

fre cmor

fre make

checkout

makefile

compile

dockerfile

all

fre pp

all

configure-yaml

checkout

install

run

trigger

split-netcdf

split-netcdf-wrapper

nccheck

histval

ppval

fre yamltools

combine-yamls

fre list

exps

platforms

pp-components

`fre`

`fre app`

`fre catalog`

`build`

`validate`

`fre cmor`

`fre make`

`checkout`

`makefile`

`compile`

`dockerfile`

`all`

`fre pp`

`all`

`configure-yaml`

`checkout`

`install`

`run`

`trigger`

`split-netcdf`

`split-netcdf-wrapper`

`nccheck`

`histval`

`ppval`

`fre yamltools`

`combine-yamls`

`fre list`

`exps`

`platforms`

`pp-components`