python_utils package

Submodules

python_utils.check_for_preexist_dir_file module

Handle the existence of a directory.

check_for_preexist_dir_file(path, method)

Checks for a preexisting directory or file and, if present, deals with it according to the specified method

Parameters:

path (str) – Path to directory
method (str) – Could be any of [ 'delete', 'reuse', 'rename', 'quit' ]

Returns:

None

Raises:

ValueError – If an invalid method for dealing with a pre-existing directory is specified
FileExistsError – If the specified directory or file already exists

python_utils.check_var_valid_value module

check_var_valid_value(var, values)

Checks whether the specified variable has a valid value

Parameters:

var – The variable
values (list) – Valid values

Returns:

True – If var has valid value; otherwise exit(1)

Raises:

ValueError – If var has an invalid value

python_utils.config_parser module

This file provides utilities for processing different configuration (config) file formats. Supported formats include:

YAML

JSON

SHELL

INI

XML

Typical usage involves first loading the config file, then using the dictionary returned by load_config to make queries.

cfg_main(): Converts between and formats different config file formats

cfg_to_ini_str(cfg, kname=None): Gets contents of config file as INI string

cfg_to_json_str(cfg): Gets contents of config file as a JSON string

cfg_to_shell_str(cfg, kname=None): Gets contents of config file as shell script string

cfg_to_xml_str(cfg): Gets contents of config file as a XML string

cfg_to_yaml_str(cfg): Gets contents of config file as a YAML string

check_structure_dict(dict_o, dict_t)

Checks if a dictionary’s structure follows a template. The invalid entries are returned as a dictionary. If all entries are valid, returns an empty dictionary.

Parameters:

dict_o (dict) – Target dictionary
dict_t (dict) – Template dictionary to compare structure to

Returns:

dict – Invalid key-value pairs.

class custom_dumper(stream, default_style=None, default_flow_style=False, canonical=None, indent=None, width=None, allow_unicode=None, line_break=None, encoding=None, explicit_start=None, explicit_end=None, version=None, tags=None, sort_keys=True)

Bases: Dumper

Custom YAML dumper to correct list indentation

Parameters:: yaml.Dumper – A YAML Dumper object to custom format
Returns:: A custom-formatted Dumper object

cycstr(loader, node): Returns a cyclestring element whose content corresponds to the input node argument

days_ago(arg): A filter for jinja2 that gives us a date string for x number of days ago

dict_to_xml(d, tag): Converts dictionary to an XML tree

extend_yaml(yaml_dict, full_dict=None, parent=None): Updates yaml_dict in place by rendering any existing Jinja2 templates that exist in a value.

filter_dict(dict_o, keys_regex)

Filters dictionary keys based on a list of keys

Parameters:

dict_o – The source dictionary
keys_regex – Keys to retain (could be regex expression)

flatten_dict(dictionary, keys=None)

Flattens a recursive dictionary (e.g.YAML/JSON) to be one level deep

Parameters:

dictionary – The source dictionary
keys (list) – Keys on top level whose contents to flatten; if None, then all of them

Returns:

A one-level deep dictionary for the selected set of keys

get_ini_value(config, section, key): Finds the value of a property in a given section

include(filepaths): Returns a dictionary that includes the contents of the referenced YAML file(s).

join_str(loader, node): Custom tag hangler to join strings

load_config_file(file_name, return_string=0)

Loads config file based on file name extension

Raises:: ValueError – If an unrecognized file extension is used.

load_ini_config(config_file, return_string=0): Loads a config file with a format similar to Microsoft’s INI files

load_json_config(config_file): Loads JSON config file

load_shell_as_ini_config(file_name, return_string=1): Loads shell config file with embedded structure in comments

load_shell_config(config_file, return_string=0)

Loads old-style shell config files. We source the config script in a subshell and get the variables it sets

Parameters:: config_file – Path to config file script
Returns:: Dictionary that should be equivalent to one obtained from parsing a YAML file.

load_xml_config(config_file, return_string=0): Loads XML config file

load_yaml_config(config_file)

Safe loads a YAML file

Parameters:: config_file – Configuration file to parse
Returns:: cfg – A Python object containing the config file data

path_join(arg): A filter for jinja2 that joins paths

startstopfreq(loader, node): Returns a Rocoto-formatted string for the contents of a cycledef tag. Assumes that the items in the node are environment variables, and returns a Rocoto-formatted string.

structure_dict(dict_o, dict_t)

Structures a dictionary based on a template dictionary

Parameters:

dict_o – Dictionary to structure (flat one level structure)
dict_t – Template dictionary used for structuring

Returns:

A dictionary with contents of dict_o following structure of dict_t

update_dict(dict_o, dict_t, provide_default=False)

Updates a dictionary with another

Parameters:

dict_o – flat dictionary used as source
dict_t – target dictionary to update

Returns:

None

xml_to_dict(root, return_string): Converts an XML tree to dictionary

python_utils.create_symlink_to_file module

create_symlink_to_file(target, symlink, relative=True)

Creates a symbolic link to the specified target file.

Parameters:

target (str) – Target file
symlink (str) – Symbolic link to target file
relative (bool) – Optional argument to specify relative symbolic link creation

Returns:

None

python_utils.define_macos_utilities module

check_darwin(cmd)

Checks if Darwin command exists

Parameters:: cmd – The command to check (e.g., gsed, gln)
Returns:: True if successful; otherwise prints error

define_macos_utilities(): Sets some environment variables for Darwin systems differently The variables are: READLINK, SED, DATE_UTIL and LN_UTIL.

python_utils.environment module

date_to_str(d, format='%Y%m%d%H%M')

Gets string from Python datetime object. By default it converts to YYYYMMDDHHmm format unless told otherwise by passing a different format.

Parameters:

d (datetime.datetime) – Datetime object
format (str) – Format of the datetime string; default is "%Y%m%d%H%M" (see format codes for other options).

Returns:

String in YYYYMMDDHHmm or shorter version of it

export_vars(dictionary=None, source_dict=None, env_vars=None)

Exports all (or a select few) global variables of the caller modules to the environment/dictionary. Calls this function at the end of a function that updates environment variables.

Parameters:

dictionary (dict) – Target dictionary to set
source_dict (dict) – Source dictionary
env_vars (list) – List of selected environment variables to export or None, in which case all environment variables are exported

Returns:

None

get_env_var(param)

Gets the value of an environment variable

Parameters:: param – The environment variable
Returns:: A string, a list of strings, or None

import_vars(dictionary=None, target_dict=None, env_vars=None)

Imports all (or a select few) environment/dictionary variables as Python global variables of the caller module. Calls this function at the beginning of a function that uses environment variables.

Note that for read-only environment variables, calling this function once at the beginning should be enough. However, if the variable is modified in the module it is called from, the variable should be explicitly tagged as global, and then its value should be exported back to the environment with a call to export_vars():

import_vars() # import all environment variables
global MY_VAR, MY_LIST_VAR
MY_PATH = "/path/to/somewhere"
MY_LIST_VAR.append("Hello")
export_vars() # this exports all global variables

This is because in shell scripting assumes that everything is global unless specifically tagged local, while the opposite is true for Python.

Parameters:

dictionary (dict) – Source dictionary
target_dict (dict) – Target dictionary
env_vars (list) – List of selected environment variables to import or None, in which case all environment variables are imported

Returns:

None

list_to_str(v, oneline=False)

Given a string or list of strings, constructs a string to be used on right hand side of shell environment variables.

Parameters:

v – A string/number, list of strings/numbers, or null string('')
oneline (bool) – If the string is a single line (True) or multiple (False) ?

Returns:

A string

set_env_var(param, value)

Sets an environment variable.

Parameters:

param – The variable to set
value – A string, a list of strings, or None

Returns:

None

str_to_date(s)

Gets Python datetime object from string.

Parameters:: s (str) – A string
Returns:: Datetime object or None

str_to_list(v, return_string=0)

Constructs a string or list of strings based on the given string. Basically does the reverse operation of list_to_str.

Parameters:: v – A string
Returns:: A string, a list of strings, or a null string ('')

str_to_type(s, return_string=0)

Checks whether the string is a float, int, boolean, datetime, or just regular string. This will be used to automatically convert environment variables to data types that are more convenient to work with. If you don’t want this functionality, pass return_string = 1.

Parameters:

s (str) – A string
return_string (int) – Set to 1 to return the string itself. Set to 2 to return the string itself only for a datetime object

Returns:

A float, int, boolean, datetime, or the string itself when all else fails

type_to_str(v)

Gets a string representing the value of a given float, int, boolean, date or list of these types.

Parameters:: v – A variable of the above types
Returns:: A string

python_utils.fv3write_parms_lambert module

To use this tool, first source the workflow environment:

$> module use /path/to/ufs-srweather-app/modulefiles
$> module load wflow_<machine>
$> conda activate srw_graphics

Make sure to adjust the modulefiles path and <machine> to correspond to your system. Even though the message printed to the console will direct users to run conda activate srw_app, this script requires an environment (e.g., srw_graphics) that includes pygraf or cartopy. The srw_graphics environment uses cartopy for plotting. If the srw_app environment is already loaded, users can simply run conda activate srw_graphics to switch environments.

For usage instructions, run:

$> python fv3write_parms_lambert.py -h

python_utils.misc module

dict_find(user_dict, substring)

Find any keys in a dictionary that contain the provided substring

Parameters:

user_dict – dictionary to search
substring – substring to search keys for

Returns:

True if substring found, otherwise False

find_pattern_in_file(pattern, file_name)

Finds a regular expression (regex) pattern in a file

Parameters:

pattern (str) – Regex pattern
file_name (str) – Name of text file

Returns:

A tuple of matched groups or None

find_pattern_in_str(pattern, source)

Finds a regular expression (regex) pattern in a string

Parameters:

pattern (str) – Regex pattern
source (str) – Text string to search for regex pattern

Returns:

A tuple of matched groups or None

lowercase(s)

Converts a given string to lowercase

Parameters:: s (str) – The string to change to lowercase
Returns:: Lowercased string

uppercase(s)

Converts a given string to uppercase

Parameters:: s (str) – The string to change to uppercase
Returns:: Uppercased string

python_utils.print_input_args module

print_input_args(valid_args)

Prints function arguments for debugging purposes

Parameters:: valid_args (dict) – Dictionary of argument-value pairs
Returns:: num_valid_args – Number of printed arguments

python_utils.print_msg module

log_info(info_msg, verbose=True, dedent_=True)

Prints information message using the logging module. This function should not be used if Python logging has not been initialized.

Parameters:

info_msg (str) – Info message to print
verbose (bool) – Set to False to silence printing
dedent (bool) – Set to False to disable “dedenting”/formatting and print string as-is

Returns:

None

print_err_msg_exit(error_msg='', stack_trace=True)

Prints out an error message to standard error and exits. It can optionally print the stack trace as well.

Parameters:

error_msg (str) – Error message to print
stack_trace (bool) – Set to True to print stack trace

print_info_msg(info_msg, verbose=True)

Prints an informational message to standard output when verbose is set to True. It does proper “dedentation”/formatting that is needed for readability of Python code.

Parameters:

info_msg (str) – Info message to print
verbose (bool) – Set to False to silence printing

Returns:

Boolean value – True if message is successfully printed; False if verbose is set to False.

python_utils.run_command module

run_command(cmd)

Runs system command in a subprocess

Parameters:: cmd (str) – Command to execute
Returns:: Tuple of (exit code, std_out, std_err)

python_utils.xml_parser module

has_tag_with_value(tree, tag, value)

Checks if XML tree has a node with tag and value

Parameters:

tree (xml.etree.ElementTree) – The XML tree
tag (str) – The tag
value (str) – Text of tag

Returns:

Boolean value

load_xml_file(xml_file)

Loads XML file

Parameters:: xml_file – Path to XML file
Returns:: tree – Root of the XML tree