dags.dag#

Module Contents#

Functions#

concatenate_functions(functions[, targets, ...])

Combine functions to one function that generates targets.

create_dag(functions, targets)

Build a directed acyclic graph (DAG) from functions.

_create_combined_function_from_dag(dag, functions, targets)

Create combined function which allows to execute a complete directed acyclic

get_ancestors(functions, targets[, include_targets])

Build a DAG and extract all ancestors of targets.

_harmonize_and_check_functions_and_targets(functions, ...)

Harmonize the type of specified functions and targets and do some checks.

_harmonize_functions(functions)

_harmonize_targets(targets, function_names)

_fail_if_targets_have_wrong_types(targets)

_fail_if_functions_are_missing(functions, targets)

_fail_if_dag_contains_cycle(dag)

Check for cycles in DAG.

_create_complete_dag(functions)

Create the complete DAG.

_get_free_arguments(func)

_limit_dag_to_targets_and_their_ancestors(dag, targets)

Limit DAG to targets and their ancestors.

_create_arguments_of_concatenated_function(functions, dag)

Create the signature of the concatenated function.

_create_execution_info(functions, dag)

Create a dictionary with all information needed to execute relevant functions.

_create_concatenated_function(execution_info, arglist, ...)

Create a concatenated function object with correct signature.

_format_list_linewise(list_)

dags.dag.concatenate_functions(functions, targets=None, return_type='tuple', aggregator=None, enforce_signature=True)[source]#

Combine functions to one function that generates targets.

Functions can depend on the output of other functions as inputs, as long as the dependencies can be described by a directed acyclic graph (DAG).

Functions that are not required to produce the targets will simply be ignored.

The arguments of the combined function are all arguments of relevant functions that are not themselves function names, in alphabetical order.

Parameters

  • functions (dict or list) – Dict or list of functions. If a list, the function name is inferred from the __name__ attribute of the entries. If a dict, the name of the function is set to the dictionary key.

  • targets (str or list or None) – Name of the function that produces the target or list of such function names. If the value is None, all variables are returned.

  • return_type (str) – One of “tuple”, “list”, “dict”. This is ignored if the targets are a single string or if an aggregator is provided.

  • aggregator (callable or None) – Binary reduction function that is used to aggregate the targets into a single target.

  • enforce_signature (bool) – If True, the signature of the concatenated function is enforced. Otherwise it is only provided for introspection purposes. Enforcing the signature has a small runtime overhead.

Returns

A function that produces targets when called with suitable arguments.

Return type

function

dags.dag.create_dag(functions, targets)[source]#

Build a directed acyclic graph (DAG) from functions.

Functions can depend on the output of other functions as inputs, as long as the dependencies can be described by a directed acyclic graph (DAG).

Functions that are not required to produce the targets will simply be ignored.

Parameters

  • functions (dict or list) – Dict or list of functions. If a list, the function name is inferred from the __name__ attribute of the entries. If a dict, the name of the function is set to the dictionary key.

  • targets (str or list or None) – Name of the function that produces the target or list of such function names. If the value is None, all variables are returned.

Returns

the DAG (as networkx.DiGraph object)

Return type

dag

dags.dag._create_combined_function_from_dag(dag, functions, targets, return_type='tuple', aggregator=None, enforce_signature=True)[source]#

Create combined function which allows to execute a complete directed acyclic graph (DAG) in one function call.

The arguments of the combined function are all arguments of relevant functions that are not themselves function names, in alphabetical order.

Parameters

  • dag (networkx.DiGraph) – a DAG of functions

  • functions (dict or list) – Dict or list of functions. If a list, the function name is inferred from the __name__ attribute of the entries. If a dict, the name of the function is set to the dictionary key.

  • targets (str or list or None) – Name of the function that produces the target or list of such function names. If the value is None, all variables are returned.

  • return_type (str) – One of “tuple”, “list”, “dict”. This is ignored if the targets are a single string or if an aggregator is provided.

  • aggregator (callable or None) – Binary reduction function that is used to aggregate the targets into a single target.

  • enforce_signature (bool) – If True, the signature of the concatenated function is enforced. Otherwise it is only provided for introspection purposes. Enforcing the signature has a small runtime overhead.

Returns

A function that produces targets when called with suitable arguments.

Return type

function

dags.dag.get_ancestors(functions, targets, include_targets=False)[source]#

Build a DAG and extract all ancestors of targets.

Parameters

  • functions (dict or list) – Dict or list of functions. If a list, the function name is inferred from the __name__ attribute of the entries. If a dict, with node names as keys or just the values as a tuple for multiple outputs.

  • targets (str) – Name of the function that produces the target function.

  • include_targets (bool) – Whether to include the target as its own ancestor.

Returns

The ancestors

Return type

set

dags.dag._harmonize_and_check_functions_and_targets(functions, targets)[source]#

Harmonize the type of specified functions and targets and do some checks.

Parameters

  • functions (dict or list) – Dict or list of functions. If a list, the function name is inferred from the __name__ attribute of the entries. If a dict, the name of the function is set to the dictionary key.

  • targets (str or list) – Name of the function that produces the target or list of such function names.

Returns

harmonized functions targets_harmonized: harmonized targets

Return type

functions_harmonized

dags.dag._harmonize_functions(functions)[source]#
dags.dag._harmonize_targets(targets, function_names)[source]#
dags.dag._fail_if_targets_have_wrong_types(targets)[source]#
dags.dag._fail_if_functions_are_missing(functions, targets)[source]#
dags.dag._fail_if_dag_contains_cycle(dag)[source]#

Check for cycles in DAG.

dags.dag._create_complete_dag(functions)[source]#

Create the complete DAG.

This DAG is constructed from all functions and not pruned by specified root nodes or targets.

Parameters

functions (dict) – Dictionary containing functions to build the DAG.

Returns

The complete DAG

Return type

networkx.DiGraph

dags.dag._get_free_arguments(func)[source]#
dags.dag._limit_dag_to_targets_and_their_ancestors(dag, targets)[source]#

Limit DAG to targets and their ancestors.

Parameters

  • dag (networkx.DiGraph) – The complete DAG.

  • targets (str) – Variable of interest.

Returns

The pruned DAG.

Return type

networkx.DiGraph

dags.dag._create_arguments_of_concatenated_function(functions, dag)[source]#

Create the signature of the concatenated function.

Parameters

  • functions (dict) – Dictionary containing functions to build the DAG.

  • dag (networkx.DiGraph) – The complete DAG.

Returns

The signature of the concatenated function.

Return type

inspect.Signature

dags.dag._create_execution_info(functions, dag)[source]#

Create a dictionary with all information needed to execute relevant functions.

Parameters

  • functions (dict) – Dictionary containing functions to build the DAG.

  • dag (networkx.DiGraph) – The complete DAG.

Returns

Dictionary with functions and their arguments for each node in the dag.

The functions are already in topological_sort order.

Return type

dict

dags.dag._create_concatenated_function(execution_info, arglist, targets, enforce_signature)[source]#

Create a concatenated function object with correct signature.

Parameters

  • execution_info (dict) – Dictionary with functions and their arguments for each node in the dag. The functions are already in topological_sort order.

  • arglist (list) – The list of arguments of the concatenated function.

  • targets (list) – List that is used to determine what is returned and the order of the outputs.

  • enforce_signature (bool) – If True, the signature of the concatenated function is enforced. Otherwise it is only provided for introspection purposes. Enforcing the signature has a small runtime overhead.

Returns

The concatenated function

Return type

callable

dags.dag._format_list_linewise(list_)[source]#