:py:mod:`dags.dag` ================== .. py:module:: dags.dag Module Contents --------------- Functions ~~~~~~~~~ .. autoapisummary:: dags.dag.concatenate_functions dags.dag.create_dag dags.dag._create_combined_function_from_dag dags.dag.get_ancestors dags.dag._harmonize_and_check_functions_and_targets dags.dag._harmonize_functions dags.dag._harmonize_targets dags.dag._fail_if_targets_have_wrong_types dags.dag._fail_if_functions_are_missing dags.dag._fail_if_dag_contains_cycle dags.dag._create_complete_dag dags.dag._get_free_arguments dags.dag._limit_dag_to_targets_and_their_ancestors dags.dag._create_arguments_of_concatenated_function dags.dag._create_execution_info dags.dag._create_concatenated_function dags.dag._format_list_linewise .. py:function:: concatenate_functions(functions, targets=None, return_type='tuple', aggregator=None, enforce_signature=True) Combine functions to one function that generates targets. Functions can depend on the output of other functions as inputs, as long as the dependencies can be described by a directed acyclic graph (DAG). Functions that are not required to produce the targets will simply be ignored. The arguments of the combined function are all arguments of relevant functions that are not themselves function names, in alphabetical order. :param functions: Dict or list of functions. If a list, the function name is inferred from the __name__ attribute of the entries. If a dict, the name of the function is set to the dictionary key. :type functions: dict or list :param targets: Name of the function that produces the target or list of such function names. If the value is `None`, all variables are returned. :type targets: str or list or None :param return_type: One of "tuple", "list", "dict". This is ignored if the targets are a single string or if an aggregator is provided. :type return_type: str :param aggregator: Binary reduction function that is used to aggregate the targets into a single target. :type aggregator: callable or None :param enforce_signature: If True, the signature of the concatenated function is enforced. Otherwise it is only provided for introspection purposes. Enforcing the signature has a small runtime overhead. :type enforce_signature: bool :returns: A function that produces targets when called with suitable arguments. :rtype: function .. py:function:: create_dag(functions, targets) Build a directed acyclic graph (DAG) from functions. Functions can depend on the output of other functions as inputs, as long as the dependencies can be described by a directed acyclic graph (DAG). Functions that are not required to produce the targets will simply be ignored. :param functions: Dict or list of functions. If a list, the function name is inferred from the __name__ attribute of the entries. If a dict, the name of the function is set to the dictionary key. :type functions: dict or list :param targets: Name of the function that produces the target or list of such function names. If the value is `None`, all variables are returned. :type targets: str or list or None :returns: the DAG (as networkx.DiGraph object) :rtype: dag .. py:function:: _create_combined_function_from_dag(dag, functions, targets, return_type='tuple', aggregator=None, enforce_signature=True) Create combined function which allows to execute a complete directed acyclic graph (DAG) in one function call. The arguments of the combined function are all arguments of relevant functions that are not themselves function names, in alphabetical order. :param dag: a DAG of functions :type dag: networkx.DiGraph :param functions: Dict or list of functions. If a list, the function name is inferred from the __name__ attribute of the entries. If a dict, the name of the function is set to the dictionary key. :type functions: dict or list :param targets: Name of the function that produces the target or list of such function names. If the value is `None`, all variables are returned. :type targets: str or list or None :param return_type: One of "tuple", "list", "dict". This is ignored if the targets are a single string or if an aggregator is provided. :type return_type: str :param aggregator: Binary reduction function that is used to aggregate the targets into a single target. :type aggregator: callable or None :param enforce_signature: If True, the signature of the concatenated function is enforced. Otherwise it is only provided for introspection purposes. Enforcing the signature has a small runtime overhead. :type enforce_signature: bool :returns: A function that produces targets when called with suitable arguments. :rtype: function .. py:function:: get_ancestors(functions, targets, include_targets=False) Build a DAG and extract all ancestors of targets. :param functions: Dict or list of functions. If a list, the function name is inferred from the __name__ attribute of the entries. If a dict, with node names as keys or just the values as a tuple for multiple outputs. :type functions: dict or list :param targets: Name of the function that produces the target function. :type targets: str :param include_targets: Whether to include the target as its own ancestor. :type include_targets: bool :returns: The ancestors :rtype: set .. py:function:: _harmonize_and_check_functions_and_targets(functions, targets) Harmonize the type of specified functions and targets and do some checks. :param functions: Dict or list of functions. If a list, the function name is inferred from the __name__ attribute of the entries. If a dict, the name of the function is set to the dictionary key. :type functions: dict or list :param targets: Name of the function that produces the target or list of such function names. :type targets: str or list :returns: harmonized functions targets_harmonized: harmonized targets :rtype: functions_harmonized .. py:function:: _harmonize_functions(functions) .. py:function:: _harmonize_targets(targets, function_names) .. py:function:: _fail_if_targets_have_wrong_types(targets) .. py:function:: _fail_if_functions_are_missing(functions, targets) .. py:function:: _fail_if_dag_contains_cycle(dag) Check for cycles in DAG. .. py:function:: _create_complete_dag(functions) Create the complete DAG. This DAG is constructed from all functions and not pruned by specified root nodes or targets. :param functions: Dictionary containing functions to build the DAG. :type functions: dict :returns: The complete DAG :rtype: networkx.DiGraph .. py:function:: _get_free_arguments(func) .. py:function:: _limit_dag_to_targets_and_their_ancestors(dag, targets) Limit DAG to targets and their ancestors. :param dag: The complete DAG. :type dag: networkx.DiGraph :param targets: Variable of interest. :type targets: str :returns: The pruned DAG. :rtype: networkx.DiGraph .. py:function:: _create_arguments_of_concatenated_function(functions, dag) Create the signature of the concatenated function. :param functions: Dictionary containing functions to build the DAG. :type functions: dict :param dag: The complete DAG. :type dag: networkx.DiGraph :returns: The signature of the concatenated function. :rtype: inspect.Signature .. py:function:: _create_execution_info(functions, dag) Create a dictionary with all information needed to execute relevant functions. :param functions: Dictionary containing functions to build the DAG. :type functions: dict :param dag: The complete DAG. :type dag: networkx.DiGraph :returns: Dictionary with functions and their arguments for each node in the dag. The functions are already in topological_sort order. :rtype: dict .. py:function:: _create_concatenated_function(execution_info, arglist, targets, enforce_signature) Create a concatenated function object with correct signature. :param execution_info: Dictionary with functions and their arguments for each node in the dag. The functions are already in topological_sort order. :type execution_info: dict :param arglist: The list of arguments of the concatenated function. :type arglist: list :param targets: List that is used to determine what is returned and the order of the outputs. :type targets: list :param enforce_signature: If True, the signature of the concatenated function is enforced. Otherwise it is only provided for introspection purposes. Enforcing the signature has a small runtime overhead. :type enforce_signature: bool :returns: The concatenated function :rtype: callable .. py:function:: _format_list_linewise(list_)