mlonmcu.setup package

Subpackages

Submodules

mlonmcu.setup.cache module

Definition of Taks Cache

class mlonmcu.setup.cache.TaskCache[source]

Bases: object

Task cache used to store dependency paths for the current and furture sessions.

This can be interpreted as a “modded” dictionary which takes a key + some flags.

find_best_match(name: str, flags=[]) → Any[source]

Utility whih tries to resolve the cache entry with the beste match.

Parameters:
namestr

The cache-key.

flagslist

Optional flags used for the lookup.

read_from_file(filename, reset=True)[source]
write_to_file(filename)[source]
mlonmcu.setup.cache.convert_key(name)[source]

mlonmcu.setup.gen_requirements module

MLonMCU Python requirements.txt generator.

This script generates a set of requirements.txt files (stored in ./requirements) that describe MLonMCU’s Python dependencies.

## Pieces

MLonMCU can be roughly broken into these named pieces along the lines of Python dependencies:

  • “core”: A core piece, which is intended to be buildable with very few external dependencies. Users can use Relay, compile models, and run autotuning with this part.

  • Extra features (i.e. TVM). These enhance MLonMCU’s functionality, but aren’t required for basic operation.

## What this tool does

From these pieces, this tool builds:

  • requirements/<name>.txt - Python dependencies for each named piece above, <name> is the same as the quoted piece name.

  • requirements/all.txt - Consolidated Python dependencies for all pieces, excluding dev below.

  • requirements/dev.txt - Python dependencies needed to develop MLONMCU, such as lint and test tools.

The data representing each piece is contained in the two maps below.

exception mlonmcu.setup.gen_requirements.ValidationError(config: str, problems: List[str])[source]

Bases: Exception

Raised when a validation error occurs.

static format_problems(config: str, problems: List[str]) → str[source]

Format a list of problems with a global config variable into human-readable output.

Parameters:
configstr

Name of the global configuration variable of concern. Prepended to the output.

problems: list[str]

A list of strings, each one a distinct problem with that config variable.

Returns:
str

A human-readable string suitable for console, listing the problems as bullet points.

mlonmcu.setup.gen_requirements.join_and_write_requirements(args: Namespace)[source]
mlonmcu.setup.gen_requirements.join_requirements() → Dict[str, Tuple[str, List[str]]][source]

Validate, then join REQUIRMENTS_BY_PIECE against CONSTRAINTS and return the result.

Returns:
An OrderedDict containing REQUIREMENTS_BY_PIECE, except any dependency mentioned in CONSTRAINTS
is replaced by a setuptools-compatible constraint.
mlonmcu.setup.gen_requirements.main()[source]
mlonmcu.setup.gen_requirements.parse_args() → Namespace[source]
mlonmcu.setup.gen_requirements.parse_semver(package: str, constraint: str, problems: List[str]) → Tuple[List[str], int, int][source]

Parse a semantic versioning constraint of the form “^X.[.Y[.Z[…]]]]”

Parameters:
packagestr

Name of the package specifying this constraint, for reporting problems.

constraintstr

The semver constraint. Must start with “^”

problemsList[str]

A list of strings describing problems that have occurred validating the configuration. Problems encountered while validating constraint are appended to this list.

Returns:
tuple[list[str], int, int]

A 3-tuple. The first element is a list containing an entry for each component in the semver string (components separated by “.”). The second element is the index of the component in the list which must not change to meet the semver constraint. The third element is an integer, the numeric value of the changing component (this can be non-trivial when the patch is the changing part but pre-, post-release, or build metadta.

See “Caret requirements” at https://python-poetry.org/docs/versions/.

mlonmcu.setup.gen_requirements.semver_to_requirements(dep: str, constraint: str, joined_deps: List[str])[source]

Convert a SemVer-style constraint to a setuptools-compatible constraint.

Parameters:
depstr

Name of the PyPI package to depend on.

constraintstr

The SemVer constraint, of the form “^<semver constraint>”

joined_depslist[str]

A list of strings, each a setuptools-compatible constraint which could be written to a line in requirements.txt. The converted constraint is appended to this list.

mlonmcu.setup.gen_requirements.validate_constraints() → List[str][source]

Validate CONSTRAINTS, returning a list of problems found.

Returns:
list[str]

A list of strings, each one describing a distinct problem found in CONSTRAINTS.

mlonmcu.setup.gen_requirements.validate_or_raise()[source]
mlonmcu.setup.gen_requirements.validate_requirements_by_piece() → List[str][source]

Validate REQUIREMENTS_BY_PIECE, returning a list of problems.

Returns:
list[str]

A list of strings, each one describing a distinct problem with REQUIREMENTS_BY_PIECE.

mlonmcu.setup.setup module

class mlonmcu.setup.setup.Setup(features=None, config=None, context=None, tasks_factory=None)[source]

Bases: object

MLonMCU dependency management interface.

DEFAULTS = {'num_threads': None, 'print_outputs': False}
FEATURES = {}
OPTIONAL = {}
REQUIRED = {}
clean_cache(interactive=True)[source]
clean_dependencies(interactive=True)[source]
generate_requirements()[source]
get_dependency_order()[source]
install_dependencies(progress=False, write_cache=True, write_env=True, rebuild=False)[source]
invoke_single_task(name, progress=False, write_cache=True, write_env=True, rebuild=False)[source]
process_features(features)[source]
setup_progress_bar(enabled)[source]
property verbose
visualize(path, ordered=False)[source]
write_cache_file()[source]
write_env_file()[source]

mlonmcu.setup.task module

Definitions of a task registry used to automatically install dependencies.

class mlonmcu.setup.task.TaskFactory[source]

Bases: object

Class which is used to register all available tasks and their annotations.

Attributes:
registrydict

Mapping of task names and their actual function

dependenciesdict

Mapping of task dependencies

providersdict

Mapping of which task provides which artifacts

typesdict

Mapping of task types

validatesdict

Mapping of validation functions for the tasks

changedlist

List of tasks?artifacts which have changed recently

needs(keys, force=True)[source]

Decorator which registers the artifacts a task needs to be processed.

optional(keys)[source]

Decorator for optional task requirements.

param(flag, options)[source]

Decorator which registers available task parameters.

provides(keys)[source]

Decorator which registers what a task provides.

register(category=TaskType.MISC)[source]

Decorator which actually registers a task in the registry.

removes(keys)[source]

Decorator for cleanuo tasks.

reset_changes()[source]

Reset all pending changes.

validate(func)[source]

Decorator which registers validation functions for a task.

class mlonmcu.setup.task.TaskGraph(names: List[str], dependencies: dict, providers: dict)[source]

Bases: object

Task graph object.

Attributes:
nameslist

list of task names in the graph

dependenciesdict

Dependencies between task artifacts

providersdict

Providers for all the artifacts

Examples
——-
TODO
export_dot(path)[source]

Visualize the task dependency graph.

get_graph() → Tuple[list, list][source]

Get nodes and edges of the task graph.

Returns:
nodeslist

List of edges

edgeslist

List of edge tuples.

get_order() → list[source]

Get execution order of tasks via topological sorting.

class mlonmcu.setup.task.TaskType(value)[source]

Bases: Enum

Enumeration for the task type.

BACKEND = 2
FEATURE = 7
FRAMEWORK = 1
FRONTEND = 5
MISC = 0
OPT = 6
PLATFORM = 8
TARGET = 4
TOOLCHAIN = 3
mlonmcu.setup.task.get_combs(data) → List[dict][source]

Utility which returns combinations of the input data.

Parameters:
datadict

Input dictionary

Returns:
combslist

All combinations of the input data.

Examples

>>> get_combs({"foo": [False, True], "bar": [5, 10]})
[{"foo": False, "bar": 5}, {"foo": False, "bar": 10}, {"foo": True, "bar": 5}, {"foo": True, "bar": 10}]

mlonmcu.setup.utils module

mlonmcu.setup.utils.apply(repo_dir: Path, patch_file: Path)[source]

Helper function for applying a patch to a repository.

Parameters:
repo_dirPath

Clone directory of repository.

patch_filePath

Path to patch file.

mlonmcu.setup.utils.clone(url: str, dest: str | bytes | PathLike, branch: str = '', submodules: list = [], recursive: bool = False, refresh: bool = False)[source]

Helper function for cloning a repository.

Parameters:
urlstr

Clone URL of the repository.

destPath

Destination directory path.

branchstr

Optional branch name or commit reference/tag.

submoduleslist of strings

Only affects when recursive is true. Submodules to be updated. If empty, all submodules will be updated.

recursivebool

If the clone should be done recursively.

refeshbool

Enables switching the url/branch if the repo already exists

mlonmcu.setup.utils.clone_wrapper(cfg: RepoConfig, dest: str | bytes | PathLike, refresh: bool = False)[source]
mlonmcu.setup.utils.cmake(src, *args, debug=False, use_ninja=False, cwd=None, **kwargs)[source]
mlonmcu.setup.utils.copy(src, dest)[source]
mlonmcu.setup.utils.download(url, dest, progress=False)[source]
mlonmcu.setup.utils.download_and_extract(url, archive, dest, progress=False, force=True)[source]
mlonmcu.setup.utils.exec(*args, **kwargs)[source]

Execute a process with the given args and using the given kwards as Popen arguments.

Parameters:
args

The command to be executed.

kwargs

Parameters to be passed to subprocess

mlonmcu.setup.utils.exec_getout(*args, live: bool = False, print_output: bool = False, handle_exit=None, prefix='', **kwargs) → str[source]

Execute a process with the given args and using the given kwards as Popen arguments and return the output.

Parameters:
args

The command to be executed.

livebool

If the stdout should be updated in real time.

print_outputbool

Print the output at the end on non-live mode.

Returns:
output

The text printed to the command line.

mlonmcu.setup.utils.execute(*args: ~typing.List[str], ignore_output: bool = False, live: bool = False, print_func: ~typing.Callable = <built-in function print>, handle_exit: ~typing.Callable | None = None, err_func: ~typing.Callable = <bound method Logger.error of <Logger mlonmcu (INFO)>>, prefix: str = '', **kwargs) → str[source]

Wrapper for running a program in a subprocess.

Parameters:
argslist

The actual command.

ignore_outputbool

Do not get the stdout and stderr or the subprocess.

livebool

Print the output line by line instead of only at the end.

print_funcCallable

Function which should be used to print sysout messages.

handle_exit: Callable

Handler for exit code.

err_funcCallable

Function which should be used to print errors.

kwargs: dict

Arbitrary keyword arguments passed through to the subprocess.

Returns:
outstr

The command line output of the command

mlonmcu.setup.utils.extract(archive, dest, progress=False)[source]
mlonmcu.setup.utils.is_populated(path)[source]
mlonmcu.setup.utils.make(*args, threads=4, use_ninja=False, cwd=None, verbose=False, **kwargs)[source]
mlonmcu.setup.utils.makeDirName(base: str, *args, flags: list | None = None) → str[source]

Creates a directory name based on configuration values.

Using snake_case style.

Parameters:
basestr

Prefix of the filename to be generated.

args

List of tuples of the form: [(True, “foo”), (False, “bar”)]

flagslist

Optional list of additional flags to be added.

Returns:
dirnamestr

The generated directory name

Examples

>>> makeDirName("base", (True, "foo"), (False, "bar"), flags=["flag"])
"base_foo_flag"
mlonmcu.setup.utils.makeFlags(*args)[source]

Resolve tuple-like arguments to a list of string.

Parameters:
args

List of tuples of the form: [(True, “foo”), (False, “bar”)]

Returns:
dirnamestr

The generated directory name

Examples

>>> makeFlags((True, "foo"), (False, "bar"))
["foo"]
mlonmcu.setup.utils.mkdirs(path: str | bytes | PathLike)[source]

Wrapper for os.makedirs which handels the special case where the path already exits.

mlonmcu.setup.utils.move(src, dest)[source]
mlonmcu.setup.utils.patch(path, cwd=None)[source]
mlonmcu.setup.utils.python(*args, **kwargs)[source]

Run a python script with the current interpreter.

mlonmcu.setup.utils.remove(path)[source]

Module contents