alkymi ⚗️

Alkymi is a pure Python (3.7+) library for describing and executing tasks and pipelines with built-in caching and
conditional evaluation based on checksums.

Alkymi is easy to install, simple to use, and has very few dependencies outside of Python’s standard library. The code
is cross-platform, and allows you to write your pipelines once and deploy to multiple operating systems (tested on
Linux, Windows and Mac).

Documentation, including a quickstart guide, is provided here.

Features

• Easily define complex data pipelines as decorated Python functions
  • This allows you to run linting, type checking, etc. on your data pipelines
• Return values are automatically cached to disk, regardless of type
• Efficiently checks if pipeline is up-to-date
  • Checks if external files have changed, bound functions have changed or if pipeline dependencies have changed
• No domain specific language (DSL) or CLI tool, just regular Python
  • Supports caching and conditional evaluation in Jupyter Notebooks
• Cross-platform - works on Linux, Windows and Mac
• Expose recipes as a command-line interface (CLI) using alkymi’s
  Lab type

Sample Usage

For examples of how to use alkymi, see the
quickstart guide.

Example code:

import numpy as np
import alkymi as alk
@alk.recipe()
def long_running_task() -> np.ndarray:
    # Perform expensive computation here ...
    hard_to_compute_result = np.array([42])
    # Return value will be automatically cached to disk
    return hard_to_compute_result
result = long_running_task.brew()  # == np.ndarray([42])

Or one of the examples, e.g. MNIST.

Installation

Install via pip:
```shell script
pip install —user alkymi

Or see the [Installation page](https://alkymi.readthedocs.io/en/latest/getting_started/installation.html).
### Testing
After installing, you can run the test suite (use the `lint`, `coverage` and `type_check` recipes to perform those
actions):
```shell script
python3 labfile.py brew test

License

alkymi is licensed under The MIT License as found in the LICENSE.md file

Upcoming Features

The following features are being considered for future implementation:

• Type annotations propagated from bound functions to recipes
• Support for call/type checking all recipes (e.g. by adding a check command to Lab)
• Cache maintenance functionality

Known Issues

• alkymi currently doesn’t check custom objects for altered external files when computing cleanliness (e.g. MyClass
  has a self._some_path that points to a file somewhere outside alkymi’s internal cache)
• alk.foreach() currently only supports enumerable inputs of type List or Dict
• Recipes marked transient will always be dirty, and thus always require reevaluation. This functionality should be
  replaced by a proper means of creating recipes that don’t cache outputs, but only run when needed to provide inputs for
  downstream recipes