项目作者: ResearchObject

项目描述 :
Python library for RO-Crate
高级语言: Jupyter Notebook
项目地址: git://github.com/ResearchObject/ro-crate-py.git
创建时间: 2019-10-21T15:44:52Z
项目社区:https://github.com/ResearchObject/ro-crate-py

开源协议:Apache License 2.0

下载


Python package Upload Python Package PyPI version DOI

ro-crate-py is a Python library to create and consume Research Object Crates. It currently supports the RO-Crate 1.1 specification.

Installation

ro-crate-py requires Python 3.9 or later. The easiest way to install is via pip:

  1. pip install rocrate

To install the package with support for converting Galaxy workflows to CWL:

  1. pip install rocrate[ga2cwl]

To install manually from this code base (e.g., to try the latest development revision):

  1. git clone https://github.com/ResearchObject/ro-crate-py
  2. cd ro-crate-py
  3. pip install .

Usage

Creating an RO-Crate

In its simplest form, an RO-Crate is a directory tree with an ro-crate-metadata.json file at the top level. This file contains metadata about the other files and directories, represented by data entities. These metadata consist both of properties of the data entities themselves and of other, non-digital entities called contextual entities. A contextual entity can represent, for instance, a person, an organization or an event.

Suppose Alice and Bob worked on a research task together, which resulted in a manuscript written by both; additionally, Alice prepared a spreadsheet containing the experimental data, which Bob used to generate a diagram. We will create placeholder files for these documents:

  1. mkdir exp
  2. touch exp/paper.pdf
  3. touch exp/results.csv
  4. touch exp/diagram.svg

Let’s make an RO-Crate to package all this:

  1. from rocrate.rocrate import ROCrate
  2. crate = ROCrate()
  3. paper = crate.add_file("exp/paper.pdf", properties={
  4. "name": "manuscript",
  5. "encodingFormat": "application/pdf"
  6. })
  7. table = crate.add_file("exp/results.csv", properties={
  8. "name": "experimental data",
  9. "encodingFormat": "text/csv"
  10. })
  11. diagram = crate.add_file("exp/diagram.svg", dest_path="images/figure.svg", properties={
  12. "name": "bar chart",
  13. "encodingFormat": "image/svg+xml"
  14. })

The dest_path argument is used to specify the relative path of the file with respect to the crate’s directory (which will be determined when the crate is written). Note that the first two add_file calls do not specify dest_path: in this case, it will be set to the source file’s basename ("paper.pdf" in the first case), so the file will be at the crate’s top level when it is written.

We’ve started by adding the data entities. Now we need contextual entities to represent Alice and Bob:

  1. from rocrate.model.person import Person
  2. alice_id = "https://orcid.org/0000-0000-0000-0000"
  3. bob_id = "https://orcid.org/0000-0000-0000-0001"
  4. alice = crate.add(Person(crate, alice_id, properties={
  5. "name": "Alice Doe",
  6. "affiliation": "University of Flatland"
  7. }))
  8. bob = crate.add(Person(crate, bob_id, properties={
  9. "name": "Bob Doe",
  10. "affiliation": "University of Flatland"
  11. }))

At this point, we have a representation of the various entities. Now we need to express the relationships between them. This is done by adding properties that reference other entities:

  1. paper["author"] = [alice, bob]
  2. table["author"] = alice
  3. diagram["author"] = bob

You can also add whole directories together with their contents. In an RO-Crate, a directory is represented by the Dataset entity. Create a directory with some placeholder files:

  1. mkdir exp/logs
  2. touch exp/logs/log1.txt
  3. touch exp/logs/log2.txt

Now add it to the crate:

  1. logs = crate.add_dataset("exp/logs")

Finally, we serialize the crate to disk:

  1. crate.write("exp_crate")

Now the exp_crate directory should contain copies of all the files we added and an ro-crate-metadata.json file with a JSON-LD representation of the entities and relationships we created, with the following layout:

  1. exp_crate/
  2. |-- images/
  3. | `-- figure.svg
  4. |-- logs/
  5. | |-- log1.txt
  6. | `-- log2.txt
  7. |-- paper.pdf
  8. |-- results.csv
  9. `-- ro-crate-metadata.json

Exploring the exp_crate directory, we see that all files and directories contained in exp/logs have been added recursively to the crate. However, in the ro-crate-metadata.json file, only the top level Dataset with @id "exp/logs" is listed. This is because we used crate.add_dataset("exp/logs") rather than adding every file individually. There is no requirement to represent every file and folder within the crate in the ro-crate-metadata.json file. If you do want to add files and directories recursively to the metadata, use crate.add_tree instead of crate.add_dataset (but note that it only works on local directory trees).

Some applications and services support RO-Crates stored as archives. To save the crate in zip format, use write_zip:

  1. crate.write_zip("exp_crate.zip")

Appending elements to property values

What ro-crate-py entities actually store is their JSON representation:

  1. paper.properties()
  1. {
  2. "@id": "paper.pdf",
  3. "@type": "File",
  4. "name": "manuscript",
  5. "encodingFormat": "application/pdf",
  6. "author": [
  7. {"@id": "https://orcid.org/0000-0000-0000-0000"},
  8. {"@id": "https://orcid.org/0000-0000-0000-0001"},
  9. ]
  10. }

When paper["author"] is accessed, a new list containing the alice and bob entities is generated on the fly. For this reason, calling append on paper["author"] won’t actually modify the paper entity in any way. To add an author, use the append_to method instead:

  1. donald = crate.add(Person(crate, "https://en.wikipedia.org/wiki/Donald_Duck", properties={
  2. "name": "Donald Duck"
  3. }))
  4. paper.append_to("author", donald)

Note that append_to also works if the property to be updated is missing or has only one value:

  1. for n in "Mickey_Mouse", "Scrooge_McDuck":
  2. p = crate.add(Person(crate, f"https://en.wikipedia.org/wiki/{n}"))
  3. donald.append_to("follows", p)

Handling of special characters

Since RO-Crate entity identifiers are URIs (relative or absolute), special characters in them must be percent-encoded. When adding data entities from the local file system, this is handled automatically by the library:

  1. >>> crate = ROCrate()
  2. >>> d = crate.add_dataset("read_crate/a b")
  3. >>> d.id
  4. 'a%20b/'
  5. >>> f = crate.add_file("read_crate/a b/c d.txt", dest_path="data/a b/c d.txt")
  6. >>> f.id
  7. 'data/a%20b/c%20d.txt'

When adding an entity whose identifier is an absolute URI (see next section), on the other hand, the identifier provided by the user is expected to be a valid URI, in particular with any special characters in path components already percent-encoded.

Adding remote entities

Data entities can also be remote:

  1. input_data = crate.add_file("http://example.org/exp_data.zip")

By default the file won’t be downloaded, and will be referenced by its URI in the serialized crate:

  1. {
  2. "@id": "http://example.org/exp_data.zip",
  3. "@type": "File"
  4. },

If you add fetch_remote=True to the add_file call, however, the library (when crate.write is called) will try to download the file and include it in the output crate.

Another option that influences the behavior when dealing with remote entities is validate_url, also False by default: if it’s set to True, when the crate is serialized, the library will try to open the URL to add / update metadata bits such as the content’s length and format (but it won’t try to download the file unless fetch_remote is also set).

Adding entities with an arbitrary type

An entity can be of any type listed in the RO-Crate context. However, only a few of them have a counterpart (e.g., File) in the library’s class hierarchy (either because they are very common or because they are associated with specific functionality that can be conveniently embedded in the class implementation). In other cases, you can explicitly pass the type via the properties argument:

  1. from rocrate.model.contextentity import ContextEntity
  2. hackathon = crate.add(ContextEntity(crate, "#bh2021", properties={
  3. "@type": "Hackathon",
  4. "name": "Biohackathon 2021",
  5. "location": "Barcelona, Spain",
  6. "startDate": "2021-11-08",
  7. "endDate": "2021-11-12"
  8. }))

Note that entities can have multiple types, e.g.:

  1. "@type" = ["File", "SoftwareSourceCode"]

Consuming an RO-Crate

An existing RO-Crate package can be loaded from a directory or zip file:

  1. crate = ROCrate('exp_crate') # or ROCrate('exp_crate.zip')
  2. for e in crate.get_entities():
  3. print(e.id, e.type)
  1. ./ Dataset
  2. ro-crate-metadata.json CreativeWork
  3. paper.pdf File
  4. results.csv File
  5. images/figure.svg File
  6. https://orcid.org/0000-0000-0000-0000 Person
  7. https://orcid.org/0000-0000-0000-0001 Person

The first two entities shown in the output are the root data entity and the metadata file descriptor, respectively. The former represents the whole crate, while the latter represents the metadata file. These are special entities managed by the ROCrate object, and are always present. The other entities are the ones we added in the section on RO-Crate creation.

As shown above, get_entities allows to iterate over all entities in the crate. You can also access only data entities with crate.data_entities and only contextual entities with crate.contextual_entities. For instance:

  1. for e in crate.data_entities:
  2. author = e.get("author")
  3. if not author:
  4. continue
  5. elif isinstance(author, list):
  6. print(e.id, [p["name"] for p in author])
  7. else:
  8. print(e.id, repr(author["name"]))
  1. paper.pdf ['Alice Doe', 'Bob Doe']
  2. results.csv 'Alice Doe'
  3. images/figure.svg 'Bob Doe'

You can fetch an entity by its @id as follows:

  1. article = crate.dereference("paper.pdf")

Advanced features

Modifying the crate from JSON-LD dictionaries

The add_jsonld method allows to add a contextual entity directly from a
JSON-LD dictionary containing at least the @id and @type keys:

  1. crate.add_jsonld({
  2. "@id": "https://orcid.org/0000-0000-0000-0000",
  3. "@type": "Person",
  4. "name": "Alice Doe"
  5. })

Existing entities can be updated from JSON-LD dictionaries via update_jsonld:

  1. crate.update_jsonld({
  2. "@id": "https://orcid.org/0000-0000-0000-0000",
  3. "name": "Alice K. Doe"
  4. })

There is also an add_or_update_jsonld method that adds the entity if it’s
not already in the crate and updates it if it already exists (note that, when
updating, the @type key is ignored). This allows to “patch” an RO-Crate from
a JSON-LD file. For instance, suppose you have the following patch.json file:

  1. {
  2. "@graph": [
  3. {
  4. "@id": "./",
  5. "author": {"@id": "https://orcid.org/0000-0000-0000-0001"}
  6. },
  7. {
  8. "@id": "https://orcid.org/0000-0000-0000-0001",
  9. "@type": "Person",
  10. "name": "Bob Doe"
  11. }
  12. ]
  13. }

Then the following sets Bob as the author of the crate according to the above
file:

  1. crate = ROCrate("temp-crate")
  2. with open("patch.json") as f:
  3. json_data = json.load(f)
  4. for d in json_data.get("@graph", []):
  5. crate.add_or_update_jsonld(d)

Command Line Interface

ro-crate-py includes a hierarchical command line interface: the rocrate tool. rocrate is the top-level command, while specific functionalities are provided via sub-commands. Currently, the tool allows to initialize a directory tree as an RO-Crate (rocrate init) and to modify the metadata of an existing RO-Crate (rocrate add).

  1. $ rocrate --help
  2. Usage: rocrate [OPTIONS] COMMAND [ARGS]...
  3. Options:
  4. --help Show this message and exit.
  5. Commands:
  6. add
  7. init
  8. write-zip

Crate initialization

The rocrate init command explores a directory tree and generates an RO-Crate metadata file (ro-crate-metadata.json) listing all files and directories as File and Dataset entities, respectively.

  1. $ rocrate init --help
  2. Usage: rocrate init [OPTIONS]
  3. Options:
  4. --gen-preview Generate a HTML preview file for the crate.
  5. -e, --exclude NAME Exclude files or directories from the metadata file.
  6. NAME may be a single name or a comma-separated list of
  7. names.
  8. -c, --crate-dir PATH The path to the root data entity of the crate.
  9. Defaults to the current working directory.
  10. --help Show this message and exit.

The command acts on the current directory, unless the -c option is specified. The metadata file is added (overwritten if present) to the directory at the top level, turning it into an RO-Crate.

Adding items to the crate

The rocrate add command allows to add file, datasets (directories), workflows and other entity types (currently testing-related metadata) to an RO-Crate:

  1. $ rocrate add --help
  2. Usage: rocrate add [OPTIONS] COMMAND [ARGS]...
  3. Options:
  4. --help Show this message and exit.
  5. Commands:
  6. dataset
  7. file
  8. test-definition
  9. test-instance
  10. test-suite
  11. workflow

Note that data entities (e.g., workflows) must already be present in the directory tree: the effect of the command is to register them in the metadata file.

Example

  1. # From the ro-crate-py repository root
  2. cd test/test-data/ro-crate-galaxy-sortchangecase

This directory is already an RO-Crate. Delete the metadata file to get a plain directory tree:

  1. rm ro-crate-metadata.json

Now the directory tree contains several files and directories, including a Galaxy workflow and a Planemo test file, but it’s not an RO-Crate since there is no metadata file. Initialize the crate:

  1. rocrate init

This creates an ro-crate-metadata.json file that lists files and directories rooted at the current directory. Note that the Galaxy workflow is listed as a plain File:

  1. {
  2. "@id": "sort-and-change-case.ga",
  3. "@type": "File"
  4. }

To register the workflow as a ComputationalWorkflow:

  1. rocrate add workflow -l galaxy sort-and-change-case.ga

Now the workflow has a type of ["File", "SoftwareSourceCode", "ComputationalWorkflow"] and points to a ComputerLanguage entity that represents the Galaxy workflow language. Also, the workflow is listed as the crate’s mainEntity (this is required by the Workflow RO-Crate profile, a subtype of RO-Crate which provides extra specifications for workflow metadata).

To add workflow testing metadata to the crate:

  1. rocrate add test-suite -i '#test1'
  2. rocrate add test-instance '#test1' http://example.com -r jobs -i '#test1_1'
  3. rocrate add test-definition '#test1' test/test1/sort-and-change-case-test.yml -e planemo -v '>=0.70'

To add files or directories after crate initialization:

  1. cp ../sample_file.txt .
  2. rocrate add file sample_file.txt -P name=sample -P description="Sample file"
  3. cp -r ../test_add_dir .
  4. rocrate add dataset test_add_dir

The above example also shows how to set arbitrary properties for the entity with -P. This is supported by most rocrate add subcommands.

  1. $ rocrate add workflow --help
  2. Usage: rocrate add workflow [OPTIONS] PATH
  3. Options:
  4. -l, --language [cwl|galaxy|knime|nextflow|snakemake|compss|autosubmit]
  5. The workflow language.
  6. -c, --crate-dir PATH The path to the root data entity of the
  7. crate. Defaults to the current working
  8. directory.
  9. -P, --property KEY=VALUE Add an additional property to the metadata
  10. for this entity. Can be used multiple times
  11. to set multiple properties.
  12. --help Show this message and exit.

License

  • Copyright 2019-2025 The University of Manchester, UK
  • Copyright 2020-2025 Vlaams Instituut voor Biotechnologie (VIB), BE
  • Copyright 2020-2025 Barcelona Supercomputing Center (BSC), ES
  • Copyright 2020-2025 Center for Advanced Studies, Research and Development in Sardinia (CRS4), IT
  • Copyright 2022-2025 École Polytechnique Fédérale de Lausanne, CH
  • Copyright 2024-2025 Data Centre, SciLifeLab, SE
  • Copyright 2024-2025 National Institute of Informatics (NII), JP
  • Copyright 2025 Senckenberg Society for Nature Research (SGN), DE
  • Copyright 2025 European Molecular Biology Laboratory (EMBL), Heidelberg, DE

Licensed under the
Apache License, version 2.0 https://www.apache.org/licenses/LICENSE-2.0,
see the file LICENSE.txt for details.

Cite as

DOI

The above DOI corresponds to the latest versioned release as published to Zenodo, where you will find all earlier releases.
To cite ro-crate-py independent of version, use https://doi.org/10.5281/zenodo.3956493, which will always redirect to the latest release.

You may also be interested in the paper Packaging research artefacts with RO-Crate.