项目作者: SupraSummus

项目描述 :
Mount IPFS directory as local FS.
高级语言: Python
项目地址: git://github.com/SupraSummus/ipfs-api-mount.git
创建时间: 2017-11-07T20:51:46Z
项目社区:https://github.com/SupraSummus/ipfs-api-mount

开源协议:MIT License

下载


ipfs-api-mount

Build Status
codecov

Mount IPFS directory as local FS.

go-ipfs daemon has this function but as of version 0.9.1 it’s slow.
ipfs-api-mount aims to be more efficient. For sequential access to
random data it’s ~3 times slower than ipfs cat but also ~20 times
faster than cating files mounted by go-ipfs.

It’s supposed that FS mounted by go-ipfs daemon is slow because of file
structure being accessed in every read. By adding caching one can improve
performance a lot.

How to use

Install package …

  1. pip install ipfs-api-mount

… and then

  1. mkdir a_dir
  2. ipfs-api-mount QmXKqqUymTQpEM89M15G23wot8g7n1qVYQQ6vVCpEofYSe a_dir &
  3. ls a_dir
  4. # aaa bbb

To unmount

  1. fusermount -u a_dir

Mount whole IPFS at once

Apart from mounting one specified CID you can also mount whole IPFS namespace. This is similar to ipfs mount provided in go-ipfs.

  1. mkdir a_dir
  2. ipfs-api-mount-whole a_dir &
  3. ls a_dir/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco
  4. # - I index.html M wiki

Python-level use

Mountpoints can be created inside python programs

  1. import os
  2. import ipfshttpclient
  3. from ipfs_api_mount.ipfs_mounted import ipfs_mounted
  4. from ipfs_api_mount.fuse_operations import IPFSOperations
  5. with ipfs_mounted(IPFSOperations('QmSomeHash', ipfshttpclient.connect())) as mountpoint:
  6. print(os.listdir(mountpoint))

Benchmark

Try it yourself and run ./benchamrk [number of Mbytes].

Example output:

  1. ipfs version 0.9.1
  2. creating 100MB of random data and uploading to ipfs ...
  3. 100MB of data at:
  4. QmTnYkR6FBajXhY6bmRnTtuQ2MA8f66BoW2pFu2Z6rParg
  5. QmaiV6qpn4k4WEy6Ge7p2s4rAMYTY6hd77dSioq4JUUaLU/data
  6. ### ipfs cat QmTnYkR6FBajXhY6bmRnTtuQ2MA8f66BoW2pFu2Z6rParg
  7. 4f63d1c2056a8c33b43dc0c2a107a1ec3d679ad7fc1b08ce96526a10c9c458d7 -
  8. real 0m0.686s
  9. user 0m0.867s
  10. sys 0m0.198s
  11. ### ipfs-api-mount QmaiV6qpn4k4WEy6Ge7p2s4rAMYTY6hd77dSioq4JUUaLU /tmp/tmp.7CyBemuY5Q
  12. ### cat /tmp/tmp.7CyBemuY5Q/data
  13. 4f63d1c2056a8c33b43dc0c2a107a1ec3d679ad7fc1b08ce96526a10c9c458d7 -
  14. real 0m2.387s
  15. user 0m0.495s
  16. sys 0m0.145s
  17. ### cat /ipfs/QmTnYkR6FBajXhY6bmRnTtuQ2MA8f66BoW2pFu2Z6rParg
  18. 4f63d1c2056a8c33b43dc0c2a107a1ec3d679ad7fc1b08ce96526a10c9c458d7 -
  19. real 0m59.976s
  20. user 0m2.975s
  21. sys 0m1.166s

More in depth description

ipfs-api-mount uses node API for listing directories and reading
objects. Objects are decoded and file structure is created localy (not
in IPFS node). Caching is added on objects level. In case of nonlinear
file access with many small reads there is a risk of cache thrashing.
If this occurs performance will be much worst than without cache. When
using the command you can adjust cache size to get best performance (but
for cache thrashing there is little hope).

Caching options

There are four cache parameters:

  • --ls-cache-size - how many directory content lists are cached. Increase this if you want subsequent ls to be faster.
  • --block-cache-size - how many data blocks are cached. This cache needs to be bigger if you are doing sequential reads in many scattered places at once (in single or multiple files). It doesn’t affect speed of reading the same spot for the second time, because this is handled by FUSE (kernel_cache option). This cache is memory-intensive - takes up to 1MB per entry.
  • --link-cache-size - Files on IPFS are trees of blocks. This cache keeps the tree structure. Increase this cache’s size if you are reading many big files simultanously (depth of a single tree is generally <4, but many of them can overflow the cache). It doesn’t affect speed of reading previously read data - this is handled by FUSE (kernel_cache option).
  • --attr-cache-size - cache related to file and directory attributes. This needs to be bigger if you are reading many files attributes, and you want subsequent reads to be faster. For example, if you do ls -l (-l will call stat() on every file) on a large directory and you want second ls -l to be faster, you need to set this cache to be bigger than number of files in the directory.

Hope that makes sense ;-)

See also