项目作者: pdonorio

项目描述 :
Scaling docker images in an Openstack cloud with Python
高级语言: Jupyter Notebook
项目地址: git://github.com/pdonorio/cloudscale.git
创建时间: 2015-11-09T12:46:34Z
项目社区:https://github.com/pdonorio/cloudscale

开源协议:

下载


Scaling Docker containers inside your cloud

A little introduction

This project has born from the need to provide the same data analytics environment based on Python to a certain number of student of our training courses. We choosed for obviuosly reason Jupyter Notebooks as the centric technologies.

At some point we started using Docker as virtual environment to reproduce the training resources.
We created public Docker images to reproduce all Python libraries (Pydata: numpy, scipy, pandas, matplotlib, searborn, bokeh, cython) and/or Data analytics frameworks (Hadoop, Spark, Machine learning) behind a password protected Jupyter server.

But installing and using Docker on both our labs workstation and student laptops required quite a work, so we decided to spawn a Docker container on our new Openstack cloud for each student.

The project

I tried to stay as generic as possible writing this code.

The main command is themachine which creates a certain number of virtual nodes on Openstack,
launch Docker on them, configure the engines to listen on a certain port, create a swarm cluster
and finally make the docker engines join the cluster, using the first node as the cluster master.

Moreover on this swarm cluster we automatically launch the same image for as many students as availabe in a CSV file,
providing at completion a table with one row per student, resource (host+port of the jupyter server) and password.

Available commands

  1. $ git clone https://github.com/pdonorio/cloudscale.git
  2. $ cd cloudscale
  3. $ invoke -l
  4. Available tasks:
  5. com_containers Execute a command on all running containers of your swarm cluster
  6. com_node Execute a command on all running nodes of one driver
  7. containers_reset List all machine with a driver, and clean up their containers
  8. driver_reset Remove PERMANENTLY all machine within a driver class
  9. machine_new A task to add a docker machine - on virtualbox
  10. machine_rm A task to remove an existing machine - on virtualbox
  11. remote_com Execute command to host via pythonic ssh (auth: password or key)
  12. themachine Launch openstack cluster + replicate docker image

The courses

Material used for courses is available on our lecture dedicated repository