[Deprecated] Note this repo is no longer maintained, and is archived for historical context only.
This repository is the home for code related to the 2 Degrees Investing Data Warehouse (DW).
Note that this project uses docker, which is a tool which allows us to easily package this project, so new collaborators don’t need to worry about setup or installation comands, or about tthis project breaking something else on their computer.
For more informattion, ROpenSci has a nice introduction to docker for R users.
To get started with development on this project, and run the import application, simply run on the terminal
docker-compose up -d
This will create two docker containers, one containing a development database (usually called twodii-datawarehouse_db_1) and one containing the python application code (twodii-datawarehouse_app_1).
Initially, the database is an empty PostgreSQL database.
To apply the twodii-datawarehouse structure to it, we must run the application code in the app container.
To enter the app container, run
docker attach <<name of app container>>
Replace <<name of app container>> with the container that was started by docker (usually twodii-datawarehouse_app_1).
This will put your terminal into the shell inside the docker container, which has all the python code for updating the database and importing data files.
The prompt inside the docker contianer will look somthing like:
root@2a77e1b22478:/usr/src/app#
where the @2a77e1b22478 component will change, as that is the container’s internal ID.
From this prompt run
# to only create SQL tables and SQL functionspython twodii-datawarehouse.py -v -m# or# for SQL creation and importing data files:python twodii-datawarehouse.py -v
Both database migration management and data import are idempotent, meaning that running the same command repeatedly is a safe operation, and will not cause errors.
You can exit back to the host shell by pressing Ctrl + P then Ctrl + Q, which will leave the app container running. or by typing exit at the container’s prompt, which will stop the app container, but leave the database container running.
To “turn off” the docker containers, from the host shell run:
docker-compose down
The development database is persisted across runs using docker volumes.
The volume can be removed (and all data inside deleted) using
docker-compose down --volumes
The files for importing are exposed to the app conatiner through volume mapping, and on the host file system, they are located in test/data_files.
This directory is excluded from git, so you can put files in this directory without worrying about accidentally comitting them.
