Financial Center: open financial data framework

• Financial Center: open financial data framework
• Overview
  • Running Shot
  • Logging Shot
  • Database Shot
  • Flask Demo Shot
• Description
• Installation guide
  • 1. Packages / Dependencies
  • 2. Database
  • 3. Redis
  • 4. Kafka
  • 5. Pip Packages
• Configure FinDy Settings
• Updates:
  • (25 Dec 2021)
  • (01 Mar 2021)
  • (16 Sep 2020):
• How to use
  • Fetch data
  • Read data
• Docker Usage
  • Docker Execute
  • View Log
• To Do
  
  

Overview

Running Shot

Logging Shot

Database Shot

Flask Demo Shot

Description

Based on registry framework, project can easily compat various of thrid-party data sources.

Below shows all data sources that project already supported.

To add more third party data sources, please reference (findy/database/plugins/***) modules for modification.

Installation guide

The FinDy installation consists of setting up the following components:

1. Packages / Dependencies

Command line tools

xcode-select --install #xcode command line tools

Homebrew

/bin/bash -c "$(curl -fsSL https://gitee.com/ineo6/homebrew-install/raw/master/install.sh)"
echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> /Users/huangdon/.zprofile
eval "$(/opt/homebrew/bin/brew shellenv)"
brew install git cmake pkg-config openssl
brew link openssl --force

2. Database

FinDy recommends using a PostgreSQL database. But you can use MySQL too, see MySQL setup guide.

# Install package
brew install postgres
postgres -V
# Start PostgreSQL server
pg_ctl -D /usr/local/var/postgres start
#Or you can start the PostgreSQL server and have it start up at login automatically
brew services start postgresql
# Stop PostgreSQL server
pg_ctl -D /usr/local/var/postgres stop
#To make it stop starting up automatically at login
brew services stop postgresql
# Login to PostgreSQL
psql -U XXX(superuser role) -d postgres
# Create a user for FinDy
CREATE USER postgres with NOSUPERUSER CREATEDB CREATEROLE INHERIT LOGIN;
# Create the FinDy production database & grant all privileges on database
CREATE DATABASE findy OWNER postgres;
# Quit the database session
\q
# Try connecting to the new database with the new user
sudo -u git -H psql -d findy
# tuning postgres (optional)
cpan -i DBD::Pg
cd pgtune
chmod +x postgresqltuner.pl
./pg_tune.sh

For MacOS, if you wish to enlarge system share memory from 4MB to 512 MB, make following:

1. Disable SIP: Restart computer to Recovery mode, Select “Terminal” and run command csrutil disable, then restart computer
2. Create /Library/LaunchDaemons/com.startup.sysctl.plist

      <?xml version="1.0" encoding="UTF-8"?>
      <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
      <!-- DISABLE SIP TO USE: macOS Recovery > Utilities > Terminal > `csrutil disable` > Reboot -->
      <plist version="1.0">
          <dict>
              <key>Label</key>
              <string>com.startup.sysctl</string>
              <key>LaunchOnlyOnce</key>
              <true></true>
              <key>StandardErrorPath</key>
                  <string>/private/tmp/sysctl.err</string>
              <key>StandardOutPath</key>
                  <string>/private/tmp/sysctl.out</string>
              <key>ProgramArguments</key>
              <array>
                  <string>/usr/sbin/sysctl</string>
                  <string>-w</string>
                  <string>kern.sysv.shmmax=16777216</string>
                  <string>kern.sysv.shmmin=1</string>
                  <string>kern.sysv.shmmni=128</string>
                  <string>kern.sysv.shmseg=512</string>
                  <string>kern.sysv.shmall=4096</string>
              </array>
              <key>RunAtLoad</key>
              <true></true>
          </dict>
      </plist>

3. Load the PLIST after a few housekeeping items

    # sanity check
    sysctl -a | grep shm
    # set permissions
    sudo chown root:wheel /Library/LaunchDaemons/com.startup.sysctl.plist
    # validate key-value pairs
    plutil /Library/LaunchDaemons/com.startup.sysctl.plist
    # load plist
    sudo launchctl bootstrap system /Library/LaunchDaemons/com.startup.sysctl.plist
    # check logs
    tail -f /tmp/sysctl.out
    tail -f /tmp/sysctl.err
    # recheck sysctl values
    sysctl -a | grep shm

3. Redis

brew install redis
ln -sfv /usr/local/opt/redis/*.plist ~/Library/LaunchAgents

Redis config is located in /usr/local/etc/redis.conf. Make a copy:

cp /usr/local/etc/redis.conf /usr/local/etc/redis.conf.orig

Disable Redis listening on TCP by setting ‘port’ to 0

sed 's/^port .*/port 0/' /usr/local/etc/redis.conf.orig | sudo tee /usr/local/etc/redis.conf

Edit file (nano /usr/local/etc/redis.conf) and uncomment:

unixsocket /tmp/redis.sock
unixsocketperm 777

Start Redis

launchctl load ~/Library/LaunchAgents/homebrew.mxcl.redis.plist

4. Kafka

# Install package
brew install zookeeper
brew install kafka
modify file /opt/homebrew/etc/kafka/server.properties, active line listeners=PLAINTEXT://:9092 as below:
############################# Socket Server Settings #############################
# The address the socket server listens on. It will get the value returned from 
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://localhost:9092
# Start kafka server
brew services start zookeeper
brew services start kafka
pip install kafka_python

5. Pip Packages

pip install -r requirements_mac_arm64.txt
# optional, generate requirement.txt file from project imports
$ pipreqs ./ --force

Configure FinDy Settings

Default (config.json) setting

{
  "version": "0.0.2",
  "debug": 0,
  "processes": 1,
  "batch_size": 10000,
  "db_name": "findy",
  "db_host": "192.168.1.133",
  "db_port": "15432",
  "db_user": "",
  "db_pass": "",
  "redis_pass": "",
  "kafka": "localhost:9092",
  "jq_username": "",
  "jq_password": "",
  "tushare_token": "",
  "trade_host": "127.0.0.1",
  "trade_port": "11111",
  "trade_socket_port": "33333",
  "trade_socket_key": "",
  "http_proxy": "127.0.0.1:1087",
  "https_proxy": "127.0.0.1:1087",
  "smtp_host": "smtpdm.aliyun.com",
  "smtp_port": "80",
  "email_username": "",
  "email_password": "",
  "wechat_app_id": "",
  "wechat_app_secrect": "",
  "wechat_agent_id": ""
}

change database host address to following setting if you install your database locally:

  "db_host": "127.0.0.1",
  "db_port": "5432",

set database user and password to your custom settings:

  "db_user": "xxx",
  "db_pass": "xxx",

chinese stock market user are required to obtain joinquant and tushare authentication.

JoinQuant

TuShare

Updates:

(25 Dec 2021)

• rewrite yahoo finance lib (yfinance) to support aiohttp requests
• remove proxy pool
• add wechat notify (see utils/notify.py for more details)
• dev/async_db branch: add async sqlalchemy ORM
• many bugfix

(01 Mar 2021)

• database session decoupling
• entity provider removed
• database querying speed boost up
• asyncio logic framework (implementations requires rewrite request logic)
• multiprocessing logic rewrite, use processes to replace pool
• add proxy pool (see proxy_pool)
• add baostock thrid party data source

(16 Sep 2020):

• Switch Sql to Postgresql, which allow remote database access, separate fetching and analysis models to two individal project(see FinanceAnalysis project for more detail).
• Centralized http request interface，add Session and retry logic (Rxpy may be involved to rewrite request model pretty soon).
• multiprocessing is involved to boost up request speed.
• request progress display.
• auto joinquant account switching (banned)
• log to console -> log to file.
• US stock market data.
• global timezone support.
• tiny bug fix.

How to use

Fetch data

from findy.interface import Region
from findy.interface.fetch import fetching
fetching(Region.CHN)  # Region.US

Read data

from datetime import datetime, timedelta
from findy.interface import Region, Provider, EntityType, SP_500_TICKER
from findy.interface.reader import DataReader
from findy.database.schema.meta.stock_meta import Stock, StockDetail
from findy.database.schema.quotes.stock.stock_1d_kdata import Stock1dKdata
from findy.database.schema.register import get_entity_schema_by_type
from findy.database.context import get_db_session
from findy.database.quote import get_entities
end = datetime.now()
start = end - timedelta(days=365)
entity_schema = get_entity_schema_by_type(EntityType.StockDetail)
db_session = get_db_session(Region.US, Provider.Yahoo, entity_schema)
entities, column_names = get_entities(
        region=Region.US,
        provider=Provider.Yahoo,
        entity_schema=entity_schema,
        db_session=db_session,
        codes=SP_500_TICKER,
        columns=[StockDetail.code],
        filters=[StockDetail.market_cap > 10_000_000_000])
codes_cap = [entity.code for entity in entities]
stock_reader = DataReader(region=Region.US,
                          provider=Provider.Yahoo,
                          data_schema=Stock1dKdata,
                          entity_schema=Stock,
                          codes=codes_cap,
                          start_timestamp=start.strftime('%Y-%m-%d'),
                          end_timestamp=end.strftime('%Y-%m-%d'))
stock_reader.load_data()
df = stock_reader.data_df.reset_index(drop=True)
gb = df.groupby('code', sort=False)
stocks = [(code, gb.get_group(code)) for code in gb.groups]

Docker Usage

Docker Execute

docker-compose stop; docker-compose rm -f ; docker-compose build --no-cache
docker-compose up -d

View Log

docker-compose logs -f

To Do

1. rewrite vendor/baostock package to support async logic
2. add more 3th-party data source packages
3. dev/async_db branch optimization