项目作者: CodeRecipe-dev

项目描述 :
Mode info at: https://coderecipe.ai/architectures/86530220
高级语言: Python
项目地址: git://github.com/CodeRecipe-dev/bigdata-on-aws.git
创建时间: 2019-05-30T16:30:23Z
项目社区:https://github.com/CodeRecipe-dev/bigdata-on-aws

开源协议:MIT License

下载


Building up an AWS big data pipeline

by: zaicheng wang

Mode info at: https://coderecipe.ai/architectures/86530220

Prerequisites

Make sure you have AWS access key and secrete keys setup locally, following this video here

Download the code locally

  1. serverless create --template-url {} --path AWS-bigdata-pipeline

Deploy to the cloud

  1. cd AWS-bigdata-pipeline
  2. npm install serverless
  3. npm install serverless-pseudo-parameters
  4. serverless deploy --stage <your-stage-name>

Put sample data to firehose using AWS CLI

  1. cd AWS-bigdata-pipeline
  2. aws firehose put-record --delivery-stream-name data-pipeline-dev-Deliverystream-XTJ8GY3H9DUH --record file://data/small.json

Run Sql query on Athena

  1. aws athena start-query-execution --cli-input-json file://athena-cli/start-query-execution.json
  2. aws athena get-query-results --query-execution-id {query id from last command}

Create Quicksight Dashboard

https://docs.aws.amazon.com/quicksight/latest/user/example-create-a-dashboard.html