项目作者: sensefly-sa

项目描述 :
Simple DynamoDB backup to S3
高级语言: Kotlin
项目地址: git://github.com/sensefly-sa/dynamodb-to-s3.git
创建时间: 2017-11-01T14:32:07Z
项目社区:https://github.com/sensefly-sa/dynamodb-to-s3

开源协议:BSD 3-Clause "New" or "Revised" License

下载


Backup DynamoDB to S3

Build Status
codecov
Download
Pull

Simple DynamoDB dump to S3.

Read & write rates are limited by applying a ratio (read-percentage/write-percentage) to table’s provisioned capacities.

Usage

  1. $ java -jar dynamodb-to-s3-<version>.jar --help
  2. Usage: <main class> [options] [command] [command options]
  3. Options:
  4. --help
  5. Print usage
  6. Commands:
  7. backup Backup DynamoDB tables to S3 bucket.
  8. Usage: backup [options]
  9. Options:
  10. * -t, --table
  11. Table to backup to S3. Comma-separated list to backup multiple tables or repeat this param.
  12. Default: []
  13. * -b, --bucket
  14. Destination S3 bucket.
  15. Default: <empty string>
  16. -c, --cron
  17. Cron pattern. (http://www.manpagez.com/man/5/crontab/)
  18. --read-percentage
  19. Read percentage based on current table capacity. Cannot be used with '--read-capacity'.
  20. --read-capacity
  21. Read capacity (useful if auto scaling enabled). Cannot be used with '--read-percentage'.
  22. -p, --pattern
  23. Destination file path pattern.
  24. Default: yyyy/MM/dd
  25. -n, --namespace
  26. Cloudwatch namespace to send metrics.
  27. --jvmMetrics
  28. Collect JVM metrics
  29. Default: false
  30. restore Restore DynamoDB table from json file hosted on S3.
  31. Usage: restore [options]
  32. Options:
  33. * -t, --table
  34. Table to restore.
  35. Default: <empty string>
  36. * -s, --source
  37. S3 URI to a JSON backup file
  38. (s3://my-bucket/folder/my-table.json).
  39. -w, --write-percentage
  40. Write capacity percentage.
  41. Default: 0.5

Backup

  1. java -jar dynamodb-to-s3-<version>.jar backup \
  2. --table my-table \
  3. --table my-other-table \
  4. --read-capacity 100 \
  5. --bucket my-bucket

Restore

  1. java -jar dynamodb-to-s3-<version>.jar restore \
  2. --table my-table \
  3. --source s3://my-bucket/2017/10/27/my-table.json

AWS credentials

AWS credentials are read using DefaultAWSCredentialsProviderChain:

  • Environment Variables: AWS_ACCESS_KEY, AWS_SECRET_KEY and AWS_REGION
  • Java System Properties: aws.accessKeyId and aws.secretKey
  • Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI
  • Credentials delivered through the Amazon EC2 container service if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment
    variable is set and security manager has permission to access the variable
  • Instance profile credentials delivered through the Amazon EC2 metadata service

Running with Docker

  1. docker run \
  2. -e AWS_ACCESS_KEY=... \
  3. -e AWS_SECRET_KEY=... \
  4. -e AWS_REGION=... \
  5. sensefly/dynamodb-to-s3 --help

Or if your instance has an IAM role, just run docker run sensefly/dynamodb-to-s3 --help

AWS Cloudwatch

To send logs to Cloudwatch, define environment variables:

  • CLOUDWATCH_LOG_GROUP
  • JAVA_OPTS: "-Dlogging.config=/log4j-cloudwatch.xml"

To send metrics to Cloudwatch, use --namespace argument.

For example, this command will backup table1 & table2 to dynamodb-backup every day at 2 AM.
It will send

  • logs to dynamodb-backup Cloudwatch log group
  • metrics to dynamodb-backup Cloudwatch metrics namespace.
  1. docker run \
  2. -e AWS_ACCESS_KEY=... \
  3. -e AWS_SECRET_KEY=... \
  4. -e AWS_REGION=... \
  5. -e CLOUDWATCH_LOG_GROUP=dynamodb-backup \
  6. -e JAVA_OPTS=-Dlogging.config=/log4j-cloudwatch.xml \
  7. sensefly/dynamodb-to-s3 backup \
  8. --table "table1,table2" \
  9. --bucket dynamodb-backup \
  10. --cron "0 2 * * *" \
  11. --read-percentage 0.8 \
  12. --pattern yyyy/MM/dd \
  13. --namespace dynamodb-backup

Build

Run ./gradlew build