项目作者: rajkstats

项目描述 :
S3 Analyzer - Web app to analyze your s3 bucket
高级语言: R
项目地址: git://github.com/rajkstats/s3_Analyzer.git
创建时间: 2019-07-22T12:26:46Z
项目社区:https://github.com/rajkstats/s3_Analyzer

开源协议:

下载


s3 Analyzer - Web app to analyze your s3 bucket

Quick Overview

This is a mini project I started to easily analyze s3 bucket with R’s flexdashboard package.

Why s3 Analyzer ?

  • See how files under your s3 bucket/path are distributed (in terms of size)
  • AWS CLI summarise provides file sizes which are not uniform in size
  • s3 Analyzer converts all different file sizes (EiB,PiB,TiB,GiB,KiB,Bytes) to a uniform unit i.e MiB
  • Renders total files under s3 bucket/path
  • Creates a frequency table for all files and shows % files in each s3 bucket
  • Renders a interactive bar plot to visually show how your files are distributed

How to use App ?

  • s3 Analyzer loads a default processed public s3 dataset GEOS-Chem on cloud (Total Files: 11,948,
    1. Total Size: 4.2 TiB)
  • You can upload your processed file (max file size = 50 MiB) using AWS CLI on your s3 bucket/path
  • Choose bin size - (50 MiB or 100 MiB)
  • Click on Apply button for action

Processed files from s3 bucket

How to get processed file from AWS CLI ?

  • Following command using the ls to list all files and
    —human-readable displays file size in Bytes/MiB/KiB/GiB/TiB/PiB/EiB
    —summarize displays the total number of objects and total size at the end of the result
    and copies the output to s3_analysis.csv file

    1. aws s3 ls --recursive --human-readable --summarize s3://nasanex/NEX-GDDP/BCSD/rcp45/ > s3_analysis_nasanex.csv
  • Upload the processed file s3_analysis_nasanex.csv to s3 Analyzer

Tools

R v3.5.1 and RStudio v1.2.1335 were used to build this tool.

The packages used were:

Interactive Plot

You can:

  • click the camera button to download bar plot as png
  • zoom with the ‘+’ and ‘-‘ buttons (top-right) or with your mouse’s scroll wheel
  • click the button showing a broken square (top-left under the zoom options) to select points on the plot using a window that’s
    draggable (click and hold the grid icon in the upper left) and resizeable (click and drag the white boxes in each corner)

Interactive Table

You can:

  • sort the columns (ascending and descending) by clicking on the column header
  • scroll the table vertcially to see all elements
  • click ‘CSV’ or ‘Excel’ to download the data to a .csv file or a .xlsx

Credits

Contact

For any feedback, comments or questions, please write to me at raj.k.stats@gmail.com.