项目作者: rajkstats
项目描述 :
S3 Analyzer - Web app to analyze your s3 bucket
高级语言: R
项目地址: git://github.com/rajkstats/s3_Analyzer.git
s3 Analyzer - Web app to analyze your s3 bucket
Quick Overview
This is a mini project I started to easily analyze s3 bucket with R’s flexdashboard package.
Why s3 Analyzer ?
- See how files under your s3 bucket/path are distributed (in terms of size)
- AWS CLI summarise provides file sizes which are not uniform in size
- s3 Analyzer converts all different file sizes (EiB,PiB,TiB,GiB,KiB,Bytes) to a uniform unit i.e MiB
- Renders total files under s3 bucket/path
- Creates a frequency table for all files and shows % files in each s3 bucket
- Renders a interactive bar plot to visually show how your files are distributed
How to use App ?
- s3 Analyzer loads a default processed public s3 dataset GEOS-Chem on cloud (Total Files: 11,948,
Total Size: 4.2 TiB)
- You can upload your processed file (max file size = 50 MiB) using AWS CLI on your s3 bucket/path
- Choose bin size - (50 MiB or 100 MiB)
- Click on Apply button for action
Processed files from s3 bucket
How to get processed file from AWS CLI ?
Following command using the ls to list all files and
—human-readable displays file size in Bytes/MiB/KiB/GiB/TiB/PiB/EiB
—summarize displays the total number of objects and total size at the end of the result
and copies the output to s3_analysis.csv file
- aws s3 ls --recursive --human-readable --summarize s3://nasanex/NEX-GDDP/BCSD/rcp45/ > s3_analysis_nasanex.csv
-
- Upload the processed file s3_analysis_nasanex.csv to s3 Analyzer
R v3.5.1 and RStudio v1.2.1335 were used to build this tool.
The packages used were:
Interactive Plot
You can:
- click the camera button to download bar plot as png
- zoom with the ‘+’ and ‘-‘ buttons (top-right) or with your mouse’s scroll wheel
- click the button showing a broken square (top-left under the zoom options) to select points on the plot using a window that’s
draggable (click and hold the grid icon in the upper left) and resizeable (click and drag the white boxes in each corner)
Interactive Table
You can:
- sort the columns (ascending and descending) by clicking on the column header
- scroll the table vertcially to see all elements
- click ‘CSV’ or ‘Excel’ to download the data to a .csv file or a .xlsx
Credits
For any feedback, comments or questions, please write to me at raj.k.stats@gmail.com.