HDFS>> Big-Data-Tools-And-Exampls>> 返回
项目作者： lizichen

项目描述 ：
Hadoop and Spark and related tools: Hive, Pig, Spark, HBase, Sqoop, Oozie, etc
高级语言： Java
项目地址: git://github.com/lizichen/Big-Data-Tools-And-Exampls.git
创建时间： 2017-02-08T20:00:34Z
项目社区：https://github.com/lizichen/Big-Data-Tools-And-Exampls
开源协议：
下载


Big Data Tools and Examples

MapReduce, HDFS, Hadoop, Hive, Pig, Spark, HBase, Sqoop, Oozie

Using git push on CentOS

  • git remote set-url origin lizichen@github.com/lizichen/Realtime-Big-Data-Analytics.git"">https://lizichen@github.com/lizichen/Realtime-Big-Data-Analytics.git
  • git push -u origin master
  • type in password to push

Compile and Run Java MapReduce Programs on Virtual Box

  1. Startup your VM

  2. Write your driver source code using a text editor like vi (or emacs):

    1.      vi MaxTemperature.java

  3. Write your mapper and reducer source code:

    1.     vi MaxTemperatureMapper.java
    2.     vi MaxTemperatureReducer.java

  4. Compile your Java code:

    1. java -version
    2. yarn classpath
    3. javac -classpath `yarn classpath` -d . MaxTemperatureMapper.java
    4. javac -classpath `yarn classpath` -d . MaxTemperatureReducer.java
    5. javac -classpath `yarn classpath`:. -d . MaxTemperature.java

  5. Create your jar file

    1. jar -cvf maxTemp.jar *.class

  6. Create your input data file on the local file system

    1. vi temperatureInputs.txt

  7. Put your input data file into HDFS

    1. hdfs dfs -ls /
    2. hdfs dfs -ls /user
    3. hdfs dfs -ls /user/cloudera
    4. hdfs dfs -mkdir /user/cloudera/class1
    5. hdfs dfs -put temperatureInputs.txt /user/cloudera/class1
    6. hdfs dfs -cat /user/cloudera/class1/temperatureInputs.txt

  8. Run your MapReduce program

    1. hadoop jar maxTemp.jar MaxTemperature /user/cloudera/class1/temperatureInputs.txt /user/cloudera/class1/output

  9. Verify that the program ran and the results are correct

    1. hdfs dfs -ls /user/cloudera/class1/output
    2. hdfs dfs -cat /user/cloudera/class1/output/part-r-00000
Class2_HW_Lizi_Chen_1647933669549.pdf
Spark_Summer2017_Class2_Homework_1647933669586.pdf
Class3_HW3_Lizi_Chen_1647933669691.pdf
Spark_Cluster_Computing_with_Working_Sets_1647933669738.pdf
Spark_Summer2017_Class3_SparkIntro_Students_1647933669787.pdf
Spark_Summer2017_Class3_Spark_Homework_1647933669899.pdf
BDAD_SU17_IPP_LiziChen_1647933670200.pdf
BDAD_SU17_ProjectInfo_1647933670293.pdf
Spark_Summer2017_Class4_HW_1647933670364.pdf
Spark_Summer2017_Class4_RDDs_ScalaTuples_1647933670381.pdf
TargetingVillagesRuralDevUsingSatImageAnalysis_1647933670418.pdf
NotOnMidterm_Spark_Summer2017_Class5_ScalaFlowControl_1647933670849.pdf
Spark_Summer2017_Class5_ScalaCollections_1647933670929.pdf
Spark_Summer2017_Class5_XML_HW_1647933671049.pdf
Spark_Summer2017_Class6_HW_1647933671233.pdf
Spark_Summer2017_Class6_PairRDDs_2017_1647933671257.pdf
Spark_Summer2017_Class7_BuldingApps_1647933671392.pdf
Spark_Summer2017_Class7_HW_1647933671424.pdf
Spark_Summer2017_Class7_PairRDDJoins_1647933671442.pdf
homework_7_LiziChen_1647933671491.pdf
Spark_Summer2017_Class8_SparkAlgorithms_1647933671683.pdf
Spark_Summer2017_Class8_SparkSQL_1647933671743.pdf
1-s2.0-S2090447914000550-main_1647933672029.pdf
Evaluating sentiment in financial news articles_1647933672223.pdf
Nardo_et_al-2016-Journal_of_Economic_Surveys_1647933672266.pdf
Paper Summary_Chunyi_1647933672289.pdf
STOCK TREND PREDICTION USING NEWS SENTIMENT ANALYSIS - Kalyani Joshi_1647933672330.pdf
Sentiment Analysis in Financial News_1647933672391.pdf
Sentiment Analysis of Financial News Articles - Robert P. Schumaker_1647933672539.pdf
Sprenger_et_al-2014-European_Financial_Management_1647933672643.pdf
Techniques and Applications for Sentiment Analysis_1647933673264.pdf
BDAD_SU17_TPP_1647933674580.pdf
DataSchema_1647933674623.pdf
BDAD_SU17_TPP_1650018480566.pdf
DataSchema_1650018480726.pdf
Techniques and Applications for Sentiment Analysis_1650018478228.pdf
Sprenger_et_al-2014-European_Financial_Management_1650018477128.pdf
Nardo_et_al-2016-Journal_of_Economic_Surveys_1650018475987.pdf
Paper Summary_Chunyi_1650018476200.pdf
STOCK TREND PREDICTION USING NEWS SENTIMENT ANALYSIS - Kalyani Joshi_1650018476412.pdf
Sentiment Analysis in Financial News_1650018476586.pdf
Sentiment Analysis of Financial News Articles - Robert P. Schumaker_1650018476770.pdf
1-s2.0-S2090447914000550-main_1650018475500.pdf
Evaluating sentiment in financial news articles_1650018475762.pdf
Spark_Summer2017_Class7_HW_1650018473989.pdf
Spark_Summer2017_Class7_PairRDDJoins_1650018474089.pdf
homework_7_LiziChen_1650018474517.pdf
Spark_Summer2017_Class8_SparkAlgorithms_1650018474669.pdf
Spark_Summer2017_Class8_SparkSQL_1650018474868.pdf
Spark_Summer2017_Class6_HW_1650018473453.pdf
Spark_Summer2017_Class6_PairRDDs_2017_1650018473572.pdf
Spark_Summer2017_Class7_BuldingApps_1650018473835.pdf
TargetingVillagesRuralDevUsingSatImageAnalysis_1650018472089.pdf
NotOnMidterm_Spark_Summer2017_Class5_ScalaFlowControl_1650018472520.pdf
Spark_Summer2017_Class5_ScalaCollections_1650018472810.pdf
Spark_Summer2017_Class5_XML_HW_1650018472957.pdf
BDAD_SU17_IPP_LiziChen_1650018471393.pdf
BDAD_SU17_ProjectInfo_1650018471599.pdf
Spark_Summer2017_Class4_HW_1650018471831.pdf
Spark_Summer2017_Class4_RDDs_ScalaTuples_1650018471895.pdf
Class2_HW_Lizi_Chen_1650018470145.pdf
Spark_Summer2017_Class2_Homework_1650018470252.pdf
Class3_HW3_Lizi_Chen_1650018470431.pdf
Spark_Cluster_Computing_with_Working_Sets_1650018470528.pdf
Spark_Summer2017_Class3_SparkIntro_Students_1650018470688.pdf
Spark_Summer2017_Class3_Spark_Homework_1650018470776.pdf