Spark：如何从spark shell运行spark文件

作者: 甲基蓝
发布时间: 2024-04-18 01:30:00 (8天前)
转自：

4 条回复

0#
回复此人
荧惑 | 2019-08-31 10-32

<div class =“post-text”itemprop =“text”> <P> 您可以使用sbt或maven来编译spark程序。只需将spark作为依赖添加到maven </p> <pre> <code> <repository> <id>Spark repository</id> <url>http://www.sparkjava.com/nexus/content/repositories/spark/</url> </repository> </code> </pre> <P> 然后依赖： </p> <pre> <code> <dependency> <groupId>spark</groupId> <artifactId>spark</artifactId> <version>1.2.0</version> </dependency> </code> </pre> <P> 在使用spark命令运行文件方面：您可以简单地执行此操作： </p> <pre> <code> echo" import org.apache.spark.sql.* ssc = new SQLContext(sc) ssc.sql("select * from mytable").collect " > spark.input </code> </pre> <P> 现在运行命令脚本： </p> <pre> <code> cat spark.input | spark-shell </code> </pre> </DIV>

编辑
1#
回复此人
Just do it | 2019-08-31 10-32

<div class =“post-text”itemprop =“text”> <P> 要从spark-shell加载外部文件，请执行此操作 </p> <pre> <code> :load PATH_TO_FILE </code> </pre> <P> 这将调用文件中的所有内容。 </p> <P> 我对你的SBT问题没有解决方案，但抱歉:-) </p> </DIV>

编辑
2#
回复此人
老人与海 | 2019-08-31 10-32

<div class =“post-text”itemprop =“text”> <P> 只是为了给出更多的答案 </p> <P> 的<strong> Spark-shell是一个scala repl </强> </p> <P> 你可以输入的<strong> ：救命 </强> 查看scala shell中可能的操作列表 </p> <pre> <code> scala> :help All commands can be abbreviated, e.g., :he instead of :help. :edit <id>|<line> edit history :help [command] print this summary or command-specific help :history [num] show the history (optional num is commands to show) :h? <string> search the history :imports [name name ...] show import history, identifying sources of names :implicits [-v] show the implicits in scope :javap <path|class> disassemble a file or class name :line <id>|<line> place line(s) at the end of history :load <path> interpret lines in a file :paste [-raw] [path] enter paste mode or paste a file :power enable power user mode :quit exit the interpreter :replay [options] reset the repl and replay all previous commands :require <path> add a jar to the classpath :reset [options] reset the repl to its initial state, forgetting all session entries :save <path> save replayable session to a file :sh <command line> run a shell command (result is implicitly => List[String]) :settings <options> update compiler options, if possible; see reset :silent disable/enable automatic printing of results :type [-v] <expr> display the type of an expression without evaluating it :kind [-v] <expr> display the kind of expression's type :warnings show the suppressed warnings from the most recent line which had any </code> </pre> <P> 的<strong> ：加载文件中的解释行 </强> </p> </DIV>

编辑

登录后才能参与评论