Apache Spark 준비

Spark 소스 코드 다운로드

Spark 빌드

maven 설치

- 例えば，http://qiita.com/chosan211/items/1472198165442e93047e などを参照してインストールする
- 手元のmac環境の場合は，macports を使って下記のように簡単にインストールできた
    - `$sudo port install maven3`
    - `$sudo port select --set maven maven3`
    - `$ mvn --version`

yk-tanigawa$ sudo port install maven3
--->  Computing dependencies for maven3
--->  Dependencies to be installed: maven_select
--->  Fetching archive for maven_select
--->  Attempting to fetch maven_select-0.3_1.darwin_14.noarch.tbz2 from https://packages.macports.org/maven_select
--->  Attempting to fetch maven_select-0.3_1.darwin_14.noarch.tbz2.rmd160 from https://packages.macports.org/maven_select
--->  Installing maven_select @0.3_1
--->  Activating maven_select @0.3_1
--->  Cleaning maven_select
--->  Fetching archive for maven3
--->  Attempting to fetch maven3-3.3.9_0.darwin_14.noarch.tbz2 from https://packages.macports.org/maven3
--->  Attempting to fetch maven3-3.3.9_0.darwin_14.noarch.tbz2.rmd160 from https://packages.macports.org/maven3
--->  Installing maven3 @3.3.9_0
--->  Activating maven3 @3.3.9_0

To make maven 3.3.9 the default, please run
    sudo port select --set maven maven3

--->  Cleaning maven3
--->  Updating database of binaries
--->  Scanning binaries for linking errors
--->  No broken files found.                         

yk-tanigawa$ sudo port select --set maven maven3
Selecting 'maven3' for 'maven' succeeded. 'maven3' is now active.

yk-tanigawa$ mvn --version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T01:41:47+09:00)
Maven home: /opt/local/share/java/maven3
Java version: 1.8.0_91, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.10.5", arch: "x86_64", family: "mac"

Spark 빌드

IntelliJ 설정

다운로드 & 설치

IntelliJ로 새 프로젝트 만들기

sbt 기반 프로젝트 만들기

Use auto-import 를 넣어두면 편리합니다.

project/src/main/scala에 scala class를 만듭니다.

Class의 이름은 htp : /// spark. 아파치. 오 rg / 두 cs / 아 st / 쿠이 ck-s rt. HTML의 예를 배우고 SimpleApp로 만듭니다.

Simple App 소스 코드 작성

build.sbt 에 spark의 Library를 사용한다는 것을 덧붙입니다.

jar 파일로 결합

yk-tanigawa$ sbt package
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=384m; support was removed in 8.0
[info] Loading project definition from /Users/yk-tanigawa/IdeaProjects/scala-spark-example01/project
[info] Set current project to scala-spark-example01 (in build file:/Users/yk-tanigawa/IdeaProjects/scala-spark-example01/)
[info] Updating {file:/Users/yk-tanigawa/IdeaProjects/scala-spark-example01/}scala-spark-example01...
[info] Resolving jline#jline;2.12.1 ...
[info] Done updating.
[info] Compiling 1 Scala source to /Users/yk-tanigawa/IdeaProjects/scala-spark-example01/target/scala-2.11/classes...
[info] Packaging /Users/yk-tanigawa/IdeaProjects/scala-spark-example01/target/scala-2.11/scala-spark-example01_2.11-1.0.jar ...
[info] Done packaging.
[success] Total time: 27 s, completed May 6, 2016 2:20:21 PM