아파 치 빔 컴 파일&실행 데모

ApacheBeam 소스 코드 변이

#    build.gradle
// mavenCentral()
maven { url 'http://maven.aliyun.com/nexus/content/groups/public/'}

#   spark-runner
 ./gradlew beam-runners-spark:build

gradle 명령 학습

./gradlew tasks --all|grep spark

#   example  
cd beam/examples/java &&  ../../gradlew build

일반적인 gradle 명령 학습:

./gradlew -v    ，    ，  gradle      。
./gradlew clean   HelloWord/app    build   
./gradlew build          
./gradlew assembleDebug     Debug 
./gradlew assembleRelease     Release  
./gradlew installRelease Release       
./gradlew uninstallRelease   Release   

Spark 환경 기반 Beam 프로그램 실행

Spark 환경 에 beam 관련 의존 Jar 패키지 추가:

bash-3.2$ ls jars/|grep beam
beam-model-pipeline-2.9.0-SNAPSHOT.jar
beam-runners-core-construction-java-2.9.0-SNAPSHOT.jar
beam-runners-core-java-2.9.0-SNAPSHOT.jar
beam-runners-spark-2.9.0-SNAPSHOT.jar
beam-sdks-java-core-2.9.0-SNAPSHOT.jar
beam-sdks-java-io-hadoop-file-system-2.9.0-SNAPSHOT-unshaded.jar
beam-vendor-grpc-v1_13_1-2.9.0-SNAPSHOT.jar

#       ，beam  guava-20.0.jar   ， spark 2.3.0      16.0，      guava   20.0。

Spark 환경 기반 운행

#   local    (      )
bin/spark-submit --class org.apache.beam.examples.WordCount --master local[1] ./beam-examples-java-2.9.0-SNAPSHOT.jar  --runner=SparkRunner --inputFile=hdfs:///tmp/README.txt --output=hdfs:///tmp/beam.txt
#   yarn    
bin/spark-submit --class org.apache.beam.examples.WordCount --master yarn ./beam-examples-java-2.9.0-SNAPSHOT.jar  --runner=SparkRunner --inputFile=hdfs:///tmp/README.txt --output=hdfs:///tmp/beam.txt

TODO:Flink 기반 Beam 프로그램 실행
참고:

https://blog.csdn.net/u012063409/article/details/54572900

https://blog.csdn.net/dream_an/article/details/56277784

https://blog.csdn.net/chenhaifeng2016/article/details/72519343

https://beam.apache.org/get-started/wordcount-example/