[error] SparkUI 포트 사용

 ERROR ui.SparkUI: Failed to bind SparkUI
java.net.BindException: Address already in use: bind: Service 'SparkUI' failed after 16 retries (starting from 4040)! Consider explicitly setting the appropriate port for the service 'SparkUI' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:433)
	at sun.nio.ch.Net.bind(Net.java:425)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at org.spark_project.jetty.server.ServerConnector.open(ServerConnector.java:317)
	at org.spark_project.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
	at org.spark_project.jetty.server.ServerConnector.doStart(ServerConnector.java:235)
	at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
	at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$newConnector$1(JettyUtils.scala:339)
	at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$httpConnect$1(JettyUtils.scala:366)
	at org.apache.spark.ui.JettyUtils$$anonfun$7.apply(JettyUtils.scala:369)
	at org.apache.spark.ui.JettyUtils$$anonfun$7.apply(JettyUtils.scala:369)
	at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2234)
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
	at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2226)
	at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:369)
	at org.apache.spark.ui.WebUI.bind(WebUI.scala:130)
	at org.apache.spark.SparkContext$$anonfun$11.apply(SparkContext.scala:460)
	at org.apache.spark.SparkContext$$anonfun$11.apply(SparkContext.scala:460)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.SparkContext.(SparkContext.scala:460)
	at org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:839)
	at org.apache.spark.streaming.StreamingContext.(StreamingContext.scala:85)
	at flume_push_streaming$.main(flume_push_streaming.scala:11)
	at flume_push_streaming.main(flume_push_streaming.scala)

오류 원인

모든 Spark 작업은 SparkUI 포트 1개를 사용하며 기본값은 4040이며 사용 중인 경우 순차적으로 추가 포트 재시도를 수행합니다.그러나 기본 재시도 횟수는 16회입니다.16번의 재시도가 실패하면 이 작업의 실행을 포기합니다.

해결 방법:

SparkConf 객체 작성 시 재연결 횟수 설정

key: spark.port.maxRetries

value: 100