Hadoop

참조 링크http://hadoop.apache.org/docs/r1.2.1/index.html1hadop 단일 노드 배치 테스트 환경

[root@lv1 ~]# useradd -u 800 hadoop
[root@lv1 ~]# passwd hadoop
[hadoop@lv1 ~]$ ssh-keygen 
[hadoop@lv1 ~]$ ssh-copy-id localhost
[hadoop@lv1 ~]$ ssh localhost
[hadoop@lv1 ~]$ logout
[hadoop@lv1 ~]$ ssh 192.168.2.145
[hadoop@lv1 ~]$ logout
[hadoop@lv1 ~]$ ssh lv1.example.com
[hadoop@lv1 ~]$ logout
lftp i:/> get pub/docs/hadoop/hadoop-1.2.1.tar.gz 
lftp i:/> get pub/docs/java/jdk-6u32-linux-x64.bin
[hadoop@lv1 ~]$ tar zxf hadoop-1.2.1.tar.gz 
[hadoop@lv1 ~]$ chown -R hadoop.hadoop hadoop-1.2.1/
[hadoop@lv1 ~]$ ln -s hadoop-1.2.1/ hadoop
[hadoop@lv1 ~]$ sh jdk-6u32-linux-x64.bin 
[hadoop@lv1 ~]$ mv jdk1.6.0_32/ hadoop
[hadoop@lv1 ~]$ cd hadoop
[hadoop@lv1 hadoop]$ ln -s jdk1.6.0_32/ jdk
[hadoop@lv1 hadoop]$ vim conf/hadoop-env.sh   
export JAVA_HOME=/home/hadoop/hadoop/jdk

테스트

[hadoop@lv1 hadoop]$ cp -r conf/ input
[hadoop@lv1 hadoop]$ bin/hadoop jar hadoop-examples-1.2.1.jar grep input output 'dfs[a-z.]+'     
[hadoop@lv1 hadoop]$ cat output/part-00000 
1    dfs.server.namenode.
1    dfsadmin
[hadoop@lv1 hadoop]$ bin/hadoop jar hadoop-examples-1.2.1.jar wordcount input test       

2.의사 분포 식 hadop 이 단일 노드 에서 임 무 를 수행 하 는 map-reduce

[hadoop@lv1 ~]$ mkdir bin
[hadoop@lv1 ~]$ ln -s /home/hadoop/hadoop/jdk/bin/jps ~/bin/     jps  

설정 파일 conf/core-site.xml:

<configuration>
     <property>
         <name>fs.default.name</name>
         <value>hdfs://localhost:9000</value>
     </property>
</configuration>

conf/hdfs-site.xml:

<configuration>
     <property>
         <name>dfs.replication</name>
         <value>1</value>
     </property>
</configuration>

conf/mapred-site.xml:

<configuration>
     <property>
         <name>mapred.job.tracker</name>
         <value>localhost:9001</value>
     </property>
</configuration>

[hadoop@lv1 hadoop]$ bin/hadoop namenode -format    namenode
[hadoop@lv1 hadoop]$ bin/start-all.sh   
[hadoop@lv1 hadoop]$ jps 
2641 NameNode      master
2912 JobTracker        ， master    
3032 TaskTracker     
2744 DataNode       
2847 SecondaryNameNode

브 라 우 저 액세스 모니터링:    NameNode - http://localhost:50070/    JobTracker - http://localhost:50030/

[hadoop@lv1 hadoop]$ bin/hadoop fs -mkdir input
[hadoop@lv1 hadoop]$ bin/hadoop fs -put conf/* input
[hadoop@lv1 hadoop]$ bin/hadoop jar hadoop-examples-1.2.1.jar wordcount input output
[hadoop@lv1 hadoop]$ bin/hadoop fs -cat output/*

3.완전 분포 식 실험 환경:192.168.2.145   lv1.example.com 시작 Secondary NameNode,JobTracker,NameNode 192.168.2.146   node1.example.com 시작 TaskTracker,DataNode 192.168.2.189   node2.example.com192.168.2.142   node3.example.com 삭제 노드 master 를 slave 사이 에 암호 인증 없 음 추가

[root@node1 ~]# useradd -u 800 hadoop
[root@node2 ~]# useradd -u 800 hadoop
[root@node1 ~]# passwd hadoop
[hadoop@lv1 ~]$ scp -r .ssh/ node1.example.com:
[hadoop@lv1 ~]$ scp -r .ssh/ node2.example.com:
[hadoop@lv1 ~]$ ssh node1.example.com   
[hadoop@lv1 ~]$ ssh node2.example.com

프로필 수정

[hadoop@lv1 hadoop]$ vim conf/masters
lv1.example.com
[hadoop@lv1 hadoop]$ vim conf/slaves
node1.example.com
node2.example.com

[hadoop@lv1 hadoop]$ vim conf/core-site.xml 
  localhost lv1.example.com

[hadoop@lv1 hadoop]$ vim conf/mapred-site.xml 
  localhost lv1.example.com

[hadoop@lv1 hadoop]$ vim conf/hdfs-site.xml 
  value   2，   2 

[hadoop@lv1 ~]$ scp -r hadoop-1.2.1/ node1.example.com:
[hadoop@lv1 ~]$ scp -r hadoop-1.2.1/ node2.example.com:

두 노드 노드 노드 로 소프트 링크 만 들 기

[hadoop@node1 ~]$ ln -s hadoop-1.2.1/ hadoop
[hadoop@node1 ~]$ mkdir bin
[hadoop@node1 ~]$ cd bin/
[hadoop@node1 bin]$ ln -s ~/hadoop/jdk/bin/jps .

namenode 다시 포맷 하기

[hadoop@lv1 hadoop]$ bin/hadoop namenode -format
[hadoop@lv1 hadoop]$ bin/start-all.sh   hadoop
starting namenode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-namenode-lv1.example.com.out
node1.example.com: starting datanode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-datanode-node1.example.com.out
node2.example.com: starting datanode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-datanode-node2.example.com.out
lv1.example.com: starting secondarynamenode, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-secondarynamenode-lv1.example.com.out
starting jobtracker, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-jobtracker-lv1.example.com.out
node1.example.com: starting tasktracker, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-tasktracker-node1.example.com.out
node2.example.com: starting tasktracker, logging to /home/hadoop/hadoop-1.2.1/libexec/../logs/hadoop-hadoop-tasktracker-node2.example.com.out

테스트

[hadoop@lv1 hadoop]$ bin/hadoop fs -put conf input
[hadoop@lv1 hadoop]$ bin/hadoop fs -ls
drwxr-xr-x   - hadoop supergroup          0 2014-06-15 02:19 /user/hadoop/input
[hadoop@lv1 hadoop]$ bin/hadoop jar hadoop-examples-1.2.1.jar grep input output 'dfs[a-z.]+'
[hadoop@lv1 hadoop]$ bin/hadoop fs -cat output/*     

브 라 우 저 접근http://192.168.2.145:50030화해시키다http://192.168.2.145:50030동적 추가 삭제 노드

[hadoop@lv1 hadoop]$ dd if=/dev/zero of=bigfile bs=1M count=100
[hadoop@lv1 hadoop]$ bin/hadoop fs -mkdir files
[hadoop@lv1 hadoop]$ bin/hadoop fs -put bigfile files    HDFS 
[hadoop@lv1 hadoop]$ bin/hadoop dfsadmin -report           100M  

[root@node3 ~]# useradd -u 800 hadoop
[root@node3 ~]# echo mmmmmm | passwd --stdin hadoop
[hadoop@lv1 ~]$ scp -r .ssh/ node3.example.com:
[hadoop@node3 bin]$ ln -s ~/hadoop-1.2.1/jdk/bin/jps .
[hadoop@node3 ~]$ ln -s hadoop-1.2.1/ hadoop
[hadoop@lv1 hadoop]$ vim conf/slaves 
[hadoop@node3 hadoop]$ bin/hadoop-daemon.sh start datanode
[hadoop@node3 hadoop]$ bin/hadoop-daemon.sh start tasktracker
[hadoop@lv1 hadoop]$ bin/hadoop dfsadmin -report      3   ，         
[hadoop@node3 hadoop]$ bin/start-balancer.sh