采用FACEBOOK的AvatarNode来实现Hadoop的双机热备功能具体如下
一.环境
机器
Primary:192.168.17.130 Standby/NFS:192.168.17.131 AvatarDataNode:192.168.17.132
软件
hadoop-release-0.20.2编译生成的hadoop-0.20.2-highavailability.jar,hadoop-0.20.3-dev-core.jar两个更新包
二.具体部署过程
1.在192.168.17.130,192.168.17.131,192.168.17.132上创建如下目录
/home/hadoop/share
/home/hadoop/name
/home/hadoop/data
/home/hadoop/tmp
2.在192.168.17.131 设置NFS 共享目录为 /home/hadoop/share/
3.192.168.17.130 挂载NFS mount 192.168.17.131://home/hadoop/share/ /home/hadoop/share/
4.在NFS上创建 shared0,shared1两个目录
5.关闭所有机器上的防火墙 /etc/init.d/iptable stop
6.192.168.17.130与192.168.17.131相互之间能无密码SSH访问，并能无密码SSH访问数据节点
7.配置 hdfs-site.xml
<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Put site-specific property overrides in this file. --><configuration> <property> <name>dfs.name.dir</name> <value>/home/hadoop/name</value> </property> <property> <name>dfs.data.dir</name> <value>/home/hadoop/data</value> </property> <property> <name>dfs.http.address0</name> <value>192.168.17.130:50070</value> </property> <property> <name>dfs.http.address1</name> <value>192.168.17.131:50070</value> </property> <property> <name>dfs.name.dir.shared0</name> <value>/home/hadoop/share/shared0</value> </property> <property> <name>dfs.name.dir.shared1</name> <value>/home/hadoop/share/shared1</value> </property> <property> <name>dfs.name.edits.dir.shared0</name> <value>/home/hadoop/share/shared0</value> </property> <property> <name>dfs.name.edits.dir.shared1</name> <value>/home/hadoop/share/shared1</value> </property> <property> <name>dfs.replication</name> <value>1</value> </property></configuration>
8.配置core-site.xml
<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Put site-specific property overrides in this file. --><configuration> <property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/tmp</value> </property> <property> <name>fs.default.name</name> <value>hdfs://192.168.17.130:9000</value> </property> <property> <name>fs.default.name0</name> <value>hdfs://192.168.17.130:9000</value> </property> <property> <name>fs.default.name1</name> <value>hdfs://192.168.17.131:9000</value> </property>
<property> <name>dfs.hosts.exclude</name> <value>excludes</value> </property>
</configuration>
9.节点配置
Primary:192.168.17.130 的 conf/master 为 192.168.17.130Standby/NFS:192.168.17.131 的 conf/master 为 192.168.17.131
conf/slaves 为 192.168.17.132
10.分发文件至各服务器
conf/core-site.xml
conf/hdfs-site.xml
{HADOOP_HOME}/hadoop-0.20.3-dev-core.jar
{HADOOP_HOME}/lib/hadoop-0.20.2-highavailability.jar
11.格式化元数据
<1. 在192.168.17.130上 {HADOOP_HOME}/bin/hadoop namenode -format
<2. 查看是否格式化成功 /home/hadoop/name
同步元数据
<3. 将scp /home/hadoop/name/* 192.168.17.130://home/hadoop/name/ -R
<4. cp /home/hadoop/name/* /home/hadoop/share/shared0/ -R
<5. cp /home/hadoop/name/* /home/hadoop/share/shared1/ -R
12.启动
bin/hadoop org.apache.hadoop.hdfs.server.namenode.AvatarNode -zero 主节点
bin/hadoop org.apache.hadoop.hdfs.server.namenode.AvatarNode -one -standby 从节点
bin/hadoop org.apache.hadoop.hdfs.server.datanode.AvatarDataNode 数据节点