Hadoop3环境部署，权威干货

加入收藏设为首页

大数据: Hadoop3环境部署，权威干货
时间：2017/5/1 23:07:22 作者：solgle 来源：solgle.com 查看：1491 评论：0
内容摘要：一、搭建操作系统，或者vm虚拟环境1. 配置主机名及网络配置修改主机名称 /etc/sysconfig/network配置网路 /etc/sysconfig/network-scripts/ifcfg-eth0添加或修改 BOOTPROTO=static

一、搭建操作系统，或者vm虚拟环境

1. 配置主机名及网络配置

修改主机名称 /etc/sysconfig/network

配置网路 /etc/sysconfig/network-scripts/ifcfg-eth0

添加或修改 BOOTPROTO=static

ONBOOT=yes

IPADDR=129.16.10.21

NETMASK=255.255.255.0

重启网卡服务生效：service network restart

同理主机节点配置ip 129.16.10.22

主机节点配置ip 129.16.10.23

[root@rac1 ~]# vi /etc/sysconfig/network

NETWORKING=yes

HOSTNAME=dataNode1

[root@rac1 network-scripts]# vi ifcfg-eth0

DEVICE="eth0"

BOOTPROTO="static"

HWADDR="00:0C:29:05:80:48"

NM_CONTROLLED="yes"

ONBOOT="yes"

TYPE="Ethernet"

UUID="0e8a4434-18c4-484f-bfe1-4f39b6b3768b"

IPADDR=129.16.10.21

NETMASK=255.255.255.0

[root@rac1 network-scripts]#

[root@rac3 network-scripts]# service network restart

Shutting down interface eth0: [ OK ]

Shutting down interface eth1: [ OK ]

Shutting down loopback interface: [ OK ]

Bringing up loopback interface: [ OK ]

Bringing up interface eth0: [ OK ]

Bringing up interface eth1:

Determining IP information for eth1... done. [ OK ]

2：配置每台主机hosts、

[root@rac2 network-scripts]# vi /etc/hosts

cat: /etc/hsots: No such file or directory

[root@rac2 network-scripts]# cat /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

129.16.10.21 dataNode1

129.16.10.22 dataNode2

129.16.10.23 nameNode

[root@rac2 network-scripts]#

二、创建用户

[root@rac3 ~]# mkdir /u01

[root@rac3 ~]# useradd hodp

[root@rac3 ~]# chown hodp -R /u01

[root@rac3 ~]#

三、配置ssh

[hodp@dataNode1 ~]$ mkdir .ssh

[hodp@dataNode1 ~]$ chmod 700 .ssh

[hodp@dataNode1 ~]$ cd .ssh

[hodp@dataNode1 .ssh]$ ssh-keygen -t rsa

[hodp@dataNode1 .ssh]$ ssh-keygen -t dsa

。。。

四、安装java虚拟机

[hodp@dataNode1 ~]$ tar -vxf server-jre-8u131-linux-x64.tar.gz

[hodp@dataNode1 ~]$ mv jdk1.8.0_131/ /u01/

[root@dataNode1 jdk1.8.0_131]# vi /etc/profile

##append code

export JAVA_HOME=/u01/jdk1.8.0_131

export CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export PATH=$JAVA_HOME/bin:$PATH

[hodp@dataNode2 jdk1.8.0_131]$ source /etc/profile

[hodp@dataNode2 jdk1.8.0_131]$ java -version

java version "1.8.0_131"

Java(TM) SE Runtime Environment (build 1.8.0_131-b11)

Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

五、下载hadoop

下载地址：https://archive.apache.org/dist/hadoop/

http://mirrors.hust.edu.cn/apache/hadoop/common/

本文出自：http://www.solgle.com/news/366.html

1. 配置core-site.xml，用于mapReduce和hdfs常用io配置

[hodp@dataNode1 hadoop]$ pwd

/u01/hadoop-3.0.0-alpha2/etc/hadoop

[hodp@dataNode1 hadoop]$ vi core-site.xml

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>

<name>hadoop.tmp.dir</name>

<value>/home/hodp/tmp</value>

<description>Abase for other temporary directories.</description>

</property>

<property>

<name>fs.defaultFS</name>

<value>hdfs://nameNode:9000</value>

</property>

<property>

<name>io.file.buffer.size</name>

<value>4096</value>

</property>

</configuration>

2. 配置hdfs-site.xml，守护进程，包括dataNode,nameNode

[hodp@dataNode1 hadoop]$ vi hdfs-site.xml

[hodp@dataNode1 hadoop]$ cat hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>

<name>dfs.nameservices</name>

<value>hadoop-cluster1</value>

</property>

<property>

<name>dfs.namenode.secondary.http-address</name>

<value>nameNode:50090</value>

</property>

<property>

<name>dfs.namenode.name.dir</name>

<value>file:///home/hodp/dfs/name</value>

</property>

<property>

<name>dfs.datanode.data.dir</name>

<value>file:///home/hodp/dfs/data</value>

</property>

<property>

<name>dfs.replication</name>

<value>2</value>

</property>

<property>

<name>dfs.webhdfs.enabled</name>

<value>true</value>

</property>

</configuration>

3. 配置mapred-site.xml 配置jobtracker,tasktracker守护进程

[hodp@dataNode1 hadoop]$ cp mapred-site.xml.template mapred-site.xml

[hodp@dataNode1 hadoop]$ vi mapred-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

<property>

<name>mapreduce.jobtracker.http.address</name>

<value>nameNode:50030</value>

</property>

<property>

<name>mapreduce.jobhistory.address</name>

<value>nameNode:10020</value>

</property>

<property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>nameNode:19888</value>

</property>

</configuration>

4. yarn-site.xml mapReduce站点配置

[hodp@dataNode1 hadoop]$ vi yarn-site.xml

<?xml version="1.0"?>

<configuration>



<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.resourcemanager.address</name>

<value>nameNode:8032</value>

</property>

<property>

<name>yarn.resourcemanager.scheduler.address</name>

<value>nameNode:8030</value>

</property>

<property>

<name>yarn.resourcemanager.resource-tracker.address</name>

<value>nameNode:8031</value>

</property>

<property>

<name>yarn.resourcemanager.admin.address</name>

<value>nameNode:8033</value>

</property>

<property>

<name>yarn.resourcemanager.webapp.address</name>

<value>nameNode:8088</value>

</property>

</configuration>

5. 配置workers

dataNode1

dataNode2

6. 增加配置JAVA_HOME

分别在文件hadoop-env.sh和yarn-env.sh中添加JAVA_HOME配置

JAVA_HOME=/u01/jdk1.8.0_131

7. 拷贝hadoop安装目录一起拷贝到其他服务器上

[hodp@dataNode1 u01]$ scp -r hadoop-3.0.0-alpha2/ dataNode2:/u01/

[hodp@dataNode1 u01]$ scp -r hadoop-3.0.0-alpha2/ nameNode:/u01/

六、格式化文件系统

[hodp@nameNode bin]$ ./hdfs namenode -format

2017-04-29 13:22:07,291 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG: user = hodp

STARTUP_MSG: host = nameNode/129.16.10.23

STARTUP_MSG: args = [-format]

STARTUP_MSG: version = 3.0.0-alpha2

STARTUP_MSG: classpath = /u01/hadoop-3.0.0-alpha2/etc/hadoop:/u01/hadoop-3.0.0-

alpha2/share/hadoop/common/lib/commons-lang3-3.3.2.jar:/u01/hadoop-3.0.0-

alpha2/share/hadoop/common/lib/curator-client-2.7.1.jar:/u01/hadoop-3.0.0-

alpha2/share/hadoop/common/lib/json-smart-1.1.1.jar:/u01/hadoop-3.0.0-

alpha2/share/hadoop/common/lib/commons-cli-1.2.jar:/u01/hadoop-3.0.0-

alpha2/share/hadoop/common/lib/zookeeper-3.4.6.jar:/u01/hadoop-3.0.0-

。。。

七、启动及环境测试

1. 启动报错

[hodp@nameNode dfs]$ /u01/hadoop-3.0.0-alpha2/sbin/start-dfs.sh

Starting namenodes on [nameNode]

Starting datanodes

localhost: datanode is running as process 2396. Stop it first.

2017-04-29 12:56:39,248 WARN util.NativeCodeLoader: Unable to load nativere applicable

---错误诊断过程

[hodp@nameNode native]$ ldd libhadoop.so.1.0.0

./libhadoop.so.1.0.0: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by ./libhadoop.so.1.0.0)

linux-vdso.so.1 => (0x00007fff38dff000)

libdl.so.2 => /lib64/libdl.so.2 (0x00007f78a2ceb000)

libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f78a2acd000)

libc.so.6 => /lib64/libc.so.6 (0x00007f78a273b000)

/lib64/ld-linux-x86-64.so.2 (0x00007f78a3116000)

[hodp@nameNode native]$

[hodp@nameNode native]$ ldd --version

ldd (GNU libc) 2.12

Copyright (C) 2010 Free Software Foundation, Inc.

This is free software; see the source for copying conditions. There is NO

warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Written by Roland McGrath and Ulrich Drepper.

[hodp@nameNode native]$

对比发现GLIBC_2.14 跟现在系统的版本不一致

有两个办法，重新编译glibc.2.14版本，安装后专门给hadoop使用，这个有点危险。第二个办法直接在log4j日志中去除告警信息。在/u01/hadoop-3.0.0-alpha2/etc/hadoop/log4j.properties文件中添加

log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR

八、测试守护

[hodp@nameNode sbin]$ ./start-yarn.sh

Starting resourcemanager

Starting nodemanagers

[hodp@nameNode sbin]$

查看服务

[hodp@nameNode sbin]$ jps

4562 Jps

4200 NodeManager

4107 ResourceManager

3756 DataNode

3663 NameNode

[hodp@nameNode sbin]$

九、 dataNode启动测试

[hodp@dataNode2 sbin]$ ./start-dfs.sh

Starting namenodes on [nameNode]

nameNode: namenode is running as process 3663. Stop it first.

Starting datanodes

localhost: Warning: Permanently added 'localhost' (RSA) to the list of known hosts.

localhost: WARNING: /u01/hadoop-3.0.0-alpha2/logs does not exist. Creating.

[hodp@dataNode2 sbin]$

[hodp@dataNode2 sbin]$ ./start-yarn.sh

Starting resourcemanager

Starting nodemanagers

[hodp@dataNode2 sbin]$

十、关闭防火墙

[hodp@dataNode1 sbin]$ chkconfig iptables off

You do not have enough privileges to perform this operation.

[hodp@dataNode1 sbin]$ su - root

Password:

[root@dataNode1 ~]# chkconfig iptables off

[root@dataNode1 ~]# service iptables stop

iptables: Flushing firewall rules: [ OK ]

iptables: Setting chains to policy ACCEPT: filter [ OK ]

iptables: Unloading modules: [ OK ]

十一、打开网页查看

http://129.16.10.23:9870/

标签：Hadoop环境部署权威干货


solgle.com 版权所有，欢迎分享！！！

上一篇：WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
下一篇：Incompatible clusterIDs in /home/hodp/dfs/data: namenode clusterID = CID-3a02cd0c-7477-48ec-bdf2-62b0049d7201

相关文章

相关评论

本类更新

本类推荐

本类排行

: 国家数据 | 中证指数 | Oracle官方文档 | 沪深行情 | 流量统计 | 深交所 | 上交所 | 中国证券登记 | 同业拆放利率 | 巨潮资讯 | 指数估值 | 看财报 |

联系我们 - 关于我们 - 网站同盟