Hadoop 3.3.6 三节点集群实战从零构建到Web UI验证1. 环境准备与规划在开始搭建Hadoop集群之前我们需要做好充分的准备工作。一个典型的Hadoop集群包含三种节点角色NameNode管理元数据、DataNode存储实际数据块和ResourceManager管理计算资源。对于生产环境建议将NameNode和ResourceManager部署在不同机器上以实现高可用但在本次三节点实验中我们将采用以下架构Master节点同时担任NameNode和ResourceManager角色Slave1节点DataNode和NodeManagerSlave2节点DataNode和NodeManager硬件要求每台机器至少4GB内存8GB以上更佳每台机器至少50GB磁盘空间千兆网络连接软件要求CentOS 7/8或Ubuntu 18.04/20.04Java 8或11推荐OpenJDKHadoop 3.3.6安装包注意所有节点的系统时间必须同步时区设置一致否则可能导致集群异常。可使用timedatectl set-timezone Asia/Shanghai统一时区。2. 基础环境配置2.1 系统设置在所有节点上执行以下操作# 关闭防火墙 systemctl stop firewalld systemctl disable firewalld # 禁用SELinux setenforce 0 sed -i s/SELINUXenforcing/SELINUXdisabled/g /etc/selinux/config # 创建专用用户 useradd -m hadoop echo hadoop:yourpassword | chpasswd usermod -aG wheel hadoop # 配置sudo权限 echo hadoop ALL(ALL) NOPASSWD: ALL /etc/sudoers2.2 主机名与hosts文件在Master节点hostnamectl set-hostname master在Slave1节点hostnamectl set-hostname slave1在Slave2节点hostnamectl set-hostname slave2在所有节点的/etc/hosts文件中添加以下内容IP根据实际情况修改192.168.1.101 master 192.168.1.102 slave1 192.168.1.103 slave2验证网络连通性ping -c 3 master ping -c 3 slave1 ping -c 3 slave22.3 JDK安装在所有节点安装Java环境# 下载OpenJDK以JDK8为例 wget https://download.java.net/openjdk/jdk8u41/ri/openjdk-8u41-b04-linux-x64-14_jan_2020.tar.gz # 解压并安装 tar -zxvf openjdk-8u41-b04-linux-x64-14_jan_2020.tar.gz -C /usr/local/ mv /usr/local/java-se-8u41-ri /usr/local/java # 配置环境变量 echo export JAVA_HOME/usr/local/java /etc/profile echo export PATH$PATH:$JAVA_HOME/bin /etc/profile source /etc/profile # 验证安装 java -version3. SSH免密登录配置集群节点间的免密登录是Hadoop正常工作的基础以下是详细配置步骤3.1 生成密钥对在所有节点以hadoop用户执行su - hadoop ssh-keygen -t rsa -P -f ~/.ssh/id_rsa3.2 配置授权在Master节点cat ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys # 将公钥分发到Slave节点 scp ~/.ssh/id_rsa.pub hadoopslave1:~/.ssh/master.pub scp ~/.ssh/id_rsa.pub hadoopslave2:~/.ssh/master.pub在Slave1节点cat ~/.ssh/master.pub ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys在Slave2节点cat ~/.ssh/master.pub ~/.ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys3.3 验证免密登录从Master节点测试ssh slave1 hostname ssh slave2 hostname预期应分别返回slave1和slave2而不需要输入密码。4. Hadoop安装与配置4.1 安装Hadoop仅在Master节点操作wget https://archive.apache.org/dist/hadoop/common/hadoop-3.3.6/hadoop-3.3.6.tar.gz tar -zxvf hadoop-3.3.6.tar.gz -C /usr/local/ mv /usr/local/hadoop-3.3.6 /usr/local/hadoop chown -R hadoop:hadoop /usr/local/hadoop4.2 环境变量配置在所有节点的/etc/profile末尾添加export HADOOP_HOME/usr/local/hadoop export PATH$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin export HADOOP_CONF_DIR$HADOOP_HOME/etc/hadoop执行source /etc/profile使配置生效。4.3 核心配置文件修改进入$HADOOP_HOME/etc/hadoop目录修改以下文件core-site.xmlconfiguration property namefs.defaultFS/name valuehdfs://master:9000/value /property property namehadoop.tmp.dir/name value/usr/local/hadoop/tmp/value /property /configurationhdfs-site.xmlconfiguration property namedfs.replication/name value2/value /property property namedfs.namenode.name.dir/name value/usr/local/hadoop/hdfs/name/value /property property namedfs.datanode.data.dir/name value/usr/local/hadoop/hdfs/data/value /property property namedfs.namenode.http-address/name valuemaster:9870/value /property /configurationmapred-site.xmlconfiguration property namemapreduce.framework.name/name valueyarn/value /property /configurationyarn-site.xmlconfiguration property nameyarn.resourcemanager.hostname/name valuemaster/value /property property nameyarn.nodemanager.aux-services/name valuemapreduce_shuffle/value /property /configurationworkers文件slave1 slave24.4 目录创建与权限设置在所有节点执行mkdir -p /usr/local/hadoop/{tmp,hdfs/{name,data}} chown -R hadoop:hadoop /usr/local/hadoop5. 集群启动与验证5.1 初始化HDFS仅在Master节点执行su - hadoop hdfs namenode -format成功格式化后会看到Storage directory /usr/local/hadoop/hdfs/name has been successfully formatted的提示。5.2 启动集群在Master节点执行start-dfs.sh start-yarn.sh验证进程jpsMaster节点应有NameNode、SecondaryNameNode、ResourceManager Slave节点应有DataNode、NodeManager5.3 Web UI验证HDFS状态访问http://master:9870YARN状态访问http://master:8088在Web界面中应能看到2个活动的DataNode节点。5.4 运行测试作业# 创建HDFS目录 hdfs dfs -mkdir -p /user/hadoop/input # 上传测试文件 hdfs dfs -put $HADOOP_HOME/etc/hadoop/*.xml /user/hadoop/input # 运行WordCount示例 hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount /user/hadoop/input /user/hadoop/output # 查看结果 hdfs dfs -cat /user/hadoop/output/*6. 集群管理技巧6.1 常用管理命令命令功能描述hdfs dfsadmin -report查看HDFS集群状态yarn node -list查看YARN节点状态hdfs dfs -du -h /path查看HDFS目录空间使用yarn application -list查看运行中的YARN应用6.2 配置文件优化建议对于生产环境建议调整以下参数hdfs-site.xmlproperty namedfs.blocksize/name value256m/value /property property namedfs.namenode.handler.count/name value100/value /propertyyarn-site.xmlproperty nameyarn.nodemanager.resource.memory-mb/name value8192/value /property property nameyarn.scheduler.maximum-allocation-mb/name value8192/value /property6.3 常见问题排查DataNode无法启动检查/usr/local/hadoop/logs/下的日志文件确保所有节点的clusterID一致位于hdfs/name/current/VERSION和hdfs/data/current/VERSIONWeb UI无法访问检查防火墙状态验证端口是否监听netstat -tulnp | grep java磁盘空间不足定期清理/usr/local/hadoop/logs/下的旧日志调整hadoop.tmp.dir位置到更大容量的分区7. 安全关闭集群当需要维护或关闭集群时应按以下顺序操作# 在Master节点执行 stop-yarn.sh stop-dfs.sh # 验证所有Java进程已退出 jps对于长期运行的集群建议配置日志轮转和监控告警可以使用以下命令设置每日日志压缩# 配置log4j日志轮转 vim $HADOOP_HOME/etc/hadoop/log4j.properties添加以下内容log4j.appender.DRFAorg.apache.log4j.DailyRollingFileAppender log4j.appender.DRFA.DatePattern.yyyy-MM-dd log4j.appender.DRFA.MaxBackupIndex30