第3章Hadoop基础操作

使用mysql统计一天需要6分10秒

统计一月数据,mysql无法满足

Hadoop可以解决该统计任务

HDFS

计算资源分布在集群的各个节点上

通过ResourceManager与NodeManger协同调配

Web方式

命令行方式

块大小:128M

Web方式

http://10.255.10.65:19888/

http://192.168.137.133:9870/

说明：
在外网:wget https://hddly.oss-cn-hangzhou.aliyuncs.com/down/file/email_log.tar.gz
在内网:wget http://10.255.10.50/file/email_log.txt

cd /root/hadooop

tar -zxvf ./email_log.tar.gz

hdfs dfs -mkdir -p /user/myname

hdfs dfs -copyFromLocal email_log.txt /user/myname/

hdfs dfs -moveFromLocal email_log.txt /user/myname/

hdfs dfs -put email_log.txt /user/myname/email_log.txt

hdfs dfs -copyToLocal /user/myname/email_log.txt ./

hdfs dfs -get /user/myname/email_log.txt ./

hdfs dfs -cat /user/myname/email_log.txt

hdfs dfs -tail /user/myname/email_log.txt

hdfs dfs -rm 文件

hdfs dfs -rmdir 目录

hdfs dfs -ls 目录

hdfs dfs -ls -R 目录

要求:请通过命令行方式在本小组集群的HDFS文件系统中
创建目录： /user/myname
(myname都要替换为本人姓名全拼)

hdfs dfs -mkdir -p /user/myname

要求:通过命令行方式实现：
1，下载文件email_log.txt（可通过：wget http://10.255.10.50/file/email_log.txt）到本地 /user/hadoop目录下；
2）上传文件email_log.txt到HDFS的本组员私人目录下；
3）上传文件email_log.txt到HDFS的本组员私人目录下并重命名为1.txt；
4）将HDFS上的私人目录下的1.txt下载到本地 /user/hadoop目录下
5) 将本地目录下的1.txt转移到HDFS的本组员私人目录下
6) 使用tail命令查看本组员私人目录下的2.txt文件
7）使用cat命令查看本员私人目录下的2.txt文件，按Ctrl+C中继查看

命令集

要求 :通过命令行方式实现：
1）在HDFS文件系统中本组员的私人目录下创建tmp目录；
2）将本地/user/hadoop目录下的email_log.txt上传到HDFS文件系统中本组员的私人目录下的tmp目录下，并重命名为2.txt；
3）删除DFS文件系统中本组员的私人目录下的tmp目录下的2.txt；
4）删除DFS文件系统中本组员的私人目录下的tmp目录

命令集

hdfs dfs -rm -r 目录

hdfs dfs -du 目录

-getmerge

-grep

设置允许快照:

创建快照

先删除测试文件

再快照恢复

wordcount:词频统计

pi:估算圆周率

wordmean:单词平均长度

wordmedian:单词长度中位数

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_CLASS</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_CLASS</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_CLASS</value>
</property>
<property>
<name>yarn.app.mapreduce.am.resource.mb</name>
<value>256</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>512</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>256</value>
</property>
<property>
<name>yarn.application.classpath</name>
<value>$HADOOP_CLASS</value>
</property>
</configuration>

hadoop jar ./hadoop-mapreduce-examples-3.3.1.jar pi 10 100

或 hadoop jar ./hadoop-mapreduce-examples-2.10.1.jar pi 10 100

hadoop jar ./hadoop-mapreduce-examples-3.3.1.jar wordcount /user/root/email_log.txt /user/root/output4

hadoop jar ./hadoop-mapreduce-examples-3.3.1.jar wordmean /user/root/email_log.txt /user/root/output5

result:
count 8000000
length 210379675

一般地，n个数据按大小顺序排列，处于最中间位置的一个数据（或最中间位置的两个数据的平均数）叫这组数据的中位数。

hadoop jar ./hadoop-mapreduce-examples-3.3.1.jar wordmedian /user/root/email_log.txt /user/root/output6

hadoop jar ./hadoop-mapreduce-examples-3.3.1.jar wordstandarddeviation /user/root/email_log.txt /user/root/output7

result:
count 8000000
length 210379675
square 1481473227

网络配置

Hadoop版本

操作终端机hosts修改

查询集群的存储系统信息HDFS

查询 集群的计算资源信息YARN

查询监控日志JOB

说明:以下内容中myname都要替换为本人姓名全拼

在Hdfs中创建私人目录

文件上传下载查看操作

文件或目录删除操作

说明:以下内容中myname都要替换为本人姓名全拼

cd /usr/local/hadoop-3.3.1/share/hadoop/mapreduce

词频统计

执行估算PI值的任务

计算单词平均长度

计算单词长度中位数

计算单词长度标准差

查看任务运行情况

管理任务,中断任务

Hadoop2.x

Hadoop2.x

Web方式

fdisk /dev/vda
进入后输入:n
然后选择主分区:p
使用默认分区号:3
回车默认
回车默认
使用wq退出

partprobe

pvcreate /dev/vda3

vgextend centos /dev/vda3

lvextend -L +10G /dev/centos/root

xfs_growfs /dev/mapper/centos-root

hdfs dfs -chmod -R 777 /user

cd /usr/local/hadoop-3.3.1/etc/hadoop

配置core-site.xml

配置hdfs-site.xml

2022-02-09 08:06:22,360 WARN hdfs.DataStreamer: Exception in createBlockOutputStream blk_1073741958_1134
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:586)
at org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:253)
at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1757)
at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1711)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:707)

systemctl disable firewalld

service stop firewalld

2022-02-09 08:06:22,540 WARN hdfs.DataStreamer: Exception in createBlockOutputStream blk_1073741960_1136
java.io.IOException: Got error, status=ERROR, status message , ack with firstBadLink as 192.168.137.136:9866
atorg.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:134)
atorg.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:110)
atorg.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1810)
atorg.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1711)
atorg.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:707)

vi /usr/local/hadoop-3.3.1/etc/hadoop/mapred-site.xml
更改三处:
HADOOP_MAPRED_HOME=$HADOOP_CLASS
添加一处
<property>
<name>yarn.application.classpath</name>
<value>$HADOOP_CLASS</value>
</property>

bin/hadoop dfsadmin -safemode leave

INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on default port 9864, call Call#159 Retry#0 org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol.rollEditLog from 10.255.10.100:39090: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Log not rolled. Name node is in safe mode.

The minimum number of live datanodes is not required. Safe mode will be turned off automatically once the thresholds have been reached. NamenodeHostName:c100

因磁盘空间不足，内存不足，系统掉电等其他原因导致dataNode datablock丢失

bin/hadoop dfsadmin -safemode leave

[root@c100 logs]# hdfs dfsadmin -safemode leave
Safe mode is OFF
[root@c100 logs]# hdfs fsck / -delete
Connecting to namenode via http://c100:9870/fsck?ugi=root&delete=1&path=%2F
FSCK started by root (auth:SIMPLE) from /10.255.10.100 for path / at Thu Apr 28 02:54:52 EDT 2022


/user/chenyanfang/1.txt: MISSING 2 blocks of total size 218379675 B.
/user/chenyanfang/2.txt: MISSING 2 blocks of total size 218379675 B.
/user/chenyanfang/email_log.txt: MISSING 2 blocks of total size 218379675 B.
/user/liuchenling/1.txt: MISSING 2 blocks of total size 218379675 B.
/user/liuchenling/email_log.txt: MISSING 2 blocks of total size 218379675 B.
/user/root/1.txt: MISSING 2 blocks of total size 218379675 B.
/user/yeying/1.txt: MISSING 2 blocks of total size 218379675 B.
/user/yeying/email_log.txt: MISSING 2 blocks of total size 218379675 B.
Status: CORRUPT
Number of data-nodes: 0
Number of racks: 0
Total dirs: 15
Total symlinks: 0

Replicated Blocks:
Total size: 1747037400 B
Total files: 8
Total blocks (validated): 16 (avg. block size 109189837 B)
********************************
UNDER MIN REPL'D BLOCKS: 16 (100.0 %)
MINIMAL BLOCK REPLICATION: 1
CORRUPT FILES: 8
MISSING BLOCKS: 16
MISSING SIZE: 1747037400 B
********************************
Minimally replicated blocks: 0 (0.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 0.0
Missing blocks: 16
Corrupt blocks: 0
Missing replicas: 0
Blocks queued for replication: 0

Erasure Coded Block Groups:
Total size: 0 B
Total files: 0
Total block groups (validated): 0
Minimally erasure-coded block groups: 0
Over-erasure-coded block groups: 0
Under-erasure-coded block groups: 0
Unsatisfactory placement block groups: 0
Average block group size: 0.0
Missing block groups: 0
Corrupt block groups: 0
Missing internal blocks: 0
Blocks queued for replication: 0
FSCK ended at Thu Apr 28 02:54:53 EDT 2022 in 396 milliseconds


The filesystem under path '/' is CORRUPT
[root@c100 logs]#

<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop-3.3.1/tmp/hadoop-${user.name}</value>
</property>

在master主机上重新格式化namenode

查看集群新的集群ID

进入各从机，将从机VERSION的 clusterID改成新集群的clusterID

<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>

java.io.IOException: org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid resource request! Cannot allocate containers as requested resource is greater than maximum allowed allocation. Requested resource type=[memory-mb], Requested resource=<memory:1536, vCores:1>, maximum allowed allocation=<memory:1024, vCores:2>, please note that maximum allowed allocation is calculated by scheduler based on maximum resource of registered NodeManagers, which might be less than configured maximum allocation=<memory:1024, vCores:4>

Yarn部署到小内存主机上，默认的配置会由于资源不足导致简单的任务也无法执行成功

配置在2G的内存的主机

mapred-site.xml

/yarn-site.xml

yarn-site.xml

检查内网，外网是否有相同的域名，如 home.hddly.cn是否在内网也存在

在WEB端datanode页的node处

hdfs dfsadmin -printTopology

find / -size -5000M -size +500M

hdfs dfs -put /var/log/messages-20220208 /user/root/message.txt

cd /usr/local/hadoop-3.3.1/share/hadoop/mapreduce

hadoop jar ./hadoop-mapreduce-examples-3.3.1.jar wordcount /user/root/message.txt /user/root/output-message

cd /usr/local/hadoop-3.3.1/logs

ls -lrt

hdfs dfs -put ./hadoop-root-nodemanager-c22.log /user/limm/
c22要改为具体的文件名

cd /usr/local/hadoop-3.3.1/share/hadoop/mapreduce

hadoop jar ./hadoop-mapreduce-examples-3.3.1.jar wordcount /user/limm/hadoop-root-nodemanager-c22.log /user/limm/output-nodemanager_log

掌握HDFS的基本操作

掌握提交MapReduce 任务

掌握对MapReduce任务的查询与中断

在集群服务器的本地目录hadoop-root-nodemanager-*.log，要求对文件中的单词进行统计，求单词的平均长度

上传文件hadoop-root-nodemanager-*.log到 hdfs的:/user/myname 目录下

使用官方的/hadoop-mapreduce-examples-3.3.1.jar提交MR任务，将结果输出到hdfs: /user/myname/output_nodemanager_wordmean 目录下

查看输出结果:查看hdfs:/user/myname/output_nodemanager_wordmean目录下结果

1，环境说明:本小组主机:,本小组成员机:,本成员机:

2，在http://master:9870上拍照截取本小组集群中本成员目录下/user/myname中上传的文件

3，在linux本组员的虚拟机上，截图运行 mr任务的命令行，以及运行结果截图，至少两张截图

4，在http://master:8088上拍照截取本组员运行的任务记录行，和任务详细信息界面，至少两张截图

5，查看http://master:9870的文件/user/myname/output_nodemanager_wordmean目录的结果文件内容，并截图

掌握查询 MapReduce任务信息

掌握查询集群的计算资源信息

掌握中断执行中的MapReduce任务

在集群服务器的目录:本地目录/usr/local/hadoop-3.3.1/logs/下，将.log的文件上传集群hdfs的 /user/myname/logs目录下
依次提交词频统计任务(wordcount)、计算平均长度任务(wordmean)、计算单词长度中位数任务(wordmedian)，查看当前集群的计算资源使用情况，以及任务列表信息，中断第2个任务(wordmean),观察后续任务的执行情况

上传日志文件hadoop-root-*.log到 hdfs的:/user/myname/logs 目录下

使用CRT分3次打开本组员服务器，这样有3个Tab页同时连接到同个服务器

使用官方的/hadoop-mapreduce-examples-3.3.1.jar分别提交MR
任务(wordcount、wordmean、wordmedian)，将结果输出到hdfs:
/user/myname/output_logs_wordcount,
/user/myname/output_logs_wordmean,
/user/myname/output_logs_wordmedian
目录下

进入http://master:8088站点，打开MapReduce任务列表

观察任务(wordcount、wordmean、wordmedian)任务运行情况，查看任务分别是由哪些节点来完成的

找到第2个任务wordmean的任务，进入该任务详细信息，然后中断它

1，环境说明:本小组主机:,本小组成员机:,本成员机:

2，在linux本组员的虚拟机上，截图运行 mr任务(wordcount、wordmean、wordmedian)的包含任务号的截图( 至少3张)
任务号如：job: http://master:8088/proxy/application_1651126254203_0008/

3，在http://master:8088上拍照截取本组员运行的3个任务记录详细信息，其中wordcount(FINISHED)、wordmean(Killed)、wordmedian(FINISHED)