|
|
[admin@hadoop-1 hadoop-3.2.2]$ hadoop jar /data/module/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar pi 10 100 w' |" N' b! ~5 [
Number of Maps = 102 R/ I. ~2 N, }7 y7 ^0 V
Samples per Map = 100
: T( P4 O6 L7 }& I& qWrote input for Map #0
# N+ ]0 d6 C6 @' TWrote input for Map #1
- f7 K% n8 b: H9 l- n' KWrote input for Map #2! I4 |5 K' t# Z- K( f P: m
Wrote input for Map #31 Y. Z6 b: p$ G: P( `+ w
Wrote input for Map #4
! g: T3 }" R+ UWrote input for Map #5
( X/ W) V6 h4 B# j6 S2 tWrote input for Map #6
5 n5 L; B" H. z8 E& z3 a5 q, {Wrote input for Map #73 l4 J7 }% T1 ?: ~6 T' U( m
Wrote input for Map #8
% W" f: O0 |5 n0 t1 w" `Wrote input for Map #9. u9 B/ C; E! u, n+ ]. g$ u0 H+ N
Starting Job* M% s3 G0 F; y& N8 P Q
2022-11-16 15:32:34,946 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
7 G3 |4 v9 \# _: h" T# Z2022-11-16 15:32:35,174 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/admin/.staging/job_1668583281306_0001. Z5 D5 d3 f& O# x7 h
2022-11-16 15:32:35,389 INFO input.FileInputFormat: Total input files to process : 10! e/ F+ _, f- c9 N
2022-11-16 15:32:35,592 INFO mapreduce.JobSubmitter: number of splits:104 o2 _- f# _2 b5 d( @ d$ u
2022-11-16 15:32:35,806 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1668583281306_0001
, C6 ^# e; J' ~$ F$ Z+ G6 b2022-11-16 15:32:35,808 INFO mapreduce.JobSubmitter: Executing with tokens: []
8 w$ i7 h4 I) e2022-11-16 15:32:36,044 INFO conf.Configuration: resource-types.xml not found: c" U; e* T* Y7 ~0 k; f" o0 D. [
2022-11-16 15:32:36,044 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
" ^, b9 }9 z% v5 U- A/ Q2022-11-16 15:32:36,423 INFO impl.YarnClientImpl: Submitted application application_1668583281306_00019 {/ F4 }/ Y+ \0 |
2022-11-16 15:32:36,504 INFO mapreduce.Job: The url to track the job: http://hadoop-2:8088/proxy/application_1668583281306_0001/
( ^ C3 S% Y# ^% P' s2022-11-16 15:32:36,505 INFO mapreduce.Job: Running job: job_1668583281306_0001
+ i9 q3 s' d" q0 @( g' U% F2022-11-16 15:32:41,561 INFO mapreduce.Job: Job job_1668583281306_0001 running in uber mode : false2 e7 n. ~& F( w* {, v
2022-11-16 15:32:41,563 INFO mapreduce.Job: map 0% reduce 0%
! w8 g! m. T+ K- O2022-11-16 15:32:41,583 INFO mapreduce.Job: Job job_1668583281306_0001 failed with state FAILED due to: Application application_1668583281306_0001 failed 2 times due to AM Container for appattempt_1668583281306_0001_000002 exited with exitCode: 1
+ l m( A6 u, _& l' o+ D/ [$ JFailing this attempt.Diagnostics: [2022-11-16 15:32:40.765]Exception from container-launch.0 w8 ], h% g9 v+ M' j+ N9 S
Container id: container_1668583281306_0001_02_000001
! q1 F9 C9 U: F) S6 vExit code: 1
3 e" f m2 P8 [( o8 A: F; i8 v7 E6 c3 `# a! J
[2022-11-16 15:32:40.799]Container exited with a non-zero exit code 1. Error file: prelaunch.err." i; n$ L. ], |" o3 y3 A7 p5 j
Last 4096 bytes of prelaunch.err :
- }1 k- o" C7 _$ i7 j1 hLast 4096 bytes of stderr :9 s& |& P% ~ L4 q8 i
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster6 x# o$ P7 z% `5 \
/ K7 w" E% ]( q
Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
' a. v) Z( h$ U& h/ H$ a. x<property>
6 W1 Y! y$ T4 M8 a) N$ x <name>yarn.app.mapreduce.am.env</name>" ~4 Z4 }; x1 o* l6 x
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>1 [1 x( ~) C4 J$ R$ ?
</property>( |! b. n: A5 U0 q2 E) n
<property>
! B" g5 d+ ?' Z <name>mapreduce.map.env</name>' r: {8 E6 a A C& P1 ^% i6 ~
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
; v8 H! [1 i- }8 y$ }; @! ?) l4 k$ q) b</property>5 x# R d5 X6 Y1 T( k* l3 `/ Z
<property> }7 Y0 R/ S8 J+ O* K
<name>mapreduce.reduce.env</name>
: Y2 V" j, f7 Q( N1 X4 ?1 u <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>! w3 `: L' b0 K p; y, H: m
</property>
2 X) r" Y0 f5 s* J1 m5 ]8 Z
r5 G7 Y0 Z, n[2022-11-16 15:32:40.800]Container exited with a non-zero exit code 1. Error file: prelaunch.err.) ~1 n& X: o! l% ?; Q9 s, k3 k$ E' d
Last 4096 bytes of prelaunch.err :
$ ?1 f7 m; N5 Z5 `3 n/ wLast 4096 bytes of stderr :
! Q& @( J4 b' X) U( N0 m8 FError: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
& {/ ~; g6 F8 _ k" H" r
4 N. Y- \. @' f5 CPlease check whether your etc/hadoop/mapred-site.xml contains the below configuration:1 E2 m$ s" E# e: i3 `
<property>
$ |' T, m5 D% W4 F <name>yarn.app.mapreduce.am.env</name>" Y; q7 |; H/ J6 y" j
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>0 L A% q9 L: S* q* ~
</property>/ A1 u+ V8 ?4 t7 J% o$ d( R& _2 ^
<property>
5 k: L* q' _" S6 d <name>mapreduce.map.env</name>
0 w3 n! S- v. }; R4 q: L <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>/ R7 L' d. L: n% O) D* O8 d
</property>
( h1 K8 P- v8 y# P1 A" F" w* N<property>
|6 z$ n' t+ v# b4 I& G; n <name>mapreduce.reduce.env</name>- X( ?; O s$ j9 z
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>6 k& q( V, @- F; z: Y2 P" M2 m' \
</property>1 ]# B# m+ N6 I$ I" q
8 p- g( g% z5 {; PFor more detailed output, check the application tracking page: http://hadoop-2:8088/cluster/app/application_1668583281306_0001 Then click on links to logs of each attempt.
! H' [& K6 u5 I1 J0 F4 J1 c, _. Failing the application.
3 N3 _; W$ G4 r7 i2022-11-16 15:32:41,608 INFO mapreduce.Job: Counters: 0( {+ a0 [; M2 T' w0 N
Job job_1668583281306_0001 failed!
) F3 t8 ]3 d5 ] B$ d6 C, r; p% a# ` j1 {8 {$ V, h
分析失败，解决方法：6 X m: w* d7 E
& O4 y) V2 [5 b* y- P[admin@hadoop-1 hadoop-3.2.2]$ vim etc/hadoop/mapred-site.xml2 g4 v2 Z$ g3 [$ @5 b9 h0 P% H
添加如下内容：/ P& j* |1 {5 K2 x# J1 \
0 i" g9 Q' A7 u* y* C<property>
6 K& G, l; L' [7 P { <name>yarn.app.mapreduce.am.env</name>
" r+ V% s; p, i <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
+ i9 c, v* j) j3 W, Q2 [</property>
* H$ ~. ~% B% B1 C8 q, q$ x$ f: s<property>
3 R% T% {$ p) M1 N' f3 N <name>mapreduce.map.env</name>4 _8 C* H# q- S$ H% N* E* u5 I
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
) P6 b$ U a% T, x2 f O/ j; k</property>: h( P7 Y2 b4 j& N$ _ }- c- v0 h
<property>) ^( h4 Y9 j) K5 B, g
<name>mapreduce.reduce.env</name>8 S( i9 C) A% q- H0 E
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
R2 Q0 f! K8 d8 O" {, |</property>5 ^% Z2 x# v' ^. e( Y. D V/ k
3 R1 e& V; h5 ]* J5 z! Y! B+ ~0 h9 G% a. m5 ~1 `8 U: T
同步到其他节点：! R+ ~- u4 R+ \2 W1 G
$ }7 r `) r* o- O- v
1 w7 ~9 V/ | f! n/ X" X c. \5 F4 c0 }8 u2 V
; r3 [. h9 f! B$ o( k) h0 n/ c[admin@hadoop-1 hadoop-3.2.2]$ xsync etc/hadoop/9 O. i8 [% a8 |' o( B1 H
==============hadoop-1==================
) ^5 ~2 {5 v. _7 csending incremental file list3 s8 ]5 e) F1 Y& N
_1 j/ ?) ^% m% k( H. c
sent 1,004 bytes received 18 bytes 2,044.00 bytes/sec7 @. A# T3 [4 Y
total size is 115,250 speedup is 112.77
9 W9 X, @. `; {7 x4 w; G5 ~& C==============hadoop-2==================, Q" f( I/ ^' ^/ H
sending incremental file list
7 E3 \& r/ l% @3 _" o# nhadoop/
" X' E9 E: n8 n5 l* W) ^$ b7 P& dhadoop/mapred-site.xml/ P# ?& _9 n$ l1 ^: {
" t1 _2 i6 d- w. vsent 1,251 bytes received 52 bytes 2,606.00 bytes/sec( Z* Q2 q5 q- S& m
total size is 115,250 speedup is 88.45% A1 K9 @3 N9 H
==============hadoop-3==================1 K$ b2 b: e9 Q w+ g
sending incremental file list+ i: a8 [) u* v; Z( F% }
hadoop/
+ a) d: d& i& F% h% Uhadoop/mapred-site.xml
$ b1 [& h2 v" @3 \% j- V- a; b+ Y" r8 f
sent 1,251 bytes received 52 bytes 2,606.00 bytes/sec
# | H$ d% E! A9 Q/ p3 Gtotal size is 115,250 speedup is 88.45
' C5 d- z; K( [% G8 f% |# f$ S5 f% x( \$ p5 [
+ J. d* g9 y8 j再次验证：
6 m; A) l3 V" R% W$ z* g; H% i+ p
. N. B2 h$ z+ ^: `! `( Y[admin@hadoop-1 hadoop-3.2.2]$ hadoop jar /data/module/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar pi 10 100
1 {9 c5 e x0 f0 F. kNumber of Maps = 10/ _2 |8 S" h: D: O. `* l
Samples per Map = 100
8 d4 l6 m7 D# B; BWrote input for Map #0) j5 z+ t8 y) S
Wrote input for Map #1# I7 `8 B& l. [% l( m. n- n
Wrote input for Map #2% f7 A' Z6 H" m! {$ M' |, R1 ?( S0 |
Wrote input for Map #3
5 `! y! ]8 R ~, y q" N# \ w) W$ aWrote input for Map #4' o- o& C: n; k0 q
Wrote input for Map #5
+ [ \4 ~1 |% b' {8 G9 MWrote input for Map #6
' k. m5 L# v8 z# N: A+ M9 _) OWrote input for Map #7, \) D( x; G0 M, A z& [
Wrote input for Map #8
9 u' D' ^3 s# V3 b, eWrote input for Map #9+ [! G9 g* f0 `: b( f% \
Starting Job
; J1 f1 q$ o! R+ c+ Y- D2022-11-16 15:36:05,630 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm29 E. X9 N, t1 r) f
2022-11-16 15:36:05,878 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/admin/.staging/job_1668583281306_0002# ]( T; M, P: S4 c+ t
2022-11-16 15:36:06,065 INFO input.FileInputFormat: Total input files to process : 10
$ Y. r/ ^. l# N2022-11-16 15:36:06,214 INFO mapreduce.JobSubmitter: number of splits:10
, l* g- T! j% i1 c; N: K) ?2022-11-16 15:36:06,444 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1668583281306_0002
6 [* d0 X! Q. f* F% o2022-11-16 15:36:06,446 INFO mapreduce.JobSubmitter: Executing with tokens: []* p; l3 e6 e: d9 \
2022-11-16 15:36:06,654 INFO conf.Configuration: resource-types.xml not found. p' t" ?$ v! m3 ^0 G
2022-11-16 15:36:06,654 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.* X; U) ]5 |, L* ~7 c/ y
2022-11-16 15:36:06,928 INFO impl.YarnClientImpl: Submitted application application_1668583281306_0002; D2 `% @" E1 M' `1 U- ~
2022-11-16 15:36:06,981 INFO mapreduce.Job: The url to track the job: http://hadoop-2:8088/proxy/application_1668583281306_0002/1 E* s+ c" r( e6 C9 p2 P
2022-11-16 15:36:06,982 INFO mapreduce.Job: Running job: job_1668583281306_0002
: R/ ?5 {9 ^$ b! l2022-11-16 15:36:17,134 INFO mapreduce.Job: Job job_1668583281306_0002 running in uber mode : false% C! N `" _4 ?0 ~/ w4 E
2022-11-16 15:36:17,136 INFO mapreduce.Job: map 0% reduce 0%
+ w, k/ O v( x& h5 v$ H2 G' ^. u7 ?2022-11-16 15:36:23,297 INFO mapreduce.Job: map 40% reduce 0%
; V& Q" T4 J# b% P2022-11-16 15:36:24,308 INFO mapreduce.Job: map 100% reduce 0%$ E" ?- U; Y6 I1 v d
2022-11-16 15:36:28,342 INFO mapreduce.Job: map 100% reduce 100%( d% b! o. I% p5 U" j0 l% X
2022-11-16 15:36:29,360 INFO mapreduce.Job: Job job_1668583281306_0002 completed successfully' q. `% U2 J% `$ f% C# x2 ~
2022-11-16 15:36:29,504 INFO mapreduce.Job: Counters: 54
9 y( m; s4 m; b; I8 P1 b( J" C File System Counters* O# _/ }/ \- V. z
FILE: Number of bytes read=226
8 e; \# p" Z* [' o" W, ^0 ?- T FILE: Number of bytes written=2630034- W: ~- [/ c$ @0 W m0 f
FILE: Number of read operations=0( l5 c8 L- o! x& W* U
FILE: Number of large read operations=03 N4 r) P9 {( E7 ?. ~/ w) q
FILE: Number of write operations=0) O' B4 u: x1 O! }
HDFS: Number of bytes read=2600
4 l$ N) C1 M( t0 P HDFS: Number of bytes written=215
# y4 N8 s' w2 r# @, {% ]' v HDFS: Number of read operations=45/ g: X- P: X4 O9 v' a( N8 J) x& `
HDFS: Number of large read operations=0
0 X$ c1 M+ ^- M HDFS: Number of write operations=37 H/ ^8 r: ]" \) m5 S
HDFS: Number of bytes read erasure-coded=0# X0 S3 q' M3 b, x, ~ W O" a, r
Job Counters
. p- N7 p P* z$ h" Z Launched map tasks=106 O: b$ d% S" U8 y( j! X
Launched reduce tasks=1
" K6 Q" H8 `" N1 C* c/ G Data-local map tasks=10
B) O1 U6 K6 x Total time spent by all maps in occupied slots (ms)=413943 Q4 i5 b- K5 g) n/ q, j" ?
Total time spent by all reduces in occupied slots (ms)=3116* v+ N& I( e& O* u) t* o% \7 i1 C
Total time spent by all map tasks (ms)=41394% ^5 `' l- K: ^0 s
Total time spent by all reduce tasks (ms)=3116' X. v1 Z, H+ L2 @0 [0 v) R
Total vcore-milliseconds taken by all map tasks=41394
* t3 {9 r( b8 \+ ^' S4 S Total vcore-milliseconds taken by all reduce tasks=3116- z: U* T% ]8 X
Total megabyte-milliseconds taken by all map tasks=42387456% s Q+ [, Y+ s# b; p& D0 a p$ }
Total megabyte-milliseconds taken by all reduce tasks=3190784, y% m. U" k7 T+ T9 F( I
Map-Reduce Framework: }1 m R4 u$ v/ j2 {
Map input records=102 H9 D, y' N% W8 O a2 K4 B- w
Map output records=20
! e$ W: Y) O# a F1 H: u6 h# x Map output bytes=180& E5 |9 x8 J+ Q; m4 ~/ Q) f/ O
Map output materialized bytes=280
. y! r h3 e0 P* E1 K. U Input split bytes=1420) W# a0 d( X9 ~
Combine input records=0
/ E' b$ K) q7 _) u Combine output records=0, W* S8 o# f K/ \
Reduce input groups=2
* J/ [- L: b8 `/ l Reduce shuffle bytes=280. g/ O+ h% b( ^- \! q
Reduce input records=20% X9 \6 o, ^' E( o
Reduce output records=0
7 W6 |( {/ w3 K( j8 f: e+ l! M( C. m5 C Spilled Records=40
. s9 D' E0 e% M% |5 W# `+ h- L# G Shuffled Maps =10
7 x+ N. d% w, ~' h+ l Failed Shuffles=0; m* h9 z! }3 P5 O
Merged Map outputs=10' V( \% ]* a+ r2 f1 j! E, t9 P2 d9 B
GC time elapsed (ms)=19151 Q3 L0 _- k5 ]7 Z$ j8 m1 ?
CPU time spent (ms)=6790- Q" h, X% i0 ?6 W' F0 e+ m$ O
Physical memory (bytes) snapshot=3809210368
: X) j3 p" z6 K8 d. } Virtual memory (bytes) snapshot=31150141440
- } e% x9 B1 j* K* | Total committed heap usage (bytes)=3503292416
/ m1 M. Q( Z5 o: i* P" c8 \ Peak Map Physical memory (bytes)=360734720
0 J% x0 B6 @0 z* c; C Peak Map Virtual memory (bytes)=2835021824" N9 ~( H* D( N8 G2 q8 U- E
Peak Reduce Physical memory (bytes)=279707648: m( m* h- g8 m+ q$ t# R
Peak Reduce Virtual memory (bytes)=2839097344
! s1 ?. H/ y2 ]4 e( m Shuffle Errors* L% F( Q7 v A3 m7 g: x1 ~
BAD_ID=0
, y( T l6 J2 F4 e- Z' \& d2 o CONNECTION=0) j; l1 F$ H* W: L
IO_ERROR=09 R/ c; ~' L& ~$ P F& M) j
WRONG_LENGTH=09 C: q: n$ b l" L' V: T
WRONG_MAP=0
" ]8 g) `9 P1 l3 a! g: s' f+ d WRONG_REDUCE=0
( E# [ b3 o1 x File Input Format Counters
4 R6 D6 d- d3 v) l' O* I$ e Bytes Read=1180
' `( k" v4 P \. J File Output Format Counters
; K7 Q* A4 X" i5 ` Bytes Written=97+ h. e7 S* }4 F0 I6 M0 x
Job Finished in 24.227 seconds
5 ~3 ^, S7 j8 i+ w/ L: @Estimated value of Pi is 3.14800000000000000000
8 v, j. v2 h4 ~+ U! o7 v
( {4 ]# b. W- f1 Y问题解决' p# v+ t3 O3 e" M5 u+ J
M9 p5 \3 }- J/ z
|
|
发表于 2022-11-16 17:00:20