|
|
[admin@hadoop-1 hadoop-3.2.2]$ hadoop jar /data/module/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar pi 10 100
`4 @: v* {8 X5 SNumber of Maps = 10
0 D1 B8 \# M& [4 c' _. o8 BSamples per Map = 100
8 i) [( _& `2 I+ \Wrote input for Map #0, t2 ~$ O# y+ P: n$ E {& c; D
Wrote input for Map #1: [1 E1 E( d5 d5 ?1 ^
Wrote input for Map #2! Y& w3 {1 g3 x' L
Wrote input for Map #3
( w6 Q) v+ h$ ]! {4 VWrote input for Map #4 X& m8 j3 G' J6 I! j
Wrote input for Map #5
/ c5 A0 l! N v4 ~' kWrote input for Map #6
6 U! h- s% k) qWrote input for Map #7" D1 l/ Q+ ~3 E
Wrote input for Map #8" T! w- d6 |; @6 _
Wrote input for Map #9
# W( ]( G6 @4 l, d4 H$ R7 D; G0 cStarting Job
+ c$ m' e4 V" g! h$ w2022-11-16 15:32:34,946 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2& G8 `; k) ]+ y/ ^; G9 w2 |
2022-11-16 15:32:35,174 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/admin/.staging/job_1668583281306_0001
3 R; f/ Q8 y- z2022-11-16 15:32:35,389 INFO input.FileInputFormat: Total input files to process : 10
7 p, r5 @9 T" ?2022-11-16 15:32:35,592 INFO mapreduce.JobSubmitter: number of splits:10
0 Y$ Y/ X. H; c2022-11-16 15:32:35,806 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1668583281306_0001
; I( h+ R5 d! g5 j D1 s3 `/ y; W2022-11-16 15:32:35,808 INFO mapreduce.JobSubmitter: Executing with tokens: []
" n% E0 I/ l4 B, i6 D! T4 m2022-11-16 15:32:36,044 INFO conf.Configuration: resource-types.xml not found
8 S4 N6 I0 m+ V2 ^" q% p7 c2022-11-16 15:32:36,044 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
0 A9 G) Z5 [. f. V3 G7 j# C: K2022-11-16 15:32:36,423 INFO impl.YarnClientImpl: Submitted application application_1668583281306_0001
7 Z& N# t# N! g, }/ f0 k, _, s2022-11-16 15:32:36,504 INFO mapreduce.Job: The url to track the job: http://hadoop-2:8088/proxy/application_1668583281306_0001/# Q6 s. K0 P1 P4 X
2022-11-16 15:32:36,505 INFO mapreduce.Job: Running job: job_1668583281306_0001
$ W1 k& j/ `* w2022-11-16 15:32:41,561 INFO mapreduce.Job: Job job_1668583281306_0001 running in uber mode : false
. j) @! e! F5 ]8 |2022-11-16 15:32:41,563 INFO mapreduce.Job: map 0% reduce 0%# ~$ W( A1 ~" C# X( ]6 O& |4 u& U
2022-11-16 15:32:41,583 INFO mapreduce.Job: Job job_1668583281306_0001 failed with state FAILED due to: Application application_1668583281306_0001 failed 2 times due to AM Container for appattempt_1668583281306_0001_000002 exited with exitCode: 17 h0 Y+ a6 D# G
Failing this attempt.Diagnostics: [2022-11-16 15:32:40.765]Exception from container-launch.+ F" Z! m& y" ]$ d$ m; p# ]/ c+ T% H
Container id: container_1668583281306_0001_02_000001
$ M5 B9 Y& c; l3 C4 t$ u3 n9 KExit code: 1
3 D/ A) y2 P# r3 T- F/ D
' q8 C. _3 F) {0 J. i1 |- \9 L[2022-11-16 15:32:40.799]Container exited with a non-zero exit code 1. Error file: prelaunch.err.! {4 }. F: G4 }1 v* T& u, j- J% w T
Last 4096 bytes of prelaunch.err :
1 F) P2 ~" s- P6 F. m" X: cLast 4096 bytes of stderr :
; ~! n A6 a g( @9 ?; D& A! K. FError: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster4 g7 ~" f$ b0 ~4 V) N4 c
( x' [9 h2 {) L+ E8 N" bPlease check whether your etc/hadoop/mapred-site.xml contains the below configuration:% r5 f. |) q# r
<property>2 Z! x, { Y/ @( o8 o% {; }- v
<name>yarn.app.mapreduce.am.env</name>, l9 y8 g. P) j
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
# B" F' ]1 O; S9 ]$ G+ z</property>3 Y3 ^ R( k' r+ |# j! m& G
<property>
, i! H% |$ d9 d3 r& i3 H1 @) h% { <name>mapreduce.map.env</name>
4 Q2 [" d' y+ N# K <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>' ^! i: f V9 S; S
</property>" _, {! z. E0 H$ H' Y( T
<property>! C) S* e* d+ L
<name>mapreduce.reduce.env</name>: j$ v( S! p2 z; q
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>' t" P& _. S: j+ p6 l$ z; y
</property>
( h# `$ }. ?; m9 a3 \& F
8 M) v4 J0 J* G( d2 n[2022-11-16 15:32:40.800]Container exited with a non-zero exit code 1. Error file: prelaunch.err.& o8 ]0 {) g: T4 X2 N
Last 4096 bytes of prelaunch.err :6 c/ E8 ~2 f" E4 d
Last 4096 bytes of stderr :' `4 p% R m" ?( I. F+ y
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster/ {5 N. ?( @" P9 r+ B* n2 S, W2 h
' b) g5 n/ p; W4 e6 q' w9 D( X
Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:# t/ S$ o" x8 H* X/ H" {: v4 F
<property>+ F' [, Q9 ]% F' E9 W
<name>yarn.app.mapreduce.am.env</name>9 U9 U9 L8 m( [; D5 j) S* ]
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>9 ~/ z+ N, [& S4 \: f8 }
</property>
# q+ ]% ~. ~+ f: ~1 V0 l<property>
& j: o2 s$ Y" ~5 F <name>mapreduce.map.env</name>, {; F" o' b0 B" a6 e
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>5 z& z2 q. ^" |. [8 _) \
</property>
9 ]1 C8 q, p: E- u% Y9 E<property>8 t' }9 b" Z8 X
<name>mapreduce.reduce.env</name>
# W1 k$ ?( p( }# g$ K& \7 b <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>. B0 c3 ]: ?* s& [) n
</property>. k+ f' h* L/ D
+ f4 l3 C. Y4 N/ n! Z( ~For more detailed output, check the application tracking page: http://hadoop-2:8088/cluster/app/application_1668583281306_0001 Then click on links to logs of each attempt.- _/ ^1 A) e; }. J- u
. Failing the application. p2 b# m' t3 ~; c/ a7 w1 I$ A
2022-11-16 15:32:41,608 INFO mapreduce.Job: Counters: 0/ X2 v) e( b# d
Job job_1668583281306_0001 failed!
& D7 Q% n& y1 x' h6 v9 l7 ~5 {1 A! y( u
分析失败，解决方法：% ~0 D P$ l! ~/ o
7 t0 x( r1 v9 m5 M) s[admin@hadoop-1 hadoop-3.2.2]$ vim etc/hadoop/mapred-site.xml$ _8 D' k% b, C0 Y" Z
添加如下内容：) o' I3 V9 V- L; T- ~1 M( h
# K% _0 I5 k2 }( _* F
<property>
, P) W# n) W7 ?* j9 q R" P <name>yarn.app.mapreduce.am.env</name>; Y/ Y; U& s- S8 _ O
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>" {* @2 ?; Y" j4 z% s
</property>
& H( h4 J% R5 A<property>3 W6 N) `- `; @! C) E
<name>mapreduce.map.env</name>- ~$ z2 C. c7 T3 a9 u
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>' _3 M% F# q- u3 l& l9 a0 D% t/ @
</property>+ J& ?- e5 h7 Q r+ ~
<property>
$ k$ Z( H" t& Q$ y7 p <name>mapreduce.reduce.env</name>
" J8 I, y7 K3 Z <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
y! z) {, u" Z7 s' E3 H* ?' S</property>
! b& X I4 ?- Y: i; T9 K1 H- E
6 w) Y \5 l* g' d7 P
/ a0 q7 Y# _( `7 s* I同步到其他节点：
$ ?* v' ]! K' w; {2 `7 v0 H. L
! U% i8 Y( D/ e) a" o; S$ N! [1 x) K+ N. \, E
6 u' c% T `3 C" j( W
3 g+ {1 d9 O6 ^[admin@hadoop-1 hadoop-3.2.2]$ xsync etc/hadoop/
" a, ~* E7 \ z, ~9 d5 W==============hadoop-1==================# _; ~( {3 F6 S! ]; F9 o! R; D
sending incremental file list* E8 x" P- y$ K. ~
, ^ |9 W' w+ h5 D8 }: ]+ o: C" u; [
sent 1,004 bytes received 18 bytes 2,044.00 bytes/sec I8 Z( R2 _+ y6 d* V
total size is 115,250 speedup is 112.77( W: Z& T+ C: F# T% c4 c% J* D
==============hadoop-2================== V: P9 } O- M5 I4 e' Q) h
sending incremental file list9 Q( h% G) y1 e5 O1 d0 L
hadoop/. o( V- ^- g8 u+ Z4 q# f: z
hadoop/mapred-site.xml
: y% x" `6 k- D# c
& b5 o X! a! h* d( S3 ksent 1,251 bytes received 52 bytes 2,606.00 bytes/sec: r h m9 i0 N, }3 i
total size is 115,250 speedup is 88.45
- K- H% Z; a( t! y4 c==============hadoop-3==================
8 }9 I$ [' z' [( y* `sending incremental file list
5 x& M3 Y* F$ i% W x4 l2 |2 u) v# ahadoop/9 r3 O6 {$ Q, ~+ N- |. E( W7 ]& t
hadoop/mapred-site.xml) `1 s0 p' s) D5 y
1 u4 {+ ?- c! l1 @" ~! v4 {; }6 H
sent 1,251 bytes received 52 bytes 2,606.00 bytes/sec
9 w* e5 N; k) D0 ?" t& ~9 C0 Qtotal size is 115,250 speedup is 88.45* Q1 B8 d* j, T
4 I h ]) m' E r n
2 s( y7 k) T1 L: D+ D9 j再次验证：
2 R z: ]9 N* d+ M2 @( `9 A9 Z
' d5 a3 d" c2 j2 U[admin@hadoop-1 hadoop-3.2.2]$ hadoop jar /data/module/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar pi 10 100
: q- G! Y! o" n2 U& i% RNumber of Maps = 10
0 _' V& Z: N4 I* b( m. iSamples per Map = 100
3 C( \% f \5 eWrote input for Map #0
1 U7 \5 Q1 M. UWrote input for Map #1
1 D3 S+ M6 |$ x9 Y8 T3 ~Wrote input for Map #2
: b% Z& T: a+ nWrote input for Map #3
( Z7 p$ v( I- \: q% FWrote input for Map #4% i- Y6 w- K5 E- j7 Z# y
Wrote input for Map #54 g) a2 y( j# w+ T
Wrote input for Map #6
( Y3 k' Z! F9 O3 W: J6 xWrote input for Map #7- A$ m! C% W; S s' Q+ R y( N" d
Wrote input for Map #8
: i6 X2 g! Z3 g& CWrote input for Map #9+ K# l2 m3 i# g: c! N- f
Starting Job
7 O& k: W( |, b Y, x+ t. s; f2022-11-16 15:36:05,630 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
) ?6 [9 c* k' \5 P3 _: l2022-11-16 15:36:05,878 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/admin/.staging/job_1668583281306_0002
; G8 @9 Q$ D2 Q: s+ B7 S2022-11-16 15:36:06,065 INFO input.FileInputFormat: Total input files to process : 100 R o: k! k0 O2 l `/ s
2022-11-16 15:36:06,214 INFO mapreduce.JobSubmitter: number of splits:10
* M6 U# F+ O* a- f) R& u/ @# R2022-11-16 15:36:06,444 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1668583281306_0002" g9 X/ l( I" }7 r% m; E9 ~2 M
2022-11-16 15:36:06,446 INFO mapreduce.JobSubmitter: Executing with tokens: []3 q! r: N1 ^6 D* t1 H: P! L: {& `
2022-11-16 15:36:06,654 INFO conf.Configuration: resource-types.xml not found
: e4 k1 a$ ~3 g8 O' N* K# \2022-11-16 15:36:06,654 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.# b9 ]9 D! u' H% Y& \3 v
2022-11-16 15:36:06,928 INFO impl.YarnClientImpl: Submitted application application_1668583281306_0002
, |3 Q& m9 I; i4 d& \6 ~$ M b2022-11-16 15:36:06,981 INFO mapreduce.Job: The url to track the job: http://hadoop-2:8088/proxy/application_1668583281306_0002/
& ?+ _! o2 N0 |' S) U0 E1 k2022-11-16 15:36:06,982 INFO mapreduce.Job: Running job: job_1668583281306_00020 W# M$ y, F7 D5 w. p* Z9 @: Z) Y
2022-11-16 15:36:17,134 INFO mapreduce.Job: Job job_1668583281306_0002 running in uber mode : false
: D) q s" N2 U6 ?# U: T2022-11-16 15:36:17,136 INFO mapreduce.Job: map 0% reduce 0%
5 x+ Y# t9 [- X9 P& U2022-11-16 15:36:23,297 INFO mapreduce.Job: map 40% reduce 0%
3 R. d* j$ h8 c% r5 q2022-11-16 15:36:24,308 INFO mapreduce.Job: map 100% reduce 0%4 M4 O" ~1 ~7 N
2022-11-16 15:36:28,342 INFO mapreduce.Job: map 100% reduce 100%- P, S6 [& R l) l
2022-11-16 15:36:29,360 INFO mapreduce.Job: Job job_1668583281306_0002 completed successfully
6 ?, G y4 R0 E- R) ], H2022-11-16 15:36:29,504 INFO mapreduce.Job: Counters: 54
% i( \5 k1 T- a" u; t- a+ O A( t File System Counters* D3 l4 r7 D7 A' O" ~/ ?: Z
FILE: Number of bytes read=226
) S# i8 V) Z1 p/ V3 S4 U FILE: Number of bytes written=2630034$ a$ S1 @6 v/ d6 T
FILE: Number of read operations=0, d. r5 W4 O7 N5 ~, Q: r1 `# \
FILE: Number of large read operations=0# s% M; X8 ]2 h; \8 t' A, W
FILE: Number of write operations=05 k) v/ \( p% t" N
HDFS: Number of bytes read=2600
5 z* P1 ?# ?! t1 b) ` HDFS: Number of bytes written=215
3 V0 M, p7 S! ~5 ]# E# o# Y3 G& u HDFS: Number of read operations=450 C# @& P% P# ~1 S( D; Y3 q
HDFS: Number of large read operations=0) |" c. [" s: d4 c' I7 W( o
HDFS: Number of write operations=3+ w1 P/ O( l$ S: A& ]6 t/ v5 C2 L5 v
HDFS: Number of bytes read erasure-coded=0
' d+ J1 L( S: g+ O Job Counters
; u# G2 \) L c: a: f5 V& o Launched map tasks=10
9 T" b: I" J0 ?. N/ F' n. ` Launched reduce tasks=1) b; C# l$ X# Y6 V( \( l& O; c
Data-local map tasks=101 J( \ x7 }8 p0 _" r* g
Total time spent by all maps in occupied slots (ms)=413941 x( D/ i6 @0 N; x& x
Total time spent by all reduces in occupied slots (ms)=31164 z; o# X* t; w, Y
Total time spent by all map tasks (ms)=41394
9 G4 n! i; G, s F' S: B Total time spent by all reduce tasks (ms)=31160 R4 S# j# x) h7 _9 E( v6 M" s
Total vcore-milliseconds taken by all map tasks=41394- Z7 R0 }/ Z% q2 M
Total vcore-milliseconds taken by all reduce tasks=3116
. _7 A2 W \' K* P- H, O Total megabyte-milliseconds taken by all map tasks=423874560 A7 j- |+ {5 K6 A
Total megabyte-milliseconds taken by all reduce tasks=31907845 {: \# { ? z1 c! K* O- j
Map-Reduce Framework
' f2 I& i! ]: U: l Map input records=10
& j( ^5 J- ~% ?) n' d4 |, h Map output records=20
9 h) m1 |( k S" r6 \& V Map output bytes=180
1 a& j' [- K1 r0 `6 U$ u) o Map output materialized bytes=280
0 K% J! [! g6 d9 ~ Input split bytes=1420
2 t& N$ i$ J( m) J2 D, { Combine input records=0# _4 e4 Q0 [! X4 V" Q: b- Y9 X# ^
Combine output records=0' | E/ _+ L* |6 E
Reduce input groups=2
, T: b {! K+ b5 }' ~+ x! N Reduce shuffle bytes=280
: X% ?3 P& w: K Reduce input records=20+ s' F% Z; v* {
Reduce output records=0! q8 {, g3 ^. D
Spilled Records=40
- P" G4 x7 K6 N# ]7 |" a Shuffled Maps =10+ q& K7 i# ?! B2 W# e: r" T3 U
Failed Shuffles=09 `2 q! q* j+ s R! Q
Merged Map outputs=10
) O: s# `% {$ A% ~ GC time elapsed (ms)=1915
7 w3 C/ e: ~% @- Y. B$ l CPU time spent (ms)=67909 s- d4 ?2 c j, G g! t
Physical memory (bytes) snapshot=38092103680 G2 s% F$ j+ E5 n0 }
Virtual memory (bytes) snapshot=311501414408 Q9 H# m/ M% A9 W
Total committed heap usage (bytes)=3503292416" V3 q, P; {2 m! B1 {% r; S
Peak Map Physical memory (bytes)=360734720& l O5 F/ R$ F1 p) y3 M
Peak Map Virtual memory (bytes)=2835021824' ]! c# b: z" q
Peak Reduce Physical memory (bytes)=279707648
8 g4 m( [$ h! V, |) \! L( Q$ X) h Peak Reduce Virtual memory (bytes)=2839097344
' `8 }; x7 g9 u8 U: [8 z$ z! Y Shuffle Errors
, J6 ~3 G* H( R# N" \2 p, W% a BAD_ID=06 }/ @9 x; }2 [, O ^2 Y) p
CONNECTION=0
' u$ J4 n4 _3 \! D o0 c IO_ERROR=0
- B# d. M( e. G4 g4 N4 V! J+ l( k WRONG_LENGTH=06 b/ u8 u% g* C `7 c5 W
WRONG_MAP=0& y& z7 W. H3 B$ G2 h6 h' |
WRONG_REDUCE=0: p4 Q# M8 T! N9 a" m
File Input Format Counters
( C0 I$ x$ G. H& V8 Y! ^ Bytes Read=1180
; W7 v: D4 v* x. {; u File Output Format Counters
6 N7 x3 i* p4 m- N0 Q7 U9 O6 i Bytes Written=972 @' R% A1 b2 l: V- t
Job Finished in 24.227 seconds7 e" _; X. L& u; D5 E% {
Estimated value of Pi is 3.14800000000000000000( s* _( D% X8 U* D4 ]* J" I
/ A3 t7 N0 E/ t9 ~2 F
问题解决' v7 L% h7 Q: M/ [7 }
5 j0 F# }6 [8 z* |6 |* N! H* E& U5 B
|
|
发表于 2022-11-16 17:00:20