|
|
[admin@hadoop-1 hadoop-3.2.2]$ hadoop jar /data/module/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar pi 10 100
5 J9 [3 r& d# G/ O$ p0 rNumber of Maps = 106 s$ x, ]5 ~% L$ v" \0 c
Samples per Map = 100% t- }, k) ?$ H- W% g. S1 S- J
Wrote input for Map #0
# B l: s* P5 p* x3 C, uWrote input for Map #1/ s9 t8 f4 J9 o5 z( q. p$ S
Wrote input for Map #2
) c7 ?4 m; g( r% y8 AWrote input for Map #35 A# a' w5 g( E4 R
Wrote input for Map #4. @ S# b' \" q
Wrote input for Map #5! ^, G- q; ?: r0 x
Wrote input for Map #6
; d, Y* H A3 \9 H, O& Z7 k. e9 C) yWrote input for Map #7
5 |& D h- g$ SWrote input for Map #8
3 A" ]# E8 I8 O; }- n! kWrote input for Map #9( z, P. C( e! }" _/ A! i9 n5 J
Starting Job, L( v F$ N% m. G) h% l9 ]5 R9 K
2022-11-16 15:32:34,946 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
8 j5 r2 H& ^* o( j5 h2022-11-16 15:32:35,174 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/admin/.staging/job_1668583281306_0001
o2 ~( L* ^$ N% x2022-11-16 15:32:35,389 INFO input.FileInputFormat: Total input files to process : 10$ Q; b2 w3 k2 @
2022-11-16 15:32:35,592 INFO mapreduce.JobSubmitter: number of splits:10
. U( v& V! Q& B0 F2022-11-16 15:32:35,806 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1668583281306_0001
7 \& P/ P3 r# b$ o5 ~. Z2022-11-16 15:32:35,808 INFO mapreduce.JobSubmitter: Executing with tokens: []# z* m; Y6 [( G) I1 [4 X
2022-11-16 15:32:36,044 INFO conf.Configuration: resource-types.xml not found: ^5 r% \+ o& V
2022-11-16 15:32:36,044 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.* i3 g5 |; S) V
2022-11-16 15:32:36,423 INFO impl.YarnClientImpl: Submitted application application_1668583281306_0001
" @) Y) x' S& K1 L d0 ]( v9 e a2022-11-16 15:32:36,504 INFO mapreduce.Job: The url to track the job: http://hadoop-2:8088/proxy/application_1668583281306_0001/6 v/ e& j0 D' q- d0 ~9 Q* [
2022-11-16 15:32:36,505 INFO mapreduce.Job: Running job: job_1668583281306_0001
1 L) V, d `6 \, h0 f: e3 i2022-11-16 15:32:41,561 INFO mapreduce.Job: Job job_1668583281306_0001 running in uber mode : false
5 {' x7 t$ ]. ~2022-11-16 15:32:41,563 INFO mapreduce.Job: map 0% reduce 0%' V, @3 I: R+ T0 E
2022-11-16 15:32:41,583 INFO mapreduce.Job: Job job_1668583281306_0001 failed with state FAILED due to: Application application_1668583281306_0001 failed 2 times due to AM Container for appattempt_1668583281306_0001_000002 exited with exitCode: 1
" K5 B9 K7 B+ Y' V# T0 N e# hFailing this attempt.Diagnostics: [2022-11-16 15:32:40.765]Exception from container-launch.7 V. x. n7 _7 a; d2 h0 b
Container id: container_1668583281306_0001_02_000001( s, t1 m7 W, m, i m V, d
Exit code: 1
% H. [3 O6 P. x- U A# `7 A. l7 I& N' d* x# C2 S
[2022-11-16 15:32:40.799]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
7 `7 T/ `! G- i. tLast 4096 bytes of prelaunch.err :
7 I2 y& ?1 h& k- h: hLast 4096 bytes of stderr :- `! X3 I- A2 l0 U1 I `
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster! A/ v. |5 r# s* ~
. x! Q' v3 E7 o" `9 D0 _Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
# u6 t \2 m' C* X<property>; o/ C, F c) q2 D. m# J3 s. s
<name>yarn.app.mapreduce.am.env</name>
/ Q" t) Q- R6 k& }/ ?$ k( Q <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
. L, q3 K% b7 S8 o</property>. y* I8 w6 S+ R. H: l3 Z8 ?
<property>7 q& ]4 s" t7 @
<name>mapreduce.map.env</name>
# \- I, q8 T, q! O$ O! c <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>! Y4 ]2 V1 A2 r, v3 D1 q' y# E, C
</property>+ h# W6 @( E# N3 [1 R5 m* z" E1 j
<property>" t; V% P( ~4 T, k) V5 E5 O
<name>mapreduce.reduce.env</name>- {0 Y7 s; h# D0 Z7 ]- ]+ s2 v
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>* q: Z& _& g* G+ x2 Y
</property>1 f* @5 D+ Y. \' N
8 }) F$ K: E" o, |' H; e7 W6 Q[2022-11-16 15:32:40.800]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
# D/ J# i: J5 l3 i2 F" }Last 4096 bytes of prelaunch.err :* \( X5 P9 M; N( ~+ f
Last 4096 bytes of stderr :
/ C5 o6 b/ Z" c; K$ L6 l- ] _) ]Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
: i5 F$ J& e0 U0 l( D j5 Z" k6 A! L) @
Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
0 @4 R& E8 D1 g2 |<property>
8 [1 }8 V9 i4 c+ n' B <name>yarn.app.mapreduce.am.env</name>
3 @- ^% |5 d D+ S/ l+ Z <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>8 d+ x- v0 p( M" z4 L
</property>
# c/ P3 o# d& L7 A& Z<property>8 f' M# R- F3 h0 q$ m+ ^
<name>mapreduce.map.env</name>1 _0 a' k5 K' l, R; |9 A
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>5 t6 ]% B5 }0 Y) l- @+ J
</property>7 P2 {; o4 E F1 p& j
<property># X1 p6 C5 V k2 G
<name>mapreduce.reduce.env</name>
3 E/ T1 r! [$ w <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>/ a. I9 Z' S: N3 e- Y! u
</property>
1 N7 ^2 N6 I# A' M+ M' E, ~0 L [5 V
For more detailed output, check the application tracking page: http://hadoop-2:8088/cluster/app/application_1668583281306_0001 Then click on links to logs of each attempt.
6 i; F& G, t" B0 P L. Failing the application. d; e4 O7 N0 E
2022-11-16 15:32:41,608 INFO mapreduce.Job: Counters: 0
1 ]4 M3 t/ e! z0 D" TJob job_1668583281306_0001 failed!
% D' A5 W$ j0 u3 m4 ^3 r
, S [. i+ ^5 S分析失败，解决方法：
6 C$ b' \: `' p: t. F
$ T5 m a) Q; P9 e% X: U- w[admin@hadoop-1 hadoop-3.2.2]$ vim etc/hadoop/mapred-site.xml( ~3 u! Y8 [' L$ d1 g2 t1 b
添加如下内容：
9 e. `+ @. C& V- s6 k; x$ n' }' N( k1 o
<property>; K# a. G m% ^7 |9 @& k! V" O
<name>yarn.app.mapreduce.am.env</name>
/ V- c$ L- O+ X7 {5 n/ h0 B: m6 c* D9 t <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>- g: {, R$ b4 l' m7 P4 X! x3 |
</property>, B. N% U- {+ Q1 U
<property>
/ t g0 {# D4 b3 Z6 W+ o <name>mapreduce.map.env</name>
* J: y1 X/ Z, h' f% O <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>3 ]" K! Z, y6 D9 H/ q
</property>
* c7 s. a8 j) I* D$ \<property>
3 ]: X% C6 [2 } <name>mapreduce.reduce.env</name>
+ x- R, l+ ]4 T. ?- v9 Q; d <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>( t, |- {( Q3 P& Y, E
</property>' P1 A! P8 _0 [ u6 S2 v
6 O& D: }7 f* N' f- j; Y, |$ A4 y" G2 j
同步到其他节点：$ i E3 S4 V- V& s# \. h, y" [' M
( A0 |8 `( c. P& v" @2 Q) d, Z0 x
L& w' X' p7 V" I' @2 H$ \; x( e2 C+ n! a e- n
% S6 G$ m) P! m+ w
[admin@hadoop-1 hadoop-3.2.2]$ xsync etc/hadoop/. t; z! H/ r- ]6 M, A
==============hadoop-1==================
8 }' M1 @- j# N0 O4 B- E3 g* }sending incremental file list& y7 a+ p4 Q9 J: v
: I% Z: B5 p: `& _ esent 1,004 bytes received 18 bytes 2,044.00 bytes/sec
- O! p1 Y4 E1 ]5 u9 Z1 }1 }total size is 115,250 speedup is 112.77. ^& B+ u5 q; V+ b; w8 g# H
==============hadoop-2==================
" k8 L5 Q% N" w3 q/ wsending incremental file list
+ O# k, x% ~4 y6 Whadoop/
) [: q" `) p3 |! u' t4 Bhadoop/mapred-site.xml# [7 X0 U# j; e) ^
: R9 a5 w' y$ Y1 ^) M
sent 1,251 bytes received 52 bytes 2,606.00 bytes/sec! d6 n6 Z& N! d+ x- }7 Z, ^& ]/ U
total size is 115,250 speedup is 88.45
, R& g3 [! O5 Q( t==============hadoop-3==================
[2 q C4 d! `, E. Nsending incremental file list: w, d* U* F: Q m$ }. t
hadoop/
9 X" U8 N$ c% y+ i |6 d- ]+ uhadoop/mapred-site.xml& @7 |. W% q7 `# V
+ d7 J5 a, C! S/ a! {9 Xsent 1,251 bytes received 52 bytes 2,606.00 bytes/sec& _& ^; y. ]4 ~+ W' ^: E6 y
total size is 115,250 speedup is 88.45& N/ G( O& m: p8 |
0 N) d, |% _2 ?& f
. w K' Z6 ?. z/ Q1 \1 h7 u4 ~, f. ^
再次验证：
/ T5 b6 W& `3 {) G0 f9 G' `+ L6 b0 }$ d- x/ I" i8 c5 A
[admin@hadoop-1 hadoop-3.2.2]$ hadoop jar /data/module/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar pi 10 1005 I+ I; r8 ?; l0 Z0 j! H+ w
Number of Maps = 102 E) y0 ^; y7 c& w. t8 i' T% y
Samples per Map = 100
. a1 _; C: g- r1 I! M5 JWrote input for Map #0
4 Y7 V- r( d! W _Wrote input for Map #15 k. z; p0 U7 F5 _+ Q4 M
Wrote input for Map #29 O5 T: ]+ U* E! v
Wrote input for Map #3; A, ^# e( [; Y9 n
Wrote input for Map #4
2 A, A2 ^1 N5 a! u* ]+ MWrote input for Map #5
0 t- g& D, U+ b2 r+ @Wrote input for Map #6$ Z" j( T" H2 T2 i( l! G( |
Wrote input for Map #7
0 b4 o0 @1 t% R# c: r( dWrote input for Map #8& b* v+ H+ {# c) ^( F1 D& U9 i
Wrote input for Map #9
" A+ \" g: t p. p- n. ^Starting Job8 X0 X6 f1 w3 Y" k# u$ n
2022-11-16 15:36:05,630 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
. [' K8 p: O! |' }" q& U, @& K/ i2022-11-16 15:36:05,878 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/admin/.staging/job_1668583281306_0002
& y( L2 ~1 ? G7 o2022-11-16 15:36:06,065 INFO input.FileInputFormat: Total input files to process : 10
2 y# q/ o* B) r9 x" c( T$ N2022-11-16 15:36:06,214 INFO mapreduce.JobSubmitter: number of splits:10 Q* V9 d. S5 }
2022-11-16 15:36:06,444 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1668583281306_0002& b* h- o' q" Z2 s$ T. a& u
2022-11-16 15:36:06,446 INFO mapreduce.JobSubmitter: Executing with tokens: []) J* \; u4 O* F9 M4 I. c( _
2022-11-16 15:36:06,654 INFO conf.Configuration: resource-types.xml not found6 G' D- ~: L! i; h8 z
2022-11-16 15:36:06,654 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.; J& w) s t8 i3 d8 W
2022-11-16 15:36:06,928 INFO impl.YarnClientImpl: Submitted application application_1668583281306_0002: U. e/ f- t( w% d- U* ]
2022-11-16 15:36:06,981 INFO mapreduce.Job: The url to track the job: http://hadoop-2:8088/proxy/application_1668583281306_0002/$ c [9 w' f4 S( r4 V" a: K4 i) p' n
2022-11-16 15:36:06,982 INFO mapreduce.Job: Running job: job_1668583281306_0002
2 g! x) Y# ^9 v- |* j/ H8 @2022-11-16 15:36:17,134 INFO mapreduce.Job: Job job_1668583281306_0002 running in uber mode : false# r9 y% h, i6 Q, ]3 u
2022-11-16 15:36:17,136 INFO mapreduce.Job: map 0% reduce 0%/ ?1 _" I2 [+ m: S' e
2022-11-16 15:36:23,297 INFO mapreduce.Job: map 40% reduce 0%
; z1 v% D5 A: F7 G8 J0 [$ w2022-11-16 15:36:24,308 INFO mapreduce.Job: map 100% reduce 0%
2 D, ]! K" `7 P X2022-11-16 15:36:28,342 INFO mapreduce.Job: map 100% reduce 100%/ B( a' [6 A2 c
2022-11-16 15:36:29,360 INFO mapreduce.Job: Job job_1668583281306_0002 completed successfully. X$ U: Q0 w! ~' U3 y, l/ o
2022-11-16 15:36:29,504 INFO mapreduce.Job: Counters: 54
. Z) Z& e1 K7 ?1 J6 B6 V File System Counters
( d: G& ^( f8 N" x FILE: Number of bytes read=226: L% w' e1 ^$ z2 o# N* ?
FILE: Number of bytes written=26300345 N4 [" ?$ ?2 e3 L# n
FILE: Number of read operations=0# w) d0 R% m5 K9 W3 Q
FILE: Number of large read operations=0
7 l0 A1 G& s, [) g, c) y' H FILE: Number of write operations=0
9 C# l& z1 }0 f& `: A7 Q9 M1 M' a) D d HDFS: Number of bytes read=2600$ w Z$ M7 T$ I: j
HDFS: Number of bytes written=215
5 @& _3 z6 t3 {- Y' T0 w6 D+ N6 Q HDFS: Number of read operations=45
& e, y" u2 O$ z. M HDFS: Number of large read operations=0( b4 Q X: |! R9 }3 r% l# v# q
HDFS: Number of write operations=3
: H1 ?/ _5 t% M* m HDFS: Number of bytes read erasure-coded=0
% a9 B, [1 c. o; m. u7 V# C Job Counters
% Q" r/ {' \2 H3 H& l: ^3 `: o Launched map tasks=10
5 l# k7 u5 _( W0 _. b/ K' } Launched reduce tasks=1+ M: h; O- F/ b& y- c) Q+ Y; l
Data-local map tasks=10
1 K! i8 k2 I, E7 I Total time spent by all maps in occupied slots (ms)=413940 s7 o& b$ f( V4 ^ O8 ?! Q
Total time spent by all reduces in occupied slots (ms)=3116. b& f A2 p h! T: ~) M
Total time spent by all map tasks (ms)=41394& M) ~# S9 @0 q$ t( p
Total time spent by all reduce tasks (ms)=3116
. g( C9 ]3 C8 a# |* P& I Total vcore-milliseconds taken by all map tasks=41394
+ W1 f7 _, o& f+ ]& p# C Total vcore-milliseconds taken by all reduce tasks=3116& d' x+ [. x' C
Total megabyte-milliseconds taken by all map tasks=42387456
6 t' q4 f5 q7 m( s( F) E Total megabyte-milliseconds taken by all reduce tasks=31907841 B g \3 ^% H! R) x5 ~/ Z. s2 ]
Map-Reduce Framework* a5 h1 f* t6 O4 [; p
Map input records=10
0 Z/ |" y3 \. N* w' g1 s1 `0 K Map output records=20# z: g, A h* b" R4 Q
Map output bytes=180( z3 c5 T4 [ n/ A1 W
Map output materialized bytes=280% Y9 W/ Z, B$ X' Q
Input split bytes=1420# x& ^: a; u& Q4 ^' Z
Combine input records=0 O* q: j1 U) U& g B5 y; V
Combine output records=0
, D# t; `. h2 h3 i2 Y c Reduce input groups=2
; h8 w' K2 i+ a F Reduce shuffle bytes=280! k" v1 r S; ]0 m4 Z8 i/ y. O! U
Reduce input records=20
0 H' m% e+ E# u, g) @& m, y Reduce output records=0
* g4 ^8 p+ b- S/ I4 I% L# k l* M; r Spilled Records=401 b6 _ _8 q3 K5 Z; }3 ~
Shuffled Maps =10% M2 q0 h; H7 p2 o- E
Failed Shuffles=0& p" S5 k; j: A4 q$ l! A: j
Merged Map outputs=10
{% H9 [* L9 ~& R5 n GC time elapsed (ms)=1915
# e- ^; K5 y" d" \; l1 T CPU time spent (ms)=6790/ I8 W( K1 F( g' {& z& t4 `+ ~# h
Physical memory (bytes) snapshot=3809210368
z5 A- z& x6 y4 ]: [/ L Virtual memory (bytes) snapshot=31150141440" T3 n4 _ L* O# t
Total committed heap usage (bytes)=35032924161 O4 L( c, N0 u: b4 [
Peak Map Physical memory (bytes)=3607347207 G) i. V; z% A- M
Peak Map Virtual memory (bytes)=2835021824
& I9 |, q m* Z/ [% A Peak Reduce Physical memory (bytes)=279707648& f8 c* p0 E0 x
Peak Reduce Virtual memory (bytes)=2839097344% J. i! v/ Z& s! m/ ], M9 t
Shuffle Errors( e7 A) r6 l0 o4 i
BAD_ID=0
! ?& o, x, Q# H0 [$ r, J CONNECTION=0
: B4 h6 }% F4 F6 I; L* V- e9 K IO_ERROR=0
/ V1 Z8 [: p5 e: l WRONG_LENGTH=0
8 J& b$ q" P3 r5 l9 j, L9 Y WRONG_MAP=0 C0 l( y4 D3 X2 Y
WRONG_REDUCE=0
M* U" H% `9 i6 [4 A# B File Input Format Counters / e6 x( j7 ]( d
Bytes Read=1180
/ ?# x( }; |$ P3 Y. e# e File Output Format Counters
2 ]0 {& @4 j/ _! I Bytes Written=97
- X2 E: y$ `6 P) A+ V# _! \3 ^Job Finished in 24.227 seconds6 }7 t! b$ J2 R$ j' n, ~# @
Estimated value of Pi is 3.148000000000000000005 S! ?) @, w2 H E* G! D
( D& ~# ?) m1 g! K9 X) `
问题解决
! h/ E6 ^3 b7 M, s4 j
0 J, e w9 y3 `* r' f. A |
|
发表于 2022-11-16 17:00:20