- 积分
- 12865
在线时间 小时
最后登录1970-1-1
|
马上注册,结交更多好友,享用更多功能,让你轻松玩转社区。
您需要 登录 才可以下载或查看,没有帐号?开始注册
x
[admin@hadoop-1 hadoop-3.2.2]$ hadoop jar /data/module/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar pi 10 100+ e' S, ]5 \ @8 Y- h) T+ }( m1 e
Number of Maps = 108 z) a1 {: s) r: K. A
Samples per Map = 100$ e5 k+ v( O4 m- S G
Wrote input for Map #0# x+ B8 f4 d4 [& o0 s; {0 \7 u0 e; S
Wrote input for Map #1( O+ w3 i1 q* R' L9 r/ p7 ~' F
Wrote input for Map #2
6 G. s. {3 K$ b2 w- m) oWrote input for Map #3; S3 c% \2 |1 C9 Q. R8 ^
Wrote input for Map #42 U+ x+ B* h+ A! F4 B1 k" l
Wrote input for Map #5' k) S. d x+ A, A
Wrote input for Map #6
. u; _, I' I: E j) R* rWrote input for Map #7
! L: \- q: B9 w7 O* `8 k' ~% g' hWrote input for Map #8" ~* D% n5 a" X* N4 H5 I0 q
Wrote input for Map #9! o7 [( b- q; K! o5 k5 i) k
Starting Job
/ O9 \( v3 m$ K8 h4 y" Z! g: f2022-11-16 15:32:34,946 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
# b! X0 [/ R2 `. u+ R4 A( C2022-11-16 15:32:35,174 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/admin/.staging/job_1668583281306_0001* m9 l) j& Q/ h z
2022-11-16 15:32:35,389 INFO input.FileInputFormat: Total input files to process : 10
9 u. { @. F2 r8 [0 G& d4 [ N9 _2022-11-16 15:32:35,592 INFO mapreduce.JobSubmitter: number of splits:10
& L1 L& [4 z4 U* V+ k2022-11-16 15:32:35,806 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1668583281306_0001/ V+ o! l1 _! t1 }) f* N
2022-11-16 15:32:35,808 INFO mapreduce.JobSubmitter: Executing with tokens: [] e9 d7 {5 o1 P# a/ V9 ^" L" |* E
2022-11-16 15:32:36,044 INFO conf.Configuration: resource-types.xml not found
( J% i" ^" g! ]; j2022-11-16 15:32:36,044 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.1 R# [* L2 x8 A |9 N
2022-11-16 15:32:36,423 INFO impl.YarnClientImpl: Submitted application application_1668583281306_0001. F% x: k# l3 H" b5 u
2022-11-16 15:32:36,504 INFO mapreduce.Job: The url to track the job: http://hadoop-2:8088/proxy/application_1668583281306_0001/) A( ?2 G. d8 z' P% n: D/ T9 s
2022-11-16 15:32:36,505 INFO mapreduce.Job: Running job: job_1668583281306_0001+ [. k; l' k4 M# |* `- n
2022-11-16 15:32:41,561 INFO mapreduce.Job: Job job_1668583281306_0001 running in uber mode : false) L' _ b+ @1 d9 d9 e& h- B
2022-11-16 15:32:41,563 INFO mapreduce.Job: map 0% reduce 0%7 L$ Z, B& t$ m5 e2 O! D1 S
2022-11-16 15:32:41,583 INFO mapreduce.Job: Job job_1668583281306_0001 failed with state FAILED due to: Application application_1668583281306_0001 failed 2 times due to AM Container for appattempt_1668583281306_0001_000002 exited with exitCode: 1+ R* y. Z1 o& j: w* Z9 p* V6 q
Failing this attempt.Diagnostics: [2022-11-16 15:32:40.765]Exception from container-launch.
; ]3 L( ?- \' w# [/ u0 Q) y: RContainer id: container_1668583281306_0001_02_000001
; l, \* e& z, e- l) G I/ C% iExit code: 1
) z1 i. j1 K) m4 D6 I7 O
8 T' X# ]* z$ r% q' |% ?/ j: v[2022-11-16 15:32:40.799]Container exited with a non-zero exit code 1. Error file: prelaunch.err." x' u3 t1 }8 Z
Last 4096 bytes of prelaunch.err :
6 s- S* e% |5 ~2 @+ |Last 4096 bytes of stderr :
% C# g) o5 d: u( D1 T8 VError: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster& a4 t: e% ?8 _7 E0 Z9 t o, J
/ T9 @' |9 x, L/ G/ e3 y7 Q
Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:7 b( h8 `9 y: P9 _* A2 ~1 g9 ^
<property>
+ R$ Z7 F' T. w* o; D0 O, M <name>yarn.app.mapreduce.am.env</name>
3 i, r- L. \4 M2 H <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>. P. P& J3 u# ~$ Y" m; \7 F( q3 M
</property>
/ B B6 O& {' L( k<property>/ P' o1 f+ g& h z( G( s& a, A
<name>mapreduce.map.env</name>
3 U) b. ]* O9 ^& i- ^8 U <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
/ J9 F+ T; R- D6 \</property>
6 i& p; p/ P8 o4 c' g: R& }<property>- l# ~0 B8 T0 a# m- J% @7 O
<name>mapreduce.reduce.env</name># O2 C, n! J" C8 F1 l( B D' K
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
, z: y6 c" x1 e</property> N. T/ A1 s$ I: I% ^4 K9 ?+ Q
% q+ V% N- {& B! H0 Z3 E8 v% H[2022-11-16 15:32:40.800]Container exited with a non-zero exit code 1. Error file: prelaunch.err.5 m' L8 P N0 p. h) ^+ ~' f2 o
Last 4096 bytes of prelaunch.err :
: E! G9 U7 u& d% @Last 4096 bytes of stderr :
$ t1 Z& ?5 x7 v$ g" BError: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster7 g5 v; V3 l; B* B% R0 p: k& n( _
5 J% b' W% I q# d! P8 YPlease check whether your etc/hadoop/mapred-site.xml contains the below configuration:! G1 D! _+ P ?) q
<property>
$ ?$ |. p1 j$ G <name>yarn.app.mapreduce.am.env</name>
" R$ |, F5 ^3 C5 E H# T <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
" T8 q8 j) b* E% p+ e& t, y</property>" m# ?7 n+ T& A9 M/ l
<property>( q1 E7 X/ l6 b' `! \& `
<name>mapreduce.map.env</name>
4 [% f. X( c8 a; u# N6 E) j1 X/ k1 r: X <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>( G1 u: q/ a) j1 {( x# R
</property>
8 N0 D, _, F3 s, H! x5 n<property>
1 F9 h O% z+ k4 m: a1 G7 M" y <name>mapreduce.reduce.env</name>% I# P% ]* J# E" q2 Z2 n& L
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
9 l2 E1 E* t/ ?, L# P7 n$ a</property>
0 T" _8 a, K" e Z* b0 I6 ]$ ~
. }' Q) q2 H' P/ uFor more detailed output, check the application tracking page: http://hadoop-2:8088/cluster/app/application_1668583281306_0001 Then click on links to logs of each attempt.
0 g& K# Q+ p, V3 }% C/ V. Failing the application.
' v: g& d; M4 A; @2022-11-16 15:32:41,608 INFO mapreduce.Job: Counters: 0
+ \( M+ i0 _) G2 }$ s pJob job_1668583281306_0001 failed!
3 L P- C% I& B$ h5 p* e
/ D% Q, `* z3 X, M" l分析失败,解决方法:( Q, T; k) y2 S. f) v0 `2 X, x0 ?
( d: f2 X( Q% X- l' x: G. M! U# C[admin@hadoop-1 hadoop-3.2.2]$ vim etc/hadoop/mapred-site.xml
* A& l4 D- S# b4 g添加如下内容:" k) u0 y! {0 B7 t( {5 i% ]9 x+ M
3 ]' x! ~) ~9 G) W, P<property>$ L+ a" }5 @/ F T
<name>yarn.app.mapreduce.am.env</name>
* \3 F6 J" C2 v' }- }! c/ @ <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
s- Z! D, O, E' q</property>5 H" F: V4 d! W+ W" P* {* C
<property>
! c* M; C5 \, y <name>mapreduce.map.env</name>, J/ i3 x# v9 a' E) @' @0 }& _
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>! ^" Z! G- B- z/ @2 Z- C6 k
</property>' R% y3 h6 J w x- z; _
<property>! w) V O% {: z: `
<name>mapreduce.reduce.env</name>9 U/ g& D* s) L
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>; Q9 Q& p( I( D
</property>
% |( ]: } a5 B: ^. G
6 a3 L& B; \7 Y# ? F9 Z# s, O- i% f5 X
同步到其他节点:
, R' I+ N7 P+ C6 w8 } z S$ f
& Z( x+ u% Q3 X& Z( |* ~: Y* g2 R6 i$ X* h) x
8 G9 b) |6 i c5 k+ C
6 Z0 ]$ q" p! P$ A. h4 p[admin@hadoop-1 hadoop-3.2.2]$ xsync etc/hadoop/
3 E( u1 B1 w0 Y8 O==============hadoop-1==================0 n, x9 k, K4 k5 C4 v
sending incremental file list
. \0 f( k, C M" }( N6 b8 t- r7 b. b1 u& J, @3 z
sent 1,004 bytes received 18 bytes 2,044.00 bytes/sec/ d. m4 q+ L+ \6 j* _$ O0 ]
total size is 115,250 speedup is 112.77
" D) |( {( `) Y- g) F+ p' N==============hadoop-2==================
8 W/ e0 H) p+ o8 w: C- K7 A' ^sending incremental file list
1 [; \; ~" E" P% \9 nhadoop/' n5 f7 ?: l1 }2 \/ G$ d
hadoop/mapred-site.xml& l% @& Y6 Y# G+ t+ a
, }8 f0 P5 r" \3 W4 i: qsent 1,251 bytes received 52 bytes 2,606.00 bytes/sec
" f) ~" `1 l- Ntotal size is 115,250 speedup is 88.45% a* H8 t6 ]. P3 a9 t4 B
==============hadoop-3==================
6 ]/ ^5 v; H1 r6 J8 ksending incremental file list
% K: L. f. h2 Y! x; Thadoop/
, |2 c( X# M5 h# uhadoop/mapred-site.xml% T/ `9 Y3 P! K1 U, Z# K: X0 Y
, o4 Q* ~7 R1 Q& q
sent 1,251 bytes received 52 bytes 2,606.00 bytes/sec+ j: z8 l G, w0 `+ o
total size is 115,250 speedup is 88.45
: m. | M. s" @4 a8 o; T O9 \, V* Q; O
! _# m8 L7 }3 o5 O" o( e! P3 ?再次验证:3 O- N d* n# `( A
# x+ x9 r2 U) R: j. b[admin@hadoop-1 hadoop-3.2.2]$ hadoop jar /data/module/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar pi 10 100/ P2 b) A5 ~& l& }. x
Number of Maps = 10, I. ?- B4 M. j+ `' Y% ?6 f
Samples per Map = 100
6 U* n4 Y) j# zWrote input for Map #00 D' A% Z' ^: f* b. ?$ A
Wrote input for Map #1) d( H# B/ z9 G, q1 |
Wrote input for Map #2
' Q1 O- B3 X6 C" o. FWrote input for Map #3
; m8 q* `8 g. v$ U9 E! HWrote input for Map #4
* V G j' i1 v! S$ H0 pWrote input for Map #50 J5 O' P8 o: p
Wrote input for Map #6
6 S9 g2 ^, P# Y9 w5 ~9 MWrote input for Map #7# e/ _1 v4 O3 N5 U& t4 G4 Y
Wrote input for Map #8" @7 }5 a/ [1 H' z
Wrote input for Map #9
% O9 j* R% X8 E1 E0 q; ^( K- IStarting Job) ^- p+ e" ]$ L, w/ k: ?. ^* c' I
2022-11-16 15:36:05,630 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
: p3 u; @; Q, W) D: k! P2022-11-16 15:36:05,878 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/admin/.staging/job_1668583281306_0002
9 Z6 D( M5 m6 o6 i; a4 G" u4 V2022-11-16 15:36:06,065 INFO input.FileInputFormat: Total input files to process : 10" a N' Y+ B- ?# r" ~! i
2022-11-16 15:36:06,214 INFO mapreduce.JobSubmitter: number of splits:10& r0 X# O3 l* f& T1 R9 K4 S* i
2022-11-16 15:36:06,444 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1668583281306_0002" f) k( ]: `/ O, Y
2022-11-16 15:36:06,446 INFO mapreduce.JobSubmitter: Executing with tokens: []$ z' q) \/ M4 c3 }) L
2022-11-16 15:36:06,654 INFO conf.Configuration: resource-types.xml not found9 I$ V9 [- H& o. O/ f; T0 d" j6 I
2022-11-16 15:36:06,654 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.: N) p0 t( W# X& G
2022-11-16 15:36:06,928 INFO impl.YarnClientImpl: Submitted application application_1668583281306_00024 M/ h! U8 Z' ?& W
2022-11-16 15:36:06,981 INFO mapreduce.Job: The url to track the job: http://hadoop-2:8088/proxy/application_1668583281306_0002/2 k! k; r% m1 ?: H* T1 \
2022-11-16 15:36:06,982 INFO mapreduce.Job: Running job: job_1668583281306_0002
1 b: D% `2 h i. G1 c2022-11-16 15:36:17,134 INFO mapreduce.Job: Job job_1668583281306_0002 running in uber mode : false8 m: x# A. i0 ^8 D6 a, b# c0 F
2022-11-16 15:36:17,136 INFO mapreduce.Job: map 0% reduce 0%0 n8 s7 o2 C6 |" d% N5 U
2022-11-16 15:36:23,297 INFO mapreduce.Job: map 40% reduce 0%9 _$ Y/ j$ ~; }% o9 [
2022-11-16 15:36:24,308 INFO mapreduce.Job: map 100% reduce 0%
4 H2 s- R: V- P2022-11-16 15:36:28,342 INFO mapreduce.Job: map 100% reduce 100%
' z- |5 b% r" z' e$ x2022-11-16 15:36:29,360 INFO mapreduce.Job: Job job_1668583281306_0002 completed successfully
1 ^1 y5 z' ]3 J9 w0 H. Q4 S2022-11-16 15:36:29,504 INFO mapreduce.Job: Counters: 54
! S# l, u) W# x9 B! X- b; z File System Counters
. }) r6 s. J# ?8 E9 b FILE: Number of bytes read=226
. t5 V1 R6 Q; y# h2 o5 b FILE: Number of bytes written=2630034! E _6 Z* D$ ]
FILE: Number of read operations=01 a9 E( {2 b7 t
FILE: Number of large read operations=0% C& e+ B$ c% P) v" G
FILE: Number of write operations=07 W; O! i1 d& C4 I9 d! B$ a
HDFS: Number of bytes read=2600
6 F1 t: e/ T% \ N0 l5 C HDFS: Number of bytes written=2154 W% Y$ m2 k) j. T- U
HDFS: Number of read operations=452 G3 n, N' y( a5 V! p
HDFS: Number of large read operations=0& V' }% \ Q4 r7 D' `
HDFS: Number of write operations=3
% A% C& `; `+ \: Q Q" V% X HDFS: Number of bytes read erasure-coded=00 q' m8 T" O4 L
Job Counters 1 M3 w$ K0 h" i% g
Launched map tasks=104 r& Q/ q0 R' y! Y9 l+ r
Launched reduce tasks=1. L- D' S( n" h: R o+ D
Data-local map tasks=10. U, \; b; Q: E( z* `/ g
Total time spent by all maps in occupied slots (ms)=41394/ J/ c6 x* I" K$ Y8 E
Total time spent by all reduces in occupied slots (ms)=3116+ R8 e F' `6 g: l* }
Total time spent by all map tasks (ms)=413943 v9 p0 ]8 s6 C2 Q% Z) q* x
Total time spent by all reduce tasks (ms)=3116& W, ]$ Q4 O* H. E+ Q3 T
Total vcore-milliseconds taken by all map tasks=41394
$ W7 x. |3 j& r2 P4 ] Total vcore-milliseconds taken by all reduce tasks=31164 a9 B' V: z% W, B. b4 t$ ~
Total megabyte-milliseconds taken by all map tasks=42387456
! Q5 [1 v0 b1 m% X6 p7 Z u6 O2 S5 D( A Total megabyte-milliseconds taken by all reduce tasks=3190784! y1 {7 E* t$ g* s$ ~- S# v
Map-Reduce Framework
/ B6 Z$ A( j/ l* G) h- v Map input records=10
. d$ g3 F2 q3 z2 l Map output records=20
" ~2 U2 E# h3 \1 x' F; M/ @0 o Map output bytes=180
p8 d: y" W- y$ s+ W5 _; @4 ^ Map output materialized bytes=280
- O/ B6 h/ Z+ p Input split bytes=1420
9 U5 W# v% z2 m9 H$ r7 @( h Combine input records=0
" H I: D3 [- q Combine output records=0
$ X! `* u: [: @- \ Reduce input groups=24 D7 M; O8 Y Y( z) H
Reduce shuffle bytes=280
! S& O3 w/ o8 A. v% q0 f$ C Reduce input records=20
0 D+ q& H) `# U) H# A Reduce output records=04 @* m$ g0 Y& u5 v8 H
Spilled Records=40
K6 s8 b: U- e0 w% I Shuffled Maps =10
8 S+ B: d! z6 b Failed Shuffles=0' B2 H1 `( y5 b3 A0 e
Merged Map outputs=10
7 |! B1 U7 K$ V" {( \. T2 a7 y GC time elapsed (ms)=19156 n- ?+ E8 T* E& c) z# d
CPU time spent (ms)=6790* ?+ ?" l2 H; s8 N
Physical memory (bytes) snapshot=3809210368
& ]2 {+ D; f0 P% U9 s: E5 Y Virtual memory (bytes) snapshot=31150141440
1 @: G. E1 p. [7 B/ r: T) c Total committed heap usage (bytes)=3503292416+ r, w, _3 r: i1 S
Peak Map Physical memory (bytes)=360734720
% s9 @+ E* i5 e Peak Map Virtual memory (bytes)=28350218241 K9 S; u# _2 t. V
Peak Reduce Physical memory (bytes)=279707648
D& X9 H' Q9 N$ Y0 n Peak Reduce Virtual memory (bytes)=2839097344; x) P) P! i1 A% c
Shuffle Errors/ m6 [. R) ` Z6 U* i" K
BAD_ID=0
6 u- a! U1 V# l5 N CONNECTION=0' N) H$ x& J; O! t1 d
IO_ERROR=0) u" ]. d! ^6 K" C
WRONG_LENGTH=09 u' s6 D0 R. t7 R; b" \, r- s
WRONG_MAP=0
0 o* k' M. }: K, J) ] WRONG_REDUCE=0
^/ e" P6 f4 @ File Input Format Counters
5 Q5 X! S# D; _. f Bytes Read=1180. z3 \" I8 U5 L& W. S5 y$ ]5 j6 `
File Output Format Counters
$ P" n1 b$ W1 v R4 W Bytes Written=97% w2 E! ~! U$ A. V8 C. O
Job Finished in 24.227 seconds: \ f/ }$ l2 T% {3 M0 g
Estimated value of Pi is 3.14800000000000000000! p* l' G, v U! F. q5 u
! B: W/ G' v6 k! W
问题解决
# o: ?4 O; A" I$ n9 {" d: W/ {( A4 C2 ?" u
|
|