|
|
[admin@hadoop-1 hadoop-3.2.2]$ hadoop jar /data/module/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar pi 10 1007 I: `5 b2 k6 D" ~. ^7 _3 H( Z6 S
Number of Maps = 103 E0 W+ E& {& P- z8 n4 j, Y
Samples per Map = 100
, x4 Q# y7 p$ L- o) y' }2 Y4 R# iWrote input for Map #02 I, Y; _! Z/ |- e+ }$ O" ^; ^
Wrote input for Map #1
+ G; I; i6 b" r2 n/ i+ T4 K h7 aWrote input for Map #21 v& [* f3 k, T" Z) c' J* r
Wrote input for Map #3# g; x, P" k; K" I. J& i& G: `
Wrote input for Map #4
, P) P: a. _, d+ ]& O' LWrote input for Map #5
# t, E9 l7 r" ?+ _. `# xWrote input for Map #6
7 P. ?4 f3 x+ uWrote input for Map #77 H4 S B" S' J6 W
Wrote input for Map #8- k. J: ]) t" h, a
Wrote input for Map #9
! ]) r" D. ?% y! _& w5 J/ U. {Starting Job$ C" d9 V4 R: c: e1 b/ h
2022-11-16 15:32:34,946 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2# M: n- n4 s1 j: ~* L6 k2 x& n
2022-11-16 15:32:35,174 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/admin/.staging/job_1668583281306_0001. l8 V$ y5 [) P" F
2022-11-16 15:32:35,389 INFO input.FileInputFormat: Total input files to process : 10
0 I9 @# ]9 L: F7 [5 {; z# |: C2022-11-16 15:32:35,592 INFO mapreduce.JobSubmitter: number of splits:10
8 y# s5 B' A( M4 r& ~+ f2022-11-16 15:32:35,806 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1668583281306_0001
' J. |, T5 t$ |( x# P; X% {2 {7 l2022-11-16 15:32:35,808 INFO mapreduce.JobSubmitter: Executing with tokens: []! E1 y: q; v! c1 o/ c: R/ H
2022-11-16 15:32:36,044 INFO conf.Configuration: resource-types.xml not found) l- {7 E' k; J! ^; J
2022-11-16 15:32:36,044 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
) a: u3 u% v% V/ n" M- ^( S2022-11-16 15:32:36,423 INFO impl.YarnClientImpl: Submitted application application_1668583281306_00017 m2 f6 A, x2 ] |- K9 x5 T. X
2022-11-16 15:32:36,504 INFO mapreduce.Job: The url to track the job: http://hadoop-2:8088/proxy/application_1668583281306_0001/
* p5 V4 R y3 {/ r9 F* d( L7 v: c2022-11-16 15:32:36,505 INFO mapreduce.Job: Running job: job_1668583281306_0001# b9 i" j1 y) \! Q5 q
2022-11-16 15:32:41,561 INFO mapreduce.Job: Job job_1668583281306_0001 running in uber mode : false% f: v/ [8 Z' Y: Z9 U
2022-11-16 15:32:41,563 INFO mapreduce.Job: map 0% reduce 0%& {7 a7 o; ?8 \ k- q# O2 F
2022-11-16 15:32:41,583 INFO mapreduce.Job: Job job_1668583281306_0001 failed with state FAILED due to: Application application_1668583281306_0001 failed 2 times due to AM Container for appattempt_1668583281306_0001_000002 exited with exitCode: 1
( K, Y5 P. O+ Z: SFailing this attempt.Diagnostics: [2022-11-16 15:32:40.765]Exception from container-launch.: M8 j' Z2 A/ J1 ~2 k) c
Container id: container_1668583281306_0001_02_0000018 _& n6 n ]4 ~% L
Exit code: 15 P5 Q3 P( r- k: M" L8 m
! y# M3 y4 x* N5 G/ t. i[2022-11-16 15:32:40.799]Container exited with a non-zero exit code 1. Error file: prelaunch.err.8 |3 t; }& ]/ ~$ {0 Q9 H) J: ]
Last 4096 bytes of prelaunch.err :
! L3 }% _# W: s2 x2 v$ vLast 4096 bytes of stderr :( x) c% a6 J& }1 H5 o; G
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster' ]5 I5 r: _( ]( w. s
6 T4 S! Z& F1 ~/ Q7 N' M# L
Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
& @ M3 ^; A3 `6 r" G5 L+ n<property>$ C1 |# a1 `% d' |" Z9 m
<name>yarn.app.mapreduce.am.env</name>% x1 \* w: ^1 ]/ p+ ^. r6 @
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
( g6 ^" B& r2 Q9 ?</property>
4 y6 g; d1 _% V/ I; O1 J<property>
7 D. W9 g2 e( B, S! v <name>mapreduce.map.env</name>
7 g4 R- G8 S; x7 z+ K <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>- y( L" c+ S9 F/ a! S' E; C1 j8 }
</property>3 P" b- e# M1 t5 Z
<property>3 D5 d. D& p* u+ ~
<name>mapreduce.reduce.env</name>
& T: j+ R0 k4 r5 C2 J1 z; h/ o <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
- L; X7 W9 J, O, ]1 i9 e</property>
: S' K8 {3 R- Y% S7 W- z" _8 \8 h2 P2 h y5 ~/ Q
[2022-11-16 15:32:40.800]Container exited with a non-zero exit code 1. Error file: prelaunch.err.2 @. x" t C/ {5 k
Last 4096 bytes of prelaunch.err :
. W. m" ?0 v! D, a6 K# OLast 4096 bytes of stderr :' {6 y$ W5 p8 i, A9 N
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
: l3 P+ d8 _ Y- d& Q( Y& p a1 u) e/ a! F
Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:$ I8 l, I9 a$ V7 O6 W
<property>
# u4 p$ \- K9 u <name>yarn.app.mapreduce.am.env</name>) g5 e# Y- T6 L0 O
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>$ N7 w! I# u# v- f! p/ ^2 T0 R! ?
</property>% F' I. h7 t6 h$ `! y
<property>
3 O) X. E/ T$ H) S: d8 H <name>mapreduce.map.env</name>1 G6 W% j- o5 W% G
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
! Q4 b" f* A* @ M</property>
$ S5 u$ Z) ~- h/ L<property>/ |- P7 D) l9 N; F% A/ P- o$ M
<name>mapreduce.reduce.env</name>) C% w2 W* O1 h! {, k) t
<value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>! j: e8 x& o) ?& T- H* ~; @
</property>
& `* s. M0 a$ q5 U! e6 z0 J; U
+ y) I/ F& }' C2 E6 x; bFor more detailed output, check the application tracking page: http://hadoop-2:8088/cluster/app/application_1668583281306_0001 Then click on links to logs of each attempt." [, C m" g4 {& a: K; q, h3 I+ z
. Failing the application.
& `9 O1 ^: ~- |! O2022-11-16 15:32:41,608 INFO mapreduce.Job: Counters: 0
0 p7 ^) o2 v! y6 ?( \1 c9 gJob job_1668583281306_0001 failed! T4 m n. t( Z: Q" A
# L$ Q9 L6 N9 o; _+ I7 C: k分析失败，解决方法：7 y) C: \- J% t& {8 \
0 H! n8 f" C, p[admin@hadoop-1 hadoop-3.2.2]$ vim etc/hadoop/mapred-site.xml% b* ]) e" {3 p
添加如下内容：. T, \. @) m8 ?, H
* |, k2 k$ v" u5 E6 O5 ~7 C0 R
<property>$ [4 T- {+ o) F% i+ j) m5 u& _
<name>yarn.app.mapreduce.am.env</name>
$ L- E \2 p" t4 G <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>5 L5 I. ^% s) m
</property>1 w4 C$ k+ d9 K; c
<property>- q1 J5 ^& H4 X$ {1 |4 t
<name>mapreduce.map.env</name>7 d$ b! J- c x
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>( M9 N$ f4 g4 V2 q1 r& {. C
</property>
" A6 a L! |$ D<property>
$ } o& ]4 J1 w* I& n! h <name>mapreduce.reduce.env</name>
$ |4 D- c4 j2 C+ U5 o* r ^( m6 Z# e <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>* l0 y5 F. g( Q6 v3 N4 f. k3 U
</property>8 x! ^; J e- N
2 {# m- n- d7 I, H' r |! m; w
3 \/ p; x$ [' \" U( K
同步到其他节点：
J7 K/ x) D0 z2 I: a( Z1 x8 x: e, g, i: s/ R6 V* X/ l- [
) o& f) r" d9 |+ S. D
4 x1 B0 [- J7 ^8 H7 n
) B8 `4 S+ C- X) U, ^[admin@hadoop-1 hadoop-3.2.2]$ xsync etc/hadoop/: k7 [6 u( z7 M* c( g s) [
==============hadoop-1==================
$ u0 ]# N: Q, h! O/ Esending incremental file list
5 }+ i3 Y* ?4 T; V4 l; `9 W$ n Q
* s& U0 h- h. Q# q3 S0 Esent 1,004 bytes received 18 bytes 2,044.00 bytes/sec
- u! o. N$ }/ C. Wtotal size is 115,250 speedup is 112.77
5 Q: C/ c K: _5 g==============hadoop-2==================) v* [# p* D/ |& W" t
sending incremental file list
?1 ?" V+ z# s* \5 dhadoop/
/ R! D( A( e! Ihadoop/mapred-site.xml
# z$ y" \4 i/ i$ u# ]1 ^
. b z. n# e2 h/ G3 \: c6 d/ Nsent 1,251 bytes received 52 bytes 2,606.00 bytes/sec
4 X O6 [; t) \+ ], mtotal size is 115,250 speedup is 88.45- | }' a7 r: k# }$ @
==============hadoop-3==================
' e2 \. Q) @: h. |4 vsending incremental file list$ n5 h# E: v) t- _2 L$ {8 o- c/ r6 |$ O
hadoop/, r3 Y; H6 \# o( |; C9 t. h
hadoop/mapred-site.xml1 u# V( c6 I; S6 ^8 d
6 R0 f3 t; |! @, n! q
sent 1,251 bytes received 52 bytes 2,606.00 bytes/sec
( I1 N' B1 y$ M1 ]total size is 115,250 speedup is 88.45. [ O5 o+ D: f1 C$ B) P0 F. E
% p% c3 L: m" X# A8 T) G3 M5 X/ F* T
再次验证：
0 i, @4 H0 N! e3 H% a, \% F4 F- k( }" F
[admin@hadoop-1 hadoop-3.2.2]$ hadoop jar /data/module/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar pi 10 100
* P! r. `, U2 u+ @Number of Maps = 10; l9 ?5 t. |( T% u% S- q; ~* x
Samples per Map = 100
1 P, a5 Q0 ]3 U1 lWrote input for Map #0
! Y. L8 V" P9 W; L AWrote input for Map #1' R; n# M4 M a: P% P$ a
Wrote input for Map #2& R0 x Q4 l0 O* X
Wrote input for Map #3
. |/ b7 T# H1 h2 E+ r. J8 MWrote input for Map #4
0 W6 J) l. v0 yWrote input for Map #59 G6 [: u' K3 g6 a; C7 y( P1 V+ m
Wrote input for Map #6; @ s+ q) s8 r- X
Wrote input for Map #7
) f" G+ _* S# Z: ^1 rWrote input for Map #8
- v% p2 z, A, B1 P: W9 I, cWrote input for Map #9* S) z6 R" P$ U1 w$ U& A* |
Starting Job
' w3 }: s3 S1 F2 X. O2 Z9 K1 L2022-11-16 15:36:05,630 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
2 q% F7 Z; V( @% s" F7 T2022-11-16 15:36:05,878 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/admin/.staging/job_1668583281306_00022 p a8 w6 \# {2 z
2022-11-16 15:36:06,065 INFO input.FileInputFormat: Total input files to process : 10
8 M/ r5 r2 U5 F( P6 C% x2022-11-16 15:36:06,214 INFO mapreduce.JobSubmitter: number of splits:10* M) L. h, ^( ~+ H7 q2 O" e
2022-11-16 15:36:06,444 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1668583281306_0002; W! U" E- w2 ?7 j3 s
2022-11-16 15:36:06,446 INFO mapreduce.JobSubmitter: Executing with tokens: []2 E" }+ R- s" G2 u
2022-11-16 15:36:06,654 INFO conf.Configuration: resource-types.xml not found2 o0 W' }0 g& C9 x
2022-11-16 15:36:06,654 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.$ A9 B% k' S% S& q
2022-11-16 15:36:06,928 INFO impl.YarnClientImpl: Submitted application application_1668583281306_00020 a. ^1 o8 B; b/ t+ y
2022-11-16 15:36:06,981 INFO mapreduce.Job: The url to track the job: http://hadoop-2:8088/proxy/application_1668583281306_0002/
+ t5 h, K9 ]5 Y/ P4 \% c l' \4 E2022-11-16 15:36:06,982 INFO mapreduce.Job: Running job: job_1668583281306_0002* q! {4 \' ~& m c6 K4 S9 Q) M" R
2022-11-16 15:36:17,134 INFO mapreduce.Job: Job job_1668583281306_0002 running in uber mode : false
% g% L" `& L3 Z; V7 M- Z, v2022-11-16 15:36:17,136 INFO mapreduce.Job: map 0% reduce 0%
# X9 z5 E3 q' ]$ X2022-11-16 15:36:23,297 INFO mapreduce.Job: map 40% reduce 0%0 ]% y; w8 i; |6 U! N# s) V
2022-11-16 15:36:24,308 INFO mapreduce.Job: map 100% reduce 0%; D1 F! `# ]; d0 P
2022-11-16 15:36:28,342 INFO mapreduce.Job: map 100% reduce 100%7 h* K f c6 D' h! H
2022-11-16 15:36:29,360 INFO mapreduce.Job: Job job_1668583281306_0002 completed successfully
+ P# M* I% k" ^4 a9 h: r. J2022-11-16 15:36:29,504 INFO mapreduce.Job: Counters: 54- B& I$ A5 P& K4 v6 Y, c
File System Counters# X2 Z/ M5 J4 w& A5 H! B
FILE: Number of bytes read=2261 q" w/ i' g& u, Z5 s# c
FILE: Number of bytes written=2630034
& Z. ?0 j, y# o, b( h* a FILE: Number of read operations=0
9 G2 A: x! J; n( u6 J FILE: Number of large read operations=0
0 u Q, b/ h2 d; w- K* _+ I. W FILE: Number of write operations=01 y; y; K( F' n: H0 t. T
HDFS: Number of bytes read=2600
, K8 c; e. `! r% J l HDFS: Number of bytes written=215 I( H! d1 j. E+ y
HDFS: Number of read operations=45& G0 B5 K: j3 S3 T( L
HDFS: Number of large read operations=0
# N |/ D3 o% K* r3 }# Y HDFS: Number of write operations=3
4 ^2 [/ Z4 S$ p: [2 B HDFS: Number of bytes read erasure-coded=01 x; R5 A, }$ V9 O) B3 g
Job Counters # X, f2 ^$ g$ F! _) `0 I! u, O
Launched map tasks=10
! v" O+ E6 w2 [ Launched reduce tasks=1
; |# u! r( [9 ]- U* r) I+ x5 i Data-local map tasks=10
7 a9 R6 q y6 f Total time spent by all maps in occupied slots (ms)=413949 {4 l# g% t% ^9 i3 W5 t* x8 e C. |% r
Total time spent by all reduces in occupied slots (ms)=3116. P% ^- i' D/ Z0 u8 s
Total time spent by all map tasks (ms)=41394! z# p1 W M w" x
Total time spent by all reduce tasks (ms)=3116
4 P0 K& U/ s1 O {) q* u Total vcore-milliseconds taken by all map tasks=41394& T: w) x; k: c" T% U
Total vcore-milliseconds taken by all reduce tasks=3116
( ] v$ |* _( e& B Total megabyte-milliseconds taken by all map tasks=423874564 M1 O9 Q# ?1 z6 x, y7 i/ l
Total megabyte-milliseconds taken by all reduce tasks=3190784: s( W3 f& E4 b* a
Map-Reduce Framework( F+ q+ b" \! K& k, n3 S; J
Map input records=10 f% b) N5 H! h& W0 _
Map output records=20
. d0 H6 v$ O' h5 ]8 P& S( D Map output bytes=1807 R2 s! s2 d1 t5 x# W1 K1 y3 ^ S0 s
Map output materialized bytes=280# `2 W# ~' O5 D, \/ U/ U+ [8 a0 ^4 q
Input split bytes=1420 S; Y/ I9 b, @3 W
Combine input records=0
* t9 w: z" ]$ A* M Combine output records=0
! }$ O+ _3 m" S& C% B Reduce input groups=28 P* Y% L1 |, I* z
Reduce shuffle bytes=2804 s/ H4 K9 v. x' @7 ~" R: D3 j9 `7 t
Reduce input records=20* ` _+ w' l" w
Reduce output records=0* h3 q1 C6 R F0 d1 {% o
Spilled Records=40' e( x1 `. ^1 H8 L: p/ Y9 e
Shuffled Maps =10
, h3 |6 T; S; a+ X Failed Shuffles=0
& t/ ]) ?. ^8 ~, r Merged Map outputs=10
A# x$ S/ T. p5 V3 q# k$ @ GC time elapsed (ms)=1915- S8 v! }* ~$ ~/ f
CPU time spent (ms)=67905 H& X/ ~+ D/ ?) U+ s
Physical memory (bytes) snapshot=3809210368
# h4 T* M' z9 U Virtual memory (bytes) snapshot=31150141440
) m. B6 W8 U- u* x) H3 ? Total committed heap usage (bytes)=3503292416: p o' |( K6 k
Peak Map Physical memory (bytes)=360734720# l5 t) M6 c5 Q0 W4 Y- X
Peak Map Virtual memory (bytes)=2835021824
4 ^# [, |' d! D' ]& m% D* | Peak Reduce Physical memory (bytes)=279707648' q; q4 o3 F4 F* {+ Z/ m
Peak Reduce Virtual memory (bytes)=2839097344
- T* v5 K6 D! r4 E, q Shuffle Errors
5 p* e" m( T! Z# J+ j1 [ BAD_ID=0
; \; n9 ?8 L: Z# l) u9 W" o CONNECTION=0 U) g- f ]& S/ P
IO_ERROR=0% F" G- l5 T5 W. H" ]( d
WRONG_LENGTH=0
+ k. T0 m, t- U x WRONG_MAP=0) i5 `% g/ `$ x3 B3 G+ x' k
WRONG_REDUCE=0% @ r! v H" L
File Input Format Counters $ e( L. f- n, _8 y! v1 R" C
Bytes Read=1180
. y# D) R6 `8 y3 }8 |& x9 @, h File Output Format Counters
: N. X$ Z& p' @+ T% o' [' k, q" z Bytes Written=976 R; q+ X* j1 o! q8 I* `0 O
Job Finished in 24.227 seconds
2 }8 T. Z. G" K; Q1 r/ F( V- mEstimated value of Pi is 3.14800000000000000000
: W0 U7 P! ?7 z% R8 e5 [2 f: b0 K% T
问题解决- ~% }: t4 k0 r+ ]( ]- D( [
; m5 F }. O8 v0 y! }$ P
|
|
发表于 2022-11-16 17:00:20