开启辅助访问 天气与日历 收藏本站联系我们相册切换到窄版

易陆发现论坛

 找回密码
 开始注册
查看: 459|回复: 0
收起左侧

hadoop Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.a

[复制链接]
发表于 2022-11-16 17:00:20 | 显示全部楼层 |阅读模式

马上注册,结交更多好友,享用更多功能,让你轻松玩转社区。

您需要 登录 才可以下载或查看,没有帐号?开始注册

x
[admin@hadoop-1 hadoop-3.2.2]$ hadoop jar /data/module/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar pi 10 100+ e' S, ]5 \  @8 Y- h) T+ }( m1 e
Number of Maps  = 108 z) a1 {: s) r: K. A
Samples per Map = 100$ e5 k+ v( O4 m- S  G
Wrote input for Map #0# x+ B8 f4 d4 [& o0 s; {0 \7 u0 e; S
Wrote input for Map #1( O+ w3 i1 q* R' L9 r/ p7 ~' F
Wrote input for Map #2
6 G. s. {3 K$ b2 w- m) oWrote input for Map #3; S3 c% \2 |1 C9 Q. R8 ^
Wrote input for Map #42 U+ x+ B* h+ A! F4 B1 k" l
Wrote input for Map #5' k) S. d  x+ A, A
Wrote input for Map #6
. u; _, I' I: E  j) R* rWrote input for Map #7
! L: \- q: B9 w7 O* `8 k' ~% g' hWrote input for Map #8" ~* D% n5 a" X* N4 H5 I0 q
Wrote input for Map #9! o7 [( b- q; K! o5 k5 i) k
Starting Job
/ O9 \( v3 m$ K8 h4 y" Z! g: f2022-11-16 15:32:34,946 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
# b! X0 [/ R2 `. u+ R4 A( C2022-11-16 15:32:35,174 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/admin/.staging/job_1668583281306_0001* m9 l) j& Q/ h  z
2022-11-16 15:32:35,389 INFO input.FileInputFormat: Total input files to process : 10
9 u. {  @. F2 r8 [0 G& d4 [  N9 _2022-11-16 15:32:35,592 INFO mapreduce.JobSubmitter: number of splits:10
& L1 L& [4 z4 U* V+ k2022-11-16 15:32:35,806 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1668583281306_0001/ V+ o! l1 _! t1 }) f* N
2022-11-16 15:32:35,808 INFO mapreduce.JobSubmitter: Executing with tokens: []  e9 d7 {5 o1 P# a/ V9 ^" L" |* E
2022-11-16 15:32:36,044 INFO conf.Configuration: resource-types.xml not found
( J% i" ^" g! ]; j2022-11-16 15:32:36,044 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.1 R# [* L2 x8 A  |9 N
2022-11-16 15:32:36,423 INFO impl.YarnClientImpl: Submitted application application_1668583281306_0001. F% x: k# l3 H" b5 u
2022-11-16 15:32:36,504 INFO mapreduce.Job: The url to track the job: http://hadoop-2:8088/proxy/application_1668583281306_0001/) A( ?2 G. d8 z' P% n: D/ T9 s
2022-11-16 15:32:36,505 INFO mapreduce.Job: Running job: job_1668583281306_0001+ [. k; l' k4 M# |* `- n
2022-11-16 15:32:41,561 INFO mapreduce.Job: Job job_1668583281306_0001 running in uber mode : false) L' _  b+ @1 d9 d9 e& h- B
2022-11-16 15:32:41,563 INFO mapreduce.Job:  map 0% reduce 0%7 L$ Z, B& t$ m5 e2 O! D1 S
2022-11-16 15:32:41,583 INFO mapreduce.Job: Job job_1668583281306_0001 failed with state FAILED due to: Application application_1668583281306_0001 failed 2 times due to AM Container for appattempt_1668583281306_0001_000002 exited with  exitCode: 1+ R* y. Z1 o& j: w* Z9 p* V6 q
Failing this attempt.Diagnostics: [2022-11-16 15:32:40.765]Exception from container-launch.
; ]3 L( ?- \' w# [/ u0 Q) y: RContainer id: container_1668583281306_0001_02_000001
; l, \* e& z, e- l) G  I/ C% iExit code: 1
) z1 i. j1 K) m4 D6 I7 O
8 T' X# ]* z$ r% q' |% ?/ j: v[2022-11-16 15:32:40.799]Container exited with a non-zero exit code 1. Error file: prelaunch.err." x' u3 t1 }8 Z
Last 4096 bytes of prelaunch.err :
6 s- S* e% |5 ~2 @+ |Last 4096 bytes of stderr :
% C# g) o5 d: u( D1 T8 VError: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster& a4 t: e% ?8 _7 E0 Z9 t  o, J
/ T9 @' |9 x, L/ G/ e3 y7 Q
Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:7 b( h8 `9 y: P9 _* A2 ~1 g9 ^
<property>
+ R$ Z7 F' T. w* o; D0 O, M  <name>yarn.app.mapreduce.am.env</name>
3 i, r- L. \4 M2 H  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>. P. P& J3 u# ~$ Y" m; \7 F( q3 M
</property>
/ B  B6 O& {' L( k<property>/ P' o1 f+ g& h  z( G( s& a, A
  <name>mapreduce.map.env</name>
3 U) b. ]* O9 ^& i- ^8 U  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
/ J9 F+ T; R- D6 \</property>
6 i& p; p/ P8 o4 c' g: R& }<property>- l# ~0 B8 T0 a# m- J% @7 O
  <name>mapreduce.reduce.env</name># O2 C, n! J" C8 F1 l( B  D' K
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
, z: y6 c" x1 e</property>  N. T/ A1 s$ I: I% ^4 K9 ?+ Q

% q+ V% N- {& B! H0 Z3 E8 v% H[2022-11-16 15:32:40.800]Container exited with a non-zero exit code 1. Error file: prelaunch.err.5 m' L8 P  N0 p. h) ^+ ~' f2 o
Last 4096 bytes of prelaunch.err :
: E! G9 U7 u& d% @Last 4096 bytes of stderr :
$ t1 Z& ?5 x7 v$ g" BError: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster7 g5 v; V3 l; B* B% R0 p: k& n( _

5 J% b' W% I  q# d! P8 YPlease check whether your etc/hadoop/mapred-site.xml contains the below configuration:! G1 D! _+ P  ?) q
<property>
$ ?$ |. p1 j$ G  <name>yarn.app.mapreduce.am.env</name>
" R$ |, F5 ^3 C5 E  H# T  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
" T8 q8 j) b* E% p+ e& t, y</property>" m# ?7 n+ T& A9 M/ l
<property>( q1 E7 X/ l6 b' `! \& `
  <name>mapreduce.map.env</name>
4 [% f. X( c8 a; u# N6 E) j1 X/ k1 r: X  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>( G1 u: q/ a) j1 {( x# R
</property>
8 N0 D, _, F3 s, H! x5 n<property>
1 F9 h  O% z+ k4 m: a1 G7 M" y  <name>mapreduce.reduce.env</name>% I# P% ]* J# E" q2 Z2 n& L
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
9 l2 E1 E* t/ ?, L# P7 n$ a</property>
0 T" _8 a, K" e  Z* b0 I6 ]$ ~
. }' Q) q2 H' P/ uFor more detailed output, check the application tracking page: http://hadoop-2:8088/cluster/app/application_1668583281306_0001 Then click on links to logs of each attempt.
0 g& K# Q+ p, V3 }% C/ V. Failing the application.
' v: g& d; M4 A; @2022-11-16 15:32:41,608 INFO mapreduce.Job: Counters: 0
+ \( M+ i0 _) G2 }$ s  pJob job_1668583281306_0001 failed!
3 L  P- C% I& B$ h5 p* e
/ D% Q, `* z3 X, M" l分析失败,解决方法:( Q, T; k) y2 S. f) v0 `2 X, x0 ?

( d: f2 X( Q% X- l' x: G. M! U# C[admin@hadoop-1 hadoop-3.2.2]$ vim etc/hadoop/mapred-site.xml
* A& l4 D- S# b4 g添加如下内容:" k) u0 y! {0 B7 t( {5 i% ]9 x+ M

3 ]' x! ~) ~9 G) W, P<property>$ L+ a" }5 @/ F  T
  <name>yarn.app.mapreduce.am.env</name>
* \3 F6 J" C2 v' }- }! c/ @  <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
  s- Z! D, O, E' q</property>5 H" F: V4 d! W+ W" P* {* C
<property>
! c* M; C5 \, y  <name>mapreduce.map.env</name>, J/ i3 x# v9 a' E) @' @0 }& _
  <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>! ^" Z! G- B- z/ @2 Z- C6 k
</property>' R% y3 h6 J  w  x- z; _
<property>! w) V  O% {: z: `
  <name>mapreduce.reduce.env</name>9 U/ g& D* s) L
  <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>; Q9 Q& p( I( D
</property>
% |( ]: }  a5 B: ^. G
6 a3 L& B; \7 Y# ?  F9 Z# s, O- i% f5 X
同步到其他节点:
, R' I+ N7 P+ C6 w8 }  z  S$ f
& Z( x+ u% Q3 X& Z( |* ~: Y* g2 R6 i$ X* h) x
8 G9 b) |6 i  c5 k+ C

6 Z0 ]$ q" p! P$ A. h4 p[admin@hadoop-1 hadoop-3.2.2]$ xsync etc/hadoop/
3 E( u1 B1 w0 Y8 O==============hadoop-1==================0 n, x9 k, K4 k5 C4 v
sending incremental file list
. \0 f( k, C  M" }( N6 b8 t- r7 b. b1 u& J, @3 z
sent 1,004 bytes  received 18 bytes  2,044.00 bytes/sec/ d. m4 q+ L+ \6 j* _$ O0 ]
total size is 115,250  speedup is 112.77
" D) |( {( `) Y- g) F+ p' N==============hadoop-2==================
8 W/ e0 H) p+ o8 w: C- K7 A' ^sending incremental file list
1 [; \; ~" E" P% \9 nhadoop/' n5 f7 ?: l1 }2 \/ G$ d
hadoop/mapred-site.xml& l% @& Y6 Y# G+ t+ a

, }8 f0 P5 r" \3 W4 i: qsent 1,251 bytes  received 52 bytes  2,606.00 bytes/sec
" f) ~" `1 l- Ntotal size is 115,250  speedup is 88.45% a* H8 t6 ]. P3 a9 t4 B
==============hadoop-3==================
6 ]/ ^5 v; H1 r6 J8 ksending incremental file list
% K: L. f. h2 Y! x; Thadoop/
, |2 c( X# M5 h# uhadoop/mapred-site.xml% T/ `9 Y3 P! K1 U, Z# K: X0 Y
, o4 Q* ~7 R1 Q& q
sent 1,251 bytes  received 52 bytes  2,606.00 bytes/sec+ j: z8 l  G, w0 `+ o
total size is 115,250  speedup is 88.45
: m. |  M. s" @4 a8 o; T  O9 \, V* Q; O

! _# m8 L7 }3 o5 O" o( e! P3 ?再次验证:3 O- N  d* n# `( A

# x+ x9 r2 U) R: j. b[admin@hadoop-1 hadoop-3.2.2]$ hadoop jar /data/module/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar pi 10 100/ P2 b) A5 ~& l& }. x
Number of Maps  = 10, I. ?- B4 M. j+ `' Y% ?6 f
Samples per Map = 100
6 U* n4 Y) j# zWrote input for Map #00 D' A% Z' ^: f* b. ?$ A
Wrote input for Map #1) d( H# B/ z9 G, q1 |
Wrote input for Map #2
' Q1 O- B3 X6 C" o. FWrote input for Map #3
; m8 q* `8 g. v$ U9 E! HWrote input for Map #4
* V  G  j' i1 v! S$ H0 pWrote input for Map #50 J5 O' P8 o: p
Wrote input for Map #6
6 S9 g2 ^, P# Y9 w5 ~9 MWrote input for Map #7# e/ _1 v4 O3 N5 U& t4 G4 Y
Wrote input for Map #8" @7 }5 a/ [1 H' z
Wrote input for Map #9
% O9 j* R% X8 E1 E0 q; ^( K- IStarting Job) ^- p+ e" ]$ L, w/ k: ?. ^* c' I
2022-11-16 15:36:05,630 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
: p3 u; @; Q, W) D: k! P2022-11-16 15:36:05,878 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/admin/.staging/job_1668583281306_0002
9 Z6 D( M5 m6 o6 i; a4 G" u4 V2022-11-16 15:36:06,065 INFO input.FileInputFormat: Total input files to process : 10" a  N' Y+ B- ?# r" ~! i
2022-11-16 15:36:06,214 INFO mapreduce.JobSubmitter: number of splits:10& r0 X# O3 l* f& T1 R9 K4 S* i
2022-11-16 15:36:06,444 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1668583281306_0002" f) k( ]: `/ O, Y
2022-11-16 15:36:06,446 INFO mapreduce.JobSubmitter: Executing with tokens: []$ z' q) \/ M4 c3 }) L
2022-11-16 15:36:06,654 INFO conf.Configuration: resource-types.xml not found9 I$ V9 [- H& o. O/ f; T0 d" j6 I
2022-11-16 15:36:06,654 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.: N) p0 t( W# X& G
2022-11-16 15:36:06,928 INFO impl.YarnClientImpl: Submitted application application_1668583281306_00024 M/ h! U8 Z' ?& W
2022-11-16 15:36:06,981 INFO mapreduce.Job: The url to track the job: http://hadoop-2:8088/proxy/application_1668583281306_0002/2 k! k; r% m1 ?: H* T1 \
2022-11-16 15:36:06,982 INFO mapreduce.Job: Running job: job_1668583281306_0002
1 b: D% `2 h  i. G1 c2022-11-16 15:36:17,134 INFO mapreduce.Job: Job job_1668583281306_0002 running in uber mode : false8 m: x# A. i0 ^8 D6 a, b# c0 F
2022-11-16 15:36:17,136 INFO mapreduce.Job:  map 0% reduce 0%0 n8 s7 o2 C6 |" d% N5 U
2022-11-16 15:36:23,297 INFO mapreduce.Job:  map 40% reduce 0%9 _$ Y/ j$ ~; }% o9 [
2022-11-16 15:36:24,308 INFO mapreduce.Job:  map 100% reduce 0%
4 H2 s- R: V- P2022-11-16 15:36:28,342 INFO mapreduce.Job:  map 100% reduce 100%
' z- |5 b% r" z' e$ x2022-11-16 15:36:29,360 INFO mapreduce.Job: Job job_1668583281306_0002 completed successfully
1 ^1 y5 z' ]3 J9 w0 H. Q4 S2022-11-16 15:36:29,504 INFO mapreduce.Job: Counters: 54
! S# l, u) W# x9 B! X- b; z        File System Counters
. }) r6 s. J# ?8 E9 b                FILE: Number of bytes read=226
. t5 V1 R6 Q; y# h2 o5 b                FILE: Number of bytes written=2630034! E  _6 Z* D$ ]
                FILE: Number of read operations=01 a9 E( {2 b7 t
                FILE: Number of large read operations=0% C& e+ B$ c% P) v" G
                FILE: Number of write operations=07 W; O! i1 d& C4 I9 d! B$ a
                HDFS: Number of bytes read=2600
6 F1 t: e/ T% \  N0 l5 C                HDFS: Number of bytes written=2154 W% Y$ m2 k) j. T- U
                HDFS: Number of read operations=452 G3 n, N' y( a5 V! p
                HDFS: Number of large read operations=0& V' }% \  Q4 r7 D' `
                HDFS: Number of write operations=3
% A% C& `; `+ \: Q  Q" V% X                HDFS: Number of bytes read erasure-coded=00 q' m8 T" O4 L
        Job Counters 1 M3 w$ K0 h" i% g
                Launched map tasks=104 r& Q/ q0 R' y! Y9 l+ r
                Launched reduce tasks=1. L- D' S( n" h: R  o+ D
                Data-local map tasks=10. U, \; b; Q: E( z* `/ g
                Total time spent by all maps in occupied slots (ms)=41394/ J/ c6 x* I" K$ Y8 E
                Total time spent by all reduces in occupied slots (ms)=3116+ R8 e  F' `6 g: l* }
                Total time spent by all map tasks (ms)=413943 v9 p0 ]8 s6 C2 Q% Z) q* x
                Total time spent by all reduce tasks (ms)=3116& W, ]$ Q4 O* H. E+ Q3 T
                Total vcore-milliseconds taken by all map tasks=41394
$ W7 x. |3 j& r2 P4 ]                Total vcore-milliseconds taken by all reduce tasks=31164 a9 B' V: z% W, B. b4 t$ ~
                Total megabyte-milliseconds taken by all map tasks=42387456
! Q5 [1 v0 b1 m% X6 p7 Z  u6 O2 S5 D( A                Total megabyte-milliseconds taken by all reduce tasks=3190784! y1 {7 E* t$ g* s$ ~- S# v
        Map-Reduce Framework
/ B6 Z$ A( j/ l* G) h- v                Map input records=10
. d$ g3 F2 q3 z2 l                Map output records=20
" ~2 U2 E# h3 \1 x' F; M/ @0 o                Map output bytes=180
  p8 d: y" W- y$ s+ W5 _; @4 ^                Map output materialized bytes=280
- O/ B6 h/ Z+ p                Input split bytes=1420
9 U5 W# v% z2 m9 H$ r7 @( h                Combine input records=0
" H  I: D3 [- q                Combine output records=0
$ X! `* u: [: @- \                Reduce input groups=24 D7 M; O8 Y  Y( z) H
                Reduce shuffle bytes=280
! S& O3 w/ o8 A. v% q0 f$ C                Reduce input records=20
0 D+ q& H) `# U) H# A                Reduce output records=04 @* m$ g0 Y& u5 v8 H
                Spilled Records=40
  K6 s8 b: U- e0 w% I                Shuffled Maps =10
8 S+ B: d! z6 b                Failed Shuffles=0' B2 H1 `( y5 b3 A0 e
                Merged Map outputs=10
7 |! B1 U7 K$ V" {( \. T2 a7 y                GC time elapsed (ms)=19156 n- ?+ E8 T* E& c) z# d
                CPU time spent (ms)=6790* ?+ ?" l2 H; s8 N
                Physical memory (bytes) snapshot=3809210368
& ]2 {+ D; f0 P% U9 s: E5 Y                Virtual memory (bytes) snapshot=31150141440
1 @: G. E1 p. [7 B/ r: T) c                Total committed heap usage (bytes)=3503292416+ r, w, _3 r: i1 S
                Peak Map Physical memory (bytes)=360734720
% s9 @+ E* i5 e                Peak Map Virtual memory (bytes)=28350218241 K9 S; u# _2 t. V
                Peak Reduce Physical memory (bytes)=279707648
  D& X9 H' Q9 N$ Y0 n                Peak Reduce Virtual memory (bytes)=2839097344; x) P) P! i1 A% c
        Shuffle Errors/ m6 [. R) `  Z6 U* i" K
                BAD_ID=0
6 u- a! U1 V# l5 N                CONNECTION=0' N) H$ x& J; O! t1 d
                IO_ERROR=0) u" ]. d! ^6 K" C
                WRONG_LENGTH=09 u' s6 D0 R. t7 R; b" \, r- s
                WRONG_MAP=0
0 o* k' M. }: K, J) ]                WRONG_REDUCE=0
  ^/ e" P6 f4 @        File Input Format Counters
5 Q5 X! S# D; _. f                Bytes Read=1180. z3 \" I8 U5 L& W. S5 y$ ]5 j6 `
        File Output Format Counters
$ P" n1 b$ W1 v  R4 W                Bytes Written=97% w2 E! ~! U$ A. V8 C. O
Job Finished in 24.227 seconds: \  f/ }$ l2 T% {3 M0 g
Estimated value of Pi is 3.14800000000000000000! p* l' G, v  U! F. q5 u
! B: W/ G' v6 k! W
问题解决
# o: ?4 O; A" I$ n9 {" d: W/ {( A4 C2 ?" u
您需要登录后才可以回帖 登录 | 开始注册

本版积分规则

关闭

站长推荐上一条 /4 下一条

如有购买积分卡请联系497906712

QQ|返回首页|Archiver|手机版|小黑屋|易陆发现技术论坛 点击这里给我发消息

GMT+8, 2023-12-2 03:38 , Processed in 0.043288 second(s), 22 queries .

Powered by LR.LINUX.cloud bbs168x X3.2 Licensed

© 2012-2025 Comsenz Inc.

快速回复 返回顶部 返回列表