将设为首页浏览此站
开启辅助访问 天气与日历 收藏本站联系我们切换到窄版

易陆发现论坛

 找回密码
 开始注册
查看: 21|回复: 0
收起左侧

hadoop Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.a

[复制链接]
发表于 2022-11-16 17:00:20 | 显示全部楼层 |阅读模式

马上注册,结交更多好友,享用更多功能,让你轻松玩转社区。

您需要 登录 才可以下载或查看,没有帐号?开始注册

x
[admin@hadoop-1 hadoop-3.2.2]$ hadoop jar /data/module/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar pi 10 100
: S  z3 @" S, ~$ q" J7 FNumber of Maps  = 108 D4 `" S8 ?3 j/ U* Z4 y- F
Samples per Map = 100
  R* A! i; j2 U7 I% X, @: ?Wrote input for Map #0
% ~  J2 l9 `' c( @' e, NWrote input for Map #1
! s. z& u  ?4 k3 HWrote input for Map #2! v1 N/ ^6 S) F4 w9 Z+ `
Wrote input for Map #3
0 d& O7 X; ?/ K& yWrote input for Map #4$ a3 S' V! P( _4 J
Wrote input for Map #5
" h  r4 L4 s& B: H5 S0 o! RWrote input for Map #6( v+ l! _! C! l  E; j. N
Wrote input for Map #7
7 _# O7 @" {" C6 b7 H3 {! S* mWrote input for Map #8% y; r! H+ U* Z2 @
Wrote input for Map #9
7 J$ x* X# M* gStarting Job
- O8 T% i. K9 o9 H+ A6 P& X2022-11-16 15:32:34,946 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm26 J- f9 Q- O$ h  L; A+ O% \
2022-11-16 15:32:35,174 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/admin/.staging/job_1668583281306_0001; J! ^; k$ b( V3 ]6 I
2022-11-16 15:32:35,389 INFO input.FileInputFormat: Total input files to process : 10: ?1 P1 S& }, _1 D* n
2022-11-16 15:32:35,592 INFO mapreduce.JobSubmitter: number of splits:10" a* I9 V  N& s
2022-11-16 15:32:35,806 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1668583281306_0001
6 P4 B# j/ W" {% s& X2022-11-16 15:32:35,808 INFO mapreduce.JobSubmitter: Executing with tokens: []
: H4 ~9 i* E; }2022-11-16 15:32:36,044 INFO conf.Configuration: resource-types.xml not found+ F+ m; G5 w" V, G8 j) P3 a
2022-11-16 15:32:36,044 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.7 I0 g6 q( [. G, M) v
2022-11-16 15:32:36,423 INFO impl.YarnClientImpl: Submitted application application_1668583281306_0001
) M4 p0 L0 t7 _- O2 V4 w' j' V2022-11-16 15:32:36,504 INFO mapreduce.Job: The url to track the job: http://hadoop-2:8088/proxy/application_1668583281306_0001/
3 G& l2 e" i3 I: r& K" c2 w2022-11-16 15:32:36,505 INFO mapreduce.Job: Running job: job_1668583281306_00011 r7 h9 {  g( ]
2022-11-16 15:32:41,561 INFO mapreduce.Job: Job job_1668583281306_0001 running in uber mode : false
. F- {5 t) }. r- F2022-11-16 15:32:41,563 INFO mapreduce.Job:  map 0% reduce 0%- D4 q  \% u3 c; ~
2022-11-16 15:32:41,583 INFO mapreduce.Job: Job job_1668583281306_0001 failed with state FAILED due to: Application application_1668583281306_0001 failed 2 times due to AM Container for appattempt_1668583281306_0001_000002 exited with  exitCode: 1  k" v, `% \8 y" v4 k2 ^
Failing this attempt.Diagnostics: [2022-11-16 15:32:40.765]Exception from container-launch.
* ?+ ]; `4 c* R7 G4 }7 zContainer id: container_1668583281306_0001_02_000001: Z; q# w7 k0 o
Exit code: 1
/ c" r3 Q5 C+ O0 ~1 a6 @- i* K; ?. K5 H" m" J: @6 y" Z2 T2 y
[2022-11-16 15:32:40.799]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
; t+ m# ~7 f3 M- y2 V& @- ULast 4096 bytes of prelaunch.err :: `" K- ~8 e8 j4 `, }$ {
Last 4096 bytes of stderr :
5 M2 O) Q# u. o+ H: S, _0 D: IError: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster4 c3 E- h2 y# Z6 i6 u$ Q1 U  n7 V. K

. j, [& A( ]0 ]' ^. i$ M+ {Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:& R: e+ p7 M* R1 Z% K+ i
<property>; S; y4 L9 B) \" g
  <name>yarn.app.mapreduce.am.env</name>
3 x' b7 h8 c  f) W  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
1 l1 D) k0 X8 S; K5 _6 r# e</property>
5 D8 B6 ?& N- ?+ i5 w  k# p" t<property>. e0 X) J5 L$ w# B0 }
  <name>mapreduce.map.env</name>
1 V' P) k" p2 c! ?0 U2 C  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
, f; f  \( B9 g1 N% g</property>
) ^# w3 N8 |: B8 c/ q<property>
9 V7 }; g' S" Q  r. c3 x  <name>mapreduce.reduce.env</name>' o8 d) ~( B( E8 |% [3 @) J
  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>* @# y+ V' b7 @
</property>
& N8 A; B# Y9 T2 N' z; x' S8 h+ `3 f" q. T$ R
[2022-11-16 15:32:40.800]Container exited with a non-zero exit code 1. Error file: prelaunch.err.3 Y: `9 e4 x7 j* k, N
Last 4096 bytes of prelaunch.err :
/ @$ ~! \. Q1 G. g, x0 |Last 4096 bytes of stderr :* S3 s. h' c8 S+ |# A) l! g- t9 h
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster: B0 n5 R! @- `$ _  A5 T' q
8 X4 ~8 Q# y5 [2 L4 F
Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
- I1 W, A; R  J' n<property>
0 J" I" ^, o( ?9 e# i. M  <name>yarn.app.mapreduce.am.env</name>
6 H% ~' K2 ?- ^7 a5 W7 x; g  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>
+ {7 g# O/ n$ c5 E  n9 n" T1 c</property>8 j, m7 N: |- r- B' U4 w2 C
<property>
1 B! {6 g* w# O  <name>mapreduce.map.env</name>
& h4 o! E; Q+ o/ J- d  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>! C6 a% u0 Z, s8 s" ~" R
</property>
& h( y/ Z2 I1 h5 k" N" D. [& O" e<property>4 w2 l# P  I3 \  n1 Q5 A% M; m3 ^) k3 k
  <name>mapreduce.reduce.env</name>
" J+ h: c* q2 g+ m) Y) ?" a  <value>HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}</value>5 u9 X2 p3 ~" K3 G2 @
</property>
; C4 b# S* @3 W) j/ `6 M$ V9 s0 F" b1 H- _. w
For more detailed output, check the application tracking page: http://hadoop-2:8088/cluster/app/application_1668583281306_0001 Then click on links to logs of each attempt.
6 ?+ X* U- r# |2 V. Failing the application.$ C6 {  r2 K" C* L
2022-11-16 15:32:41,608 INFO mapreduce.Job: Counters: 0( ?! x/ m7 B  p3 ~1 ?3 i/ E, q3 {" a
Job job_1668583281306_0001 failed!
, V  j4 b& A& \1 I6 W/ X5 n0 a# S9 A* A- C" J0 C
分析失败,解决方法:
9 |1 Q6 r3 I; O5 x+ _: J; o
1 K8 o: P. i- w( u! p[admin@hadoop-1 hadoop-3.2.2]$ vim etc/hadoop/mapred-site.xml, R! r% f) V& @2 ?3 |! v
添加如下内容:
5 A3 [% D# W  N4 ~$ T* l9 R/ S+ ?  p3 ^% O" a" e
<property>& W9 a( i! p! R2 _6 }
  <name>yarn.app.mapreduce.am.env</name>. F9 N+ n5 z1 H/ R9 m
  <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
$ x3 f9 C% e! Z</property>: x2 e7 ~. o7 u$ k% r
<property>
# q; m/ h; R8 C  I( q& }+ I: J  <name>mapreduce.map.env</name>
" G7 k0 H& G3 u  _: w  <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
5 g, _) X0 ]) H5 V</property>5 U& n4 c* t* [$ Y9 u/ {
<property>
3 l+ t6 w: Z8 s! F5 {$ D  <name>mapreduce.reduce.env</name>4 M/ Q; L+ b0 Z7 _
  <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
9 e' F+ L8 X* B% D) U8 r</property>5 M) r6 `  \4 g- I
* a3 K% ~9 Y2 I' {" X& ~. X

+ ]# I8 J) N$ E4 f5 ?- a同步到其他节点:3 q( C8 e+ e0 Z' z" R
& ]6 \5 }  \% b& ?- m+ a

3 ~/ v" z3 d, r% W# D" O; u% [
5 g+ l$ E; E" M8 T  u( `% ^" z' I. p: S/ R1 l! Q6 u
[admin@hadoop-1 hadoop-3.2.2]$ xsync etc/hadoop/- X2 w' h& h# Z+ J' z  M
==============hadoop-1==================
5 A7 ]- K3 O( G$ _$ c/ _sending incremental file list9 k: c- J! }* z- V7 f6 _1 L

$ U+ h3 J( P6 b& K- J! g, x. Isent 1,004 bytes  received 18 bytes  2,044.00 bytes/sec
6 _& l' A  d; D' @9 k+ w, xtotal size is 115,250  speedup is 112.77
% f. E; i1 Y1 l==============hadoop-2==================: G7 A3 p- ]7 D! H7 B
sending incremental file list
( {5 ]8 N% H' A7 v  a, |hadoop/
# M- |* ?. V, ?8 ]0 [0 Shadoop/mapred-site.xml6 O# R  v- ]9 m* G! k
) I( |9 T# ~+ c/ G4 b2 D
sent 1,251 bytes  received 52 bytes  2,606.00 bytes/sec" B; w. w$ ^% [. E5 A" R  P
total size is 115,250  speedup is 88.45
% O$ g; Y' Y7 m! E4 E- g==============hadoop-3==================# z; d0 E- ]9 B8 t" |3 \! I8 h
sending incremental file list
$ P! l! p+ Q# r. w8 W, t/ dhadoop/& Q/ J2 G- Y6 T% U2 Z# ?
hadoop/mapred-site.xml) R8 i$ Y. C7 W- P# ?( l, M
  J2 K8 m4 |9 ?5 M7 \; E
sent 1,251 bytes  received 52 bytes  2,606.00 bytes/sec
# a4 H: a) u9 R+ Q  ~" P0 }" Ptotal size is 115,250  speedup is 88.45
2 ]. q( x! x: @* M
7 F) J6 }1 G. G* ~0 o' }: |7 R/ ]$ s7 o7 u- S
再次验证:  @9 l* T$ r/ S+ W( G

6 Q0 I5 g* D0 s" r2 z[admin@hadoop-1 hadoop-3.2.2]$ hadoop jar /data/module/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar pi 10 1002 w! L( x9 b+ b0 [
Number of Maps  = 106 D6 n7 `7 |! G2 w' n
Samples per Map = 100
9 K: Z+ w7 F0 d1 [" B5 fWrote input for Map #00 m, t& o+ z: `
Wrote input for Map #13 G8 _" F3 {' `  R# p9 ~
Wrote input for Map #2: \( R5 y; M( E; a5 d/ H
Wrote input for Map #3
: p" e8 }/ m- n/ s' EWrote input for Map #47 L3 g  }7 D2 S' a' f8 }8 o& E. ]
Wrote input for Map #5
9 x8 ]) Z9 p, j- r( j0 }/ Q+ m. q! TWrote input for Map #6" g5 C3 P  y6 N, `6 E5 S+ U- ~
Wrote input for Map #7
6 p6 ]4 J) _, n+ f4 G$ BWrote input for Map #81 N0 r# {+ a# z/ s
Wrote input for Map #96 j+ o& \. o; {' m# L$ M, n
Starting Job! ]5 a+ w3 N! a. {* A/ K5 s
2022-11-16 15:36:05,630 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2! Y9 K) b7 d& }9 C
2022-11-16 15:36:05,878 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/admin/.staging/job_1668583281306_0002' K3 `) Y: h; p! t
2022-11-16 15:36:06,065 INFO input.FileInputFormat: Total input files to process : 10
  e7 f; S7 B' u9 m' O2022-11-16 15:36:06,214 INFO mapreduce.JobSubmitter: number of splits:10
+ q9 Y& ^) u$ t/ X6 ^) [2022-11-16 15:36:06,444 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1668583281306_0002
* W; Y5 Q& x/ K# ~2022-11-16 15:36:06,446 INFO mapreduce.JobSubmitter: Executing with tokens: []+ ~7 n* v! G, A9 ]% \5 a% `
2022-11-16 15:36:06,654 INFO conf.Configuration: resource-types.xml not found
: I6 @) j2 [  M2022-11-16 15:36:06,654 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.3 b+ H) G1 h9 ^1 x
2022-11-16 15:36:06,928 INFO impl.YarnClientImpl: Submitted application application_1668583281306_0002
( {' y* J6 q: N- U& o, |) b2022-11-16 15:36:06,981 INFO mapreduce.Job: The url to track the job: http://hadoop-2:8088/proxy/application_1668583281306_0002/
+ F* M% z, v& }. g7 h  H+ m6 x8 s8 {2022-11-16 15:36:06,982 INFO mapreduce.Job: Running job: job_1668583281306_0002; w/ l; r9 G! ]! F
2022-11-16 15:36:17,134 INFO mapreduce.Job: Job job_1668583281306_0002 running in uber mode : false5 O4 G: ^7 P5 _7 ?( V: u
2022-11-16 15:36:17,136 INFO mapreduce.Job:  map 0% reduce 0%
6 ]2 S5 S* z- R" K. H2022-11-16 15:36:23,297 INFO mapreduce.Job:  map 40% reduce 0%6 w1 a4 {1 Q0 e4 z1 z0 W
2022-11-16 15:36:24,308 INFO mapreduce.Job:  map 100% reduce 0%; i, C6 D5 E; V+ W* h) n
2022-11-16 15:36:28,342 INFO mapreduce.Job:  map 100% reduce 100%
% r# m8 @- P  A1 n& k- k1 P2022-11-16 15:36:29,360 INFO mapreduce.Job: Job job_1668583281306_0002 completed successfully
1 b0 ]4 p4 c4 i, v  J: ?# _0 c2022-11-16 15:36:29,504 INFO mapreduce.Job: Counters: 54
- d7 |( U' H0 x" d5 i  V        File System Counters5 J4 \9 E- B1 p* }
                FILE: Number of bytes read=2260 |# ~8 x7 o) r, n( s! i
                FILE: Number of bytes written=2630034/ w( X& G& M# Z; l
                FILE: Number of read operations=0
9 l5 g1 w) |* T* v/ q; ]                FILE: Number of large read operations=01 N$ R4 n9 f; ^6 I8 ^
                FILE: Number of write operations=0
) l9 z: j+ j7 b                HDFS: Number of bytes read=2600
1 m4 }' x) R5 v3 f4 i                HDFS: Number of bytes written=215
1 C4 B5 }2 r4 P- P) ]7 ^                HDFS: Number of read operations=45
7 I/ Z- L6 ]5 b( l                HDFS: Number of large read operations=0# t% l( g' S' j0 t, f
                HDFS: Number of write operations=3
: I6 S* R9 X; n# R# J                HDFS: Number of bytes read erasure-coded=0! ?: }) d! Y5 w8 Z- h# [' ]
        Job Counters : S- e# x. H9 b5 @, c
                Launched map tasks=10) M. U0 l( c. f
                Launched reduce tasks=1
& S) Q0 `) V* K' C9 o6 d8 G                Data-local map tasks=104 D# z2 v6 \+ ^5 U1 C2 K
                Total time spent by all maps in occupied slots (ms)=41394, u: a% J+ O+ S  t! f& h2 _
                Total time spent by all reduces in occupied slots (ms)=31162 j% i2 A7 |  q& R- }
                Total time spent by all map tasks (ms)=41394
/ g4 R2 ]7 w5 T2 g4 P                Total time spent by all reduce tasks (ms)=3116* T* X  E5 B8 v. H% {
                Total vcore-milliseconds taken by all map tasks=413949 }1 [* ]; u& _* o3 g% G# o
                Total vcore-milliseconds taken by all reduce tasks=3116
! b7 r9 q6 l8 ~( B. @5 y                Total megabyte-milliseconds taken by all map tasks=42387456
) l6 t& P" U% c% E' z2 A                Total megabyte-milliseconds taken by all reduce tasks=31907842 i: B+ g; X! W" X- h# L3 K/ }2 |
        Map-Reduce Framework# L8 D1 |# j7 V) Z6 `6 t& u- W7 `
                Map input records=102 C+ f0 W& Y2 G+ l( \
                Map output records=20
9 Q& I* k- i: M* ~- {                Map output bytes=180' r! ~3 k, t, a( z4 ^
                Map output materialized bytes=280
! d6 o* q0 k& T7 M( \2 X( [( E( `                Input split bytes=1420* Q# I7 M9 |9 l+ o
                Combine input records=0
; ~& D: \6 a, ?8 m3 x                Combine output records=0- o0 D, L- [2 F( b
                Reduce input groups=2
3 s; I, d6 Z+ i- u                Reduce shuffle bytes=2808 C; \( |* h6 N8 [8 x2 R2 H: }' ?! Z
                Reduce input records=20
0 `6 D( Y+ @: E2 z                Reduce output records=02 a9 L* f  v6 ]" O- z
                Spilled Records=40+ d6 ?( l' p5 r* |  N8 T9 U
                Shuffled Maps =10! ~& T3 l6 G  X$ u  x8 |7 W
                Failed Shuffles=0
8 x2 D8 y0 g3 v9 H                Merged Map outputs=10
% X: I. N2 z) v- R, M9 _                GC time elapsed (ms)=1915' U8 F3 C7 M+ x4 p7 {/ ], ~7 c
                CPU time spent (ms)=6790
8 |1 N. y" i$ P( r9 F: k3 O                Physical memory (bytes) snapshot=3809210368
" G0 q. n2 `+ w5 G6 o0 d8 f- D9 Y                Virtual memory (bytes) snapshot=31150141440  `  C& N' n# u. E6 ^9 y
                Total committed heap usage (bytes)=3503292416
- v7 B5 Q0 l7 m% K9 S/ T9 W9 S                Peak Map Physical memory (bytes)=360734720
  V" r2 c; R7 g                Peak Map Virtual memory (bytes)=2835021824
9 e. V" A3 R* D) X5 H                Peak Reduce Physical memory (bytes)=279707648; F$ P0 e  _3 Q
                Peak Reduce Virtual memory (bytes)=2839097344: K  v( H: H# `; v' h. u/ W
        Shuffle Errors: G/ ^, f* ~% Y# ]9 o1 K
                BAD_ID=0
5 E* i8 t$ E% g8 [; M9 i9 V                CONNECTION=0
- J. k( q2 k; H3 S( z4 E1 b0 p/ z                IO_ERROR=07 O# j* \% I- E, ?4 O4 f+ L+ z
                WRONG_LENGTH=0
/ T7 n7 r$ u- ?& _) F                WRONG_MAP=0
$ h9 Q% J6 {4 J8 h  u. U                WRONG_REDUCE=06 H0 I' D7 W5 `
        File Input Format Counters + ?. {3 f+ [& v
                Bytes Read=11801 B& B; y1 v7 P  u7 ?, M* l8 |
        File Output Format Counters
1 n8 }) q( f  ?                Bytes Written=97
  k/ L7 V6 d  b0 D0 m/ WJob Finished in 24.227 seconds
% \/ O# m! d) ~; @, |& d4 W  PEstimated value of Pi is 3.14800000000000000000
2 W+ B# M- z+ W; @
# i3 U+ n9 |8 O) C" |) i问题解决
; R8 x; X6 f0 P8 E
8 R: w* M& E9 D3 ?/ Y2 c* Z+ J
您需要登录后才可以回帖 登录 | 开始注册

本版积分规则

关闭

站长推荐上一条 /4 下一条

如有购买积分卡请联系497906712

QQ|返回首页|Archiver|手机版|小黑屋|易陆发现 点击这里给我发消息

GMT+8, 2022-12-10 09:56 , Processed in 0.045540 second(s), 22 queries .

Powered by LR.LINUX.cloud bbs168x X3.2 Licensed

© 2012-2022 Comsenz Inc.

快速回复 返回顶部 返回列表