Ambari是Hadoop集群的一个管理服务,很多时候如果需要定制自己的Hadoop发行版或者开发Hadoop云服务的时候,就需要涉及到一些自定义的开发,这个时候可能就会使用到Ambari。

Ambari开发语言以Java和Py为主,同时还有大量的shell脚本,因为Ambari最终生成的是对应的机器的安装包,所以需要考虑一下自己的编译环境,工程默认提供了一个基于centos的docker编译环境,可以通过start-build-env.sh脚本进入环境。该脚本是将本地的环境,例如mvn,java以及源码工程目录挂载到docker容器中。

编译准备

源码地址:https://github.com/apache/ambari.git

在编译之前,需要先设置参数:

1
mvn versions:set-property -Dproperty=revision -DnewVersion=2.7.5

因为我本次是基于master代码,最新是2.7.5的正式版,所以使用了这个版本,确定版本后,通过如下代码进行编译:

1
mvn -B clean install package rpm:rpm -DnewVersion=2.7.5 -DskipTests -Drat.skip -Dpython.ver="python >= 2.6" -Preplaceurl

-Drat.skip参数是为了跳过licensing 检查,防止报错Too many files with unapproved license

编译问题

Node版本问题

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
[INFO] --- exec-maven-plugin:1.2.1:exec (Bower install) @ ambari-admin ---
bower error Unexpected token {Stack trace:
SyntaxError: Unexpected token {
at exports.runInThisContext (vm.js:53:16)
at Module._compile (module.js:373:25)
at Object.Module._extensions..js (module.js:416:10)
at Module.load (module.js:343:32)
at Function.Module._load (module.js:300:12)
at Module.require (module.js:353:17)
at require (internal/module.js:12:17)
at Object.<anonymous> (/usr/local/apache-ambari-2.7.5-src/ambari-admin/src/main/resources/ui/admin-web/node_modules/bower/node_modules/bower-registry-client/node_modules/request/lib/cookies.js:3:13)
at Module._compile (module.js:409:26)
at Object.Module._extensions..js (module.js:416:10)Console trace:
Trace
at StandardRenderer.error (/usr/local/apache-ambari-2.7.5-src/ambari-admin/src/main/resources/ui/admin-web/node_modules/bower/lib/renderers/StandardRenderer.js:72:17)
at Logger.<anonymous> (/usr/local/apache-ambari-2.7.5-src/ambari-admin/src/main/resources/ui/admin-web/node_modules/bower/bin/bower:111:22)
at emitOne (events.js:77:13)
at Logger.emit (events.js:169:7)
at Logger.emit (/usr/local/apache-ambari-2.7.5-src/ambari-admin/src/main/resources/ui/admin-web/node_modules/bower/node_modules/bower-logger/lib/Logger.js:29:39)
at /usr/local/apache-ambari-2.7.5-src/ambari-admin/src/main/resources/ui/admin-web/node_modules/bower/lib/commands/index.js:40:20
at _rejected (/usr/local/apache-ambari-2.7.5-src/ambari-admin/src/main/resources/ui/admin-web/node_modules/bower/node_modules/q/q.js:797:24)
at /usr/local/apache-ambari-2.7.5-src/ambari-admin/src/main/resources/ui/admin-web/node_modules/bower/node_modules/q/q.js:823:30
at Promise.when (/usr/local/apache-ambari-2.7.5-src/ambari-admin/src/main/resources/ui/admin-web/node_modules/bower/node_modules/q/q.js:1035:31)
at Promise.promise.promiseDispatch (/usr/local/apache-ambari-2.7.5-src/ambari-admin/src/main/resources/ui/admin-web/node_modules/bower/node_modules/q/q.js:741:41)System info:
Bower version: 1.3.8
Node version: 4.5.0
OS: Linux 3.10.0-1062.el7.x86_64 x64
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Admin View 2.7.5.0.0 ........................ FAILURE [03:29 min]
[INFO] ambari-utility 1.0.0.0-SNAPSHOT .................... SKIPPED
[INFO] ambari-metrics 2.7.5.0.0 ........................... SKIPPED
[INFO] Ambari Metrics Common 2.7.5.0.0 .................... SKIPPED
[INFO] Ambari Metrics Hadoop Sink 2.7.5.0.0 ............... SKIPPED
[INFO] Ambari Metrics Flume Sink 2.7.5.0.0 ................ SKIPPED
[INFO] Ambari Metrics Kafka Sink 2.7.5.0.0 ................ SKIPPED
[INFO] Ambari Metrics Storm Sink 2.7.5.0.0 ................ SKIPPED
[INFO] Ambari Metrics Storm Sink (Legacy) 2.7.5.0.0 ....... SKIPPED
[INFO] Ambari Metrics Collector 2.7.5.0.0 ................. SKIPPED
[INFO] Ambari Metrics Monitor 2.7.5.0.0 ................... SKIPPED
[INFO] Ambari Metrics Grafana 2.7.5.0.0 ................... SKIPPED
[INFO] Ambari Metrics Host Aggregator 2.7.5.0.0 ........... SKIPPED
[INFO] Ambari Metrics Assembly 2.7.5.0.0 .................. SKIPPED
[INFO] Ambari Service Advisor 1.0.0.0-SNAPSHOT ............ SKIPPED
[INFO] Ambari Server 2.7.5.0.0 ............................ SKIPPED
[INFO] Ambari Functional Tests 2.7.5.0.0 .................. SKIPPED
[INFO] Ambari Agent 2.7.5.0.0 ............................. SKIPPED
[INFO] ambari-logsearch 2.7.5.0.0 ......................... SKIPPED
[INFO] Ambari Logsearch Appender 2.7.5.0.0 ................ SKIPPED
[INFO] Ambari Logsearch Config Api 2.7.5.0.0 .............. SKIPPED
[INFO] Ambari Logsearch Config JSON 2.7.5.0.0 ............. SKIPPED
[INFO] Ambari Logsearch Config Solr 2.7.5.0.0 ............. SKIPPED
[INFO] Ambari Logsearch Config Zookeeper 2.7.5.0.0 ........ SKIPPED
[INFO] Ambari Logsearch Config Local 2.7.5.0.0 ............ SKIPPED
[INFO] Ambari Logsearch Log Feeder Plugin Api 2.7.5.0.0 ... SKIPPED
[INFO] Ambari Logsearch Log Feeder Container Registry 2.7.5.0.0 SKIPPED
[INFO] Ambari Logsearch Log Feeder 2.7.5.0.0 .............. SKIPPED
[INFO] Ambari Logsearch Web 2.7.5.0.0 ..................... SKIPPED
[INFO] Ambari Logsearch Server 2.7.5.0.0 .................. SKIPPED
[INFO] Ambari Logsearch Assembly 2.7.5.0.0 ................ SKIPPED
[INFO] Ambari Logsearch Integration Test 2.7.5.0.0 ........ SKIPPED
[INFO] ambari-infra 2.7.5.0.0 ............................. SKIPPED
[INFO] Ambari Infra Solr Client 2.7.5.0.0 ................. SKIPPED
[INFO] Ambari Infra Solr Plugin 2.7.5.0.0 ................. SKIPPED
[INFO] Ambari Infra Manager 2.7.5.0.0 ..................... SKIPPED
[INFO] Ambari Infra Assembly 2.7.5.0.0 .................... SKIPPED
[INFO] Ambari Infra Manager Integration Tests 2.7.5.0.0 ... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:30 min
[INFO] Finished at: 2020-08-05T15:50:13+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (Bower install) on project ambari-admin: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

碰到这个问题是因为代码里面指定的node版本和环境设置的版本不一致,可以由多种解决办法,我使用的最简单的一种就是修改ambari-admin工程里面的exec-maven-plugin插件,将:

1
<argument>${basedir}/src/main/resources/ui/admin-web/node_modules/bower/bin/bower</argument>

替换成:

1
<argument>bower</argument>

也可以修改pom文件中的nodeVersionnpmVersion,查看环境中的版本号,例如做如下配置:

1
2
3
<!--修改ambari-admin模块下的pom.xml文件 -->
<nodeVersion>3.10.10</nodeVersion>
<npmVersion>v6.17.1</npmVersion>

这样的话也能解决问题。

下载路径问题

Ambari的监控部分需要依赖Hbase等组件作为存储,默认会使用HDP的官方网站的地址进行下载,因为该地址已经失效,我就将其修改为了Apache的下载地址,修改ambari-metrics-timelineservice工程下的pom文件中的properties部分:

1
2
3
4
5
6
7
8
9
<!--TODO change to HDP URL-->
<hbase.tar>https://archive.apache.org/dist/hbase/2.0.2/hbase-2.0.2-bin.tar.gz</hbase.tar>
<hbase.folder>hbase-2.0.2.3.1.4.0-315</hbase.folder>
<hadoop.tar>https://archive.apache.org/dist/hadoop/core/hadoop-3.1.1/hadoop-3.1.1.tar.gz</hadoop.tar>
<hadoop.folder>hadoop-3.1.1.3.1.4.0-315</hadoop.folder>
<grafana.folder>grafana-6.4.2</grafana.folder>
<grafana.tar>https://dl.grafana.com/oss/release/grafana-6.4.2.linux-amd64.tar.gz</grafana.tar>
<phoenix.tar>https://downloads.apache.org/phoenix/apache-phoenix-5.0.0-HBase-2.0/bin/apache-phoenix-5.0.0-HBase-2.0-bin.tar.gz</phoenix.tar>
<phoenix.folder>phoenix-5.0.0.3.1.4.0-315</phoenix.folder>

将对应的key修改为上面的内容,即可下载对应的二进制包。

同时因为Ambari默认会使用HDP的maven rep去下载hbase,phoenix等依赖的jar,该maven rep里面也下架了相关的依赖,所以需要修改对应的版本让其默认从apache maven rep中进行下载:

1
2
3
4
5
6
7
8
<properties>
<!-- Needed for generating FindBugs warnings using parent pom -->
<!--<yarn.basedir>${project.parent.parent.basedir}</yarn.basedir>-->
<protobuf.version>2.5.0</protobuf.version>
<hadoop.version>3.1.1</hadoop.version>
<phoenix.version>5.0.0-HBase-2.0</phoenix.version>
<hbase.version>2.0.2</hbase.version>
</properties>

调整对应依赖的版本如上图所示,即可,中间大部分问题都会聚焦在版本的对应上,最后编译成功:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ........................................ SUCCESS [ 7.942 s]
[INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.360 s]
[INFO] Ambari Web ......................................... SUCCESS [03:20 min]
[INFO] Ambari Views ....................................... SUCCESS [ 2.930 s]
[INFO] Ambari Admin View .................................. SUCCESS [01:03 min]
[INFO] ambari-utility ..................................... SUCCESS [ 7.990 s]
[INFO] ambari-metrics ..................................... SUCCESS [ 1.056 s]
[INFO] Ambari Metrics Common .............................. SUCCESS [ 14.867 s]
[INFO] Ambari Metrics Hadoop Sink ......................... SUCCESS [03:12 min]
[INFO] Ambari Metrics Flume Sink .......................... SUCCESS [03:01 min]
[INFO] Ambari Metrics Kafka Sink .......................... SUCCESS [02:57 min]
[INFO] Ambari Metrics Storm Sink .......................... SUCCESS [ 6.595 s]
[INFO] Ambari Metrics Storm Sink (Legacy) ................. SUCCESS [ 7.187 s]
[INFO] Ambari Metrics Collector ........................... SUCCESS [14:11 min]
[INFO] Ambari Metrics Monitor ............................. SUCCESS [ 3.587 s]
[INFO] Ambari Metrics Grafana ............................. SUCCESS [04:53 min]
[INFO] Ambari Metrics Host Aggregator ..................... SUCCESS [56:29 min]
[INFO] Ambari Metrics Assembly ............................ SUCCESS [33:47 min]
[INFO] Ambari Service Advisor ............................. SUCCESS [ 13.602 s]
[INFO] Ambari Server ...................................... SUCCESS [ 01:20 h]
[INFO] Ambari Functional Tests ............................ SUCCESS [ 14.766 s]
[INFO] Ambari Agent ....................................... SUCCESS [06:55 min]
[INFO] ambari-logsearch ................................... SUCCESS [ 0.665 s]
[INFO] Ambari Logsearch Appender .......................... SUCCESS [ 7.123 s]
[INFO] Ambari Logsearch Config Api ........................ SUCCESS [ 3.373 s]
[INFO] Ambari Logsearch Config JSON ....................... SUCCESS [ 2.865 s]
[INFO] Ambari Logsearch Config Solr ....................... SUCCESS [ 2.571 s]
[INFO] Ambari Logsearch Config Zookeeper .................. SUCCESS [ 5.323 s]
[INFO] Ambari Logsearch Config Local ...................... SUCCESS [ 3.058 s]
[INFO] Ambari Logsearch Log Feeder Plugin Api ............. SUCCESS [ 13.483 s]
[INFO] Ambari Logsearch Log Feeder Container Registry ..... SUCCESS [ 4.152 s]
[INFO] Ambari Logsearch Log Feeder ........................ SUCCESS [02:15 min]
[INFO] Ambari Logsearch Web ............................... SUCCESS [17:46 min]
[INFO] Ambari Logsearch Server ............................ SUCCESS [05:05 min]
[INFO] Ambari Logsearch Assembly .......................... SUCCESS [ 9.715 s]
[INFO] Ambari Logsearch Integration Test .................. SUCCESS [01:36 min]
[INFO] ambari-infra ....................................... SUCCESS [ 2.566 s]
[INFO] Ambari Infra Solr Client ........................... SUCCESS [01:06 min]
[INFO] Ambari Infra Solr Plugin ........................... SUCCESS [01:20 min]
[INFO] Ambari Infra Manager ............................... SUCCESS [03:56 min]
[INFO] Ambari Infra Assembly .............................. SUCCESS [ 2.229 s]
[INFO] Ambari Infra Manager Integration Tests ............. SUCCESS [01:30 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:07 h
[INFO] Finished at: 2021-05-07T16:39:14+00:00
[INFO] Final Memory: 400M/1073M
[INFO] ------------------------------------------------------------------------
centos7:~/src $

找到对应的包进行安装就行。

安装包在~/src/ambari-server/target/rpm/ambari-server/RPMS/x86_64目录下,使用yum install ambari-server*.rpm进行安装,进行安装的时候可能会碰到:

1
2
3
4
5
6
7
centos7:~/src/ambari-server/target/rpm/ambari-server/RPMS/x86_64 $ yum install ambari-server*.rpm
Loaded plugins: fastestmirror, ovl
ovl: Error while doing RPMdb copy-up:
[Errno 13] Permission denied: '/var/lib/rpm/Triggername'
You need to be root to perform this command.
centos7:~/src/ambari-server/target/rpm/ambari-server/RPMS/x86_64 $ sudo yum install ambari-server*.rpm
bash: sudo: command not found

安装权限不够,sudo也不存在,这时候可以使用root的权限重新进入容器:

1
docker exec -u root -i -t 64814ad3cd7b /bin/bash

再进行安装即可。

上面安装的是ambari server,还需安装agent,对应的包在ambari-agent/target/rpm/ambari-agent/RPMS/x86_64/ambari-agent-2.6.2.0-0.x86_64.rpm下,安装同理。


扫码手机观看或分享: