折腾缘由

Mac编译hadoop网上资料很多，但实际上还需根据自己的环境一步一步来，升级mac后编译过程莫名其妙出错，后来在stackoverflow上有人遇到类似的问题，但不完全一样，折腾了一天，记录下来，以备后查。

先上编译失败的结果，如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [  1.173 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  1.334 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  1.293 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  1.978 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.332 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.720 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  3.722 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  4.176 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  4.740 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  3.002 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:15 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  4.701 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 10.665 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.031 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [02:11 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 15.733 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [  4.172 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  3.373 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.044 s]
[INFO] hadoop-yarn ........................................ SUCCESS [  0.050 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [ 25.481 s]
[INFO] hadoop-yarn-common ................................. SUCCESS [ 19.995 s]
[INFO] hadoop-yarn-server ................................. SUCCESS [  0.064 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [  7.516 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 10.173 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  2.523 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [  4.935 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 12.745 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [  3.749 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [  4.226 s]
[INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [  2.561 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [  0.052 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  1.996 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  1.560 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [  0.035 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [  4.021 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [  3.682 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.121 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 12.740 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 12.236 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  3.988 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [  5.620 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [  3.714 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [  3.391 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  1.486 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  4.121 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [  2.478 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  3.016 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  6.649 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.990 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  4.203 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  2.924 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  2.015 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  2.149 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  2.454 s]
[INFO] Apache Hadoop Pipes ................................ FAILURE [  0.889 s]
[INFO] Apache Hadoop OpenStack support .................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
[INFO] Apache Hadoop Azure support ........................ SKIPPED
[INFO] Apache Hadoop Client ............................... SKIPPED
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 07:32 min
[INFO] Finished at: 2016-08-24T20:25:04+08:00
[INFO] Final Memory: 114M/865M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-pipes: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec failonerror="true" dir="/Users/huug/Documents/Pro/hadoop-2.7.3/src/hadoop-tools/hadoop-pipes/target/native" executable="cmake">... @ 5:143 in /Users/huug/Documents/Pro/hadoop-2.7.3/src/hadoop-tools/hadoop-pipes/target/antrun/build-main.xml

首次编译长达近一个小时的等待，等到了BUILD FAILURE，除了能看到编译hadoop pipes时失败和几行简单的fail trace，完全看不到是因为什么失败，于是在编译命令加上-X打开debug模式，重新编译后，看到了具体的fail trace，如下：

1
2
3
4
5
[exec] CMake Error at /usr/local/Cellar/cmake/3.6.1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:148 (message):
[exec] Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
[exec] system variable OPENSSL_ROOT_DIR (missing: OPENSSL_INCLUDE_DIR)
......
[exec] -- Configuring incomplete, errors occurred!

可以看到fail trace第二行Could NOT find OpenSSL，编译时没有找到openssl，第三行missing: OPENSSL_INCLUDE_DIR 没有找到名为OPENSSL_INCLUDE_DIR环境变量，貌似问题是没有安装openssl以及没有设置环境变量，但实际上在编译前，我已经安装好，并配置了环境变量，所以问题不是表面这样，于是折腾从openssl开始……

编译hadoop所需环境

可以从官网查看编译说明，也可以从下载的hadoop源码文件BUILDING.txt查看编译所需环境。

1
2
3
huugdeMacBook-Pro:src huug$ ls
BUILDING.txt             hadoop-client            
......

BUILD.txt的Requirements内容如下：

1
2
3
4
5
6
7
8
9
10
11
Requirements:
* Unix System
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
* Zlib devel (if compiling native code)
* openssl devel ( if compiling native hadoop-pipes and to get the best HDFS encryption performance )
* Linux FUSE (Filesystem in Userspace) version 2.6 or above ( if compiling fuse_dfs )
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

以上是编译hadoop所需环境，自行安装配置好相应环境变量后就可以编译了，编译指令文档中也有说明：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Building distributions:
Create binary distribution without native code and without documentation:
  $ mvn package -Pdist -DskipTests -Dtar
Create binary distribution with native code and with documentation:
  $ mvn package -Pdist,native,docs -DskipTests -Dtar
Create source distribution:
  $ mvn package -Psrc -DskipTests
Create source and binary distributions with native code and documentation:
  $ mvn package -Pdist,native,docs,src -DskipTests -Dtar
Create a local staging version of the website (in /tmp/hadoop-site)
  $ mvn clean site; mvn site:stage -DstagingDirectory=/tmp/hadoop-site

好，下面来说说我编译时遇到的问题-openssl。

openssl的问题

系统自带的openssl不仅版本低，希望可以替换成通过brew安装的版本openssl-1.0.2j,尝试了几种方式来替换系统的openssl。

修改系统环境变量

在配置编译环境时，已经在用户环境变量~/.bash_file中添加了OPENSSL_ROOT_DIR和OPENSSL_INCLUDE_DIR，但事实证明编译时还是用的系统自带的openssl，那么是不是可以直接向系统环境变量里添加环境变量，覆盖掉原来系统的版本？重新export环境变量,指定新版openssl路径后，查看openssl版本，仍然是系统的openssl版本。

1
2
openssl version
OpenSSL 0.9.8zh 14 Jan 2016

由此猜测，openssl不是这么简单的引用/usr/bin下的openssl。
那是否可以简单粗暴点。

替换系统版本

既然不能直接使用，替换成新版本就成了。于是cd到系统自带openssl路径下，rm掉，结果提示没有权限，那将老版本移到别的目录，依然提示没有权限，而且将新版本在copy到该路径下，还是提示没有权限，忽然想到我使用的就是管理员权限的账户，为什么会没有权限！Google之，有了答案，OS在升级到10.11的El Captain后引入的一个叫做系统完整性保护System Integrity Protection的系统安全功能，对系统足够高的保护，就算拥有管理员权限，修改系统级的东西，依然很费劲。那岂不是没法使用openssl的新版本了？办法是有的，就是禁用SIP功能。

1
2
3
1 重启系统，启动时同时按下cmd+r进入Recovery模式
2 出现 OS X Utilities后，选择Utilities下拉菜单中的Terminal
3 在Terminal窗口中输入[csrutil disable; reboot]命令

系统自动重启后，就没有SIP功能。在terminal中将老版本备份到其他文件夹，建立软连接

1
ln -s /usr/local/Cellar/openssl/1.0.2j/bin/openssl /usr/bin/openssl

OK，这是一种方式，有点麻烦。

⚠️为了避免禁用SIP后，系统遭到恶意控制，最好开启SIP，还是重启进入到terminal输入如下命令：

1
csrutil enable; reboot

直接创建软连接

1
ln -s /usr/local/Cellar/openssl/1.0.2j/bin/openssl /usr/local/bin/openssl

这是最最最简单的方式。

重新编译

经过折腾，重新编译，终于成功了

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [  1.330 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  1.465 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  1.051 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  2.267 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.220 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  1.671 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  3.453 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  4.249 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [  4.777 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  3.036 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:31 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  4.881 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 10.164 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.072 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [02:20 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 14.938 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [  4.591 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  3.414 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.055 s]
[INFO] hadoop-yarn ........................................ SUCCESS [  0.054 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [ 26.163 s]
[INFO] hadoop-yarn-common ................................. SUCCESS [ 20.439 s]
[INFO] hadoop-yarn-server ................................. SUCCESS [  0.042 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [  7.460 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 10.403 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  2.524 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [  5.036 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 13.076 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [  4.021 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [  4.238 s]
[INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [  2.608 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [  0.039 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  1.844 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  1.614 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [  0.032 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [  3.584 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [  4.134 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.111 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 12.934 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 11.902 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  2.567 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [  5.716 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [  3.823 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [  3.930 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  2.051 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  5.279 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [  2.357 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  3.158 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  6.317 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.519 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  3.583 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  2.672 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.687 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  1.464 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  1.900 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  7.404 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  3.814 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [15:34 min]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 23.893 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [  6.736 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.884 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  4.398 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [  9.149 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.039 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:30 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 25:56 min
[INFO] Finished at: 2016-08-25T23:22:42+08:00
[INFO] Final Memory: 131M/906M
[INFO] ------------------------------------------------------------------------

编译结果在hadoop的src/hadoop-dist/target/下

1
2
3
4
5
6
7
8
9
huugdeMacBook-Pro:target huug$ ls
antrun                             hadoop-dist-2.7.3-test-sources.jar
classes                            hadoop-dist-2.7.3.jar
dist-layout-stitching.sh           javadoc-bundle-options
dist-tar-stitching.sh              maven-archiver
hadoop-2.7.3                       maven-shared-archive-resources
hadoop-2.7.3.tar.gz                test-classes
hadoop-dist-2.7.3-javadoc.jar      test-dir
hadoop-dist-2.7.3-sources.jar

cd到cd hadoop-2.7.3目录下，里面的东西就是从官网下载解压的hadoop文件

1
2
3
huugdeMacBook-Pro:hadoop-2.7.3 huug$ ls
LICENSE.txt README.txt  etc         lib         sbin
NOTICE.txt  bin         include     libexec     share