[MAC] 하둡 설치

JAVA 설치

- JAVA 8

Java Archive Downloads - Java SE 8

하둡 설치

$ brew install hadoop
## 하둡의 버전과 설치된 경로를 확인
$ brew info hadoop
## 경로는 위 명령어로 확인 및 하둡 버전 X.X.X는 본인이 설치한 버전으로 수정
$ cd /usr/local/Cellar/hadoop/3.3.4/libexec/etc/hadoop

## 경로 확인
$ /usr/libexec/java_home -V
$ open hadoop-env.sh

$ open core-site.xml
<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

$ open hdfs-site.xml
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/path/to/hadoop-3.3.2/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/path/to/hadoop-3.3.2/dfs/data</value>
    </property>
</configuration>

$ open mapred-site.xml
<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.application.classpath</name>
    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
  </property>
</configuration>

$ open yarn-site.xml
<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property> <property>
    <name>yarn.nodemanager.env-whitelist</name>
    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value> </property>
</configuration>

$ ssh localhost
## 본인 하둡 경로 들어가기
$ cd /usr/local/Cellar/hadoop/3.3.4
## 파일시스템 포맷
$ hdfs namenode -format
$ cd /usr/local/Cellar/hadoop/3.3.4/libexec
$ sbin/start-all.sh

$ cd /usr/local/Cellar/hadoop/3.3.4
$ bin/hdfs dfs -mkdir /user
$ bin/hdfs dfs -mkdir /user/<username>
$ jps

하둡 실행

- Cluster status : http://localhost:8088
- HDFS status : http://localhost:9870
- Secondary NameNode status : http://localhost:9868