Install oozie-3.3.2 on Hadoop 1.1.1

After a few hours tweaking and googling, I managed to install apache oozie 3.3.2 on Hadoop 1.1.1.

The document provided in apache oozie 3.3.2 is not very clear. After some googling, I found this blog is a great guide for you install oozie 3.3.1 on Hadoop 0.23.0.

http://srikanthayalasomayajulu.blogspot.sg/2013/02/apache-oozie-331-installation-on-apache.html

If you want to install oozie on MRv2/YARN, please follow the blog post.

Here we go, we're going to install oozie 3.3.2 on Hadoop 1.1.1, which is offically supported version.

Apache oozie 3.3.2 official quick start page: http://oozie.apache.org/docs/3.3.2/DG_QuickStart.html

My local environment:

  • Apache Hadoop 1.1.1 pseudo setup
  • Apache oozie 3.3.2, the latest release
  • Java 1.7 u25
  • Apache Maven 3.0.4

Oozie server installation:

    • Download apache oozie 3.3.2 via: http://www.apache.org/dyn/closer.cgi/oozie/3.3.2
    • Unpack the oozie-3.3.2.tar.gz under development directory: ~/dev/oozie-3.3.2
    • The following two properties are required in hadoop core-site.xml:
    • <!-- OOZIE -->
        <property>
          <name>hadoop.proxyuser.[OOZIE_SERVER_USER].hosts</name>
          <value>[OOZIE_SERVER_HOSTNAME]</value>
        </property>
        <property>
          <name>hadoop.proxyuser.[OOZIE_SERVER_USER].groups</name>
          <value>[USER_GROUPS_THAT_ALLOW_IMPERSONATION]</value>
        </property>
       Replaced [OOZIE_SERVER_USER] with gsun , [OOZIE_SERVER_HOSTNAME] with * and [OOZIE_GROUPS_THAT_ALLOW_IMPERSONATION] with *  . TIP: in this blog post, the $OOZIE_HOME refers to ~/dev/oozie-3.3.2
    • Download extjs-2.2.zip, don't unzip it right now.
    • Building a distro
    • cd $OOZIE_HOME
      ./bin/mkdistro.sh -DskipTests
       It will take a while, go get a coffee, you'll be fine.
    • If the build successfully, we could go to the next step. In $OOZIE_HOME a directory called  *hadooplibs/* directory will be created containing the Hadoop JARs for the versions of Hadoop that the Oozie distribution supports.
      MacBookPro:oozie-3.3.2 gsun$ cd hadooplibs/
      MacBookPro:hadooplibs gsun$ ls
      hadoop-1/        hadoop-3/        hadoop-distcp-2/ hadoop-test-1/   hadoop-test-3/   target/
      hadoop-2/        hadoop-distcp-1/ hadoop-distcp-3/ hadoop-test-2/   pom.xml
      MacBookPro:hadooplibs gsun$ tree hadoop-1/
      hadoop-1/
      ├── pom.xml
      └── target
          ├── archive-tmp
          ├── hadooplibs
          │   └── hadooplib-1.1.1.oozie-3.3.2
          │       ├── commons-beanutils-1.7.0.jar
          │       ├── commons-beanutils-core-1.8.0.jar
          │       ├── commons-codec-1.4.jar
          │       ├── commons-collections-3.2.1.jar
          │       ├── commons-configuration-1.6.jar
          │       ├── commons-digester-1.8.jar
          │       ├── commons-el-1.0.jar
          │       ├── commons-io-2.1.jar
          │       ├── commons-lang-2.4.jar
          │       ├── commons-logging-1.1.jar
          │       ├── commons-math-2.1.jar
          │       ├── commons-net-1.4.1.jar
          │       ├── hadoop-client-1.1.1.jar
          │       ├── hadoop-core-1.1.1.jar
          │       ├── hsqldb-1.8.0.7.jar
          │       ├── jackson-core-asl-1.8.8.jar
          │       ├── jackson-mapper-asl-1.8.8.jar
          │       ├── log4j-1.2.16.jar
          │       ├── oro-2.0.8.jar
          │       └── xmlenc-0.52.jar
          ├── maven-archiver
          │   └── pom.properties
          └── oozie-hadoop-1.1.1.oozie-3.3.2.jar
      
      5 directories, 23 files
      MacBookPro:hadooplibs gsun$
       
    • Create a directory named libext/ in $OOZIE_HOME, copy all of jars inside hadoop-1 into $OOZIE_HOME/libext/
    • Building with ExtJS library for Oozie Web Console. Copy the extjs-2.2.zip into $OOZIE_HOME/webapp/src/main/webapp, then unzip it.
    • Run oozie-setup.sh to create an oozie.war file.   
      > cd $OOZIE_HOME 
      > ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin/oozie-setup.sh  -extjs $OOZIE_HOME/webapp/src/main/webapp/ext-2.2.zip
       If the build success, go to the next step, or refer to apache Oozie official docs for oozie-setup.sh usage.
    • Copy the newly minted oozie.war file to your Tomcat deployment directory.
      > cd $OOZIE_HOME
      > vi ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf/oozie-site.xml
       Change the property below to true as shown.
      <property>
              <name>oozie.service.JPAService.create.db.schema</name>
              <value>true</value>
              <description>
                  Creates Oozie DB.
                  If set to true, it creates the DB schema if it does not exist. If the DB schema exists is a NOP.
                  If set to false, it does not create the DB schema. If the DB schema does not exist it fails start up.
              </description>
          </property>
       
    • Run below command:
      > cd $OOZIE_HOME 
      > ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin/ooziedb.sh create -sqlfile oozie.sql -run
      if the command is successfully run, the SQL has been created successfully.Start the oozie server.
> cd $OOZIE_HOME
./distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin/oozied.sh run
    •  From quick start in Oozie official site,  we know that all Oozie server scripts (=oozie-setup.sh=, oozied.sh , oozie-start.sh , oozie-run.sh and oozie-stop.sh ) run only under the Unix user that owns the Oozie installation directory, if necessary use sudo -u OOZIE_USER when invoking the scripts.As of Oozie 3.3.2, use of oozie-start.sh , oozie-run.sh , and oozie-stop.sh has been deprecated and will print a warning. The oozied.sh script should be used instead; passing it start , run , or stop as an argument will perform the behaviors of oozie-start.sh , oozie-run.sh , and oozie-stop.sh respectively.Check the Oozie log file  ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs/oozie.log to ensure Oozie started properly.
    • Using the Oozie command line tool check the status of Oozie:
    • $ ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin/oozie admin -oozie http://[ip address of namenode]:11000/oozie -statusIn my case, the Oozie server can not start due to below exception:
java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
java.lang.NoClassDefFoundError: org/apache/hadoop/util/ReflectionUtils
  •  You have to copy hadoop-core.jar and commons-configuration.jar under $HADOOP_HOME/lib into $OOZIE_HOME/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie/WEB-INF/lib . You should see the oozie server up and running. Enjoy oozie!
Another problem I met during development is that oozie thrown below exception after job submitted:

相关推荐