Install oozie-3.3.2 on Hadoop 1.1.1
After a few hours tweaking and googling, I managed to install apache oozie 3.3.2 on Hadoop 1.1.1.
The document provided in apache oozie 3.3.2 is not very clear. After some googling, I found this blog is a great guide for you install oozie 3.3.1 on Hadoop 0.23.0.
http://srikanthayalasomayajulu.blogspot.sg/2013/02/apache-oozie-331-installation-on-apache.html
If you want to install oozie on MRv2/YARN, please follow the blog post.
Here we go, we're going to install oozie 3.3.2 on Hadoop 1.1.1, which is offically supported version.
Apache oozie 3.3.2 official quick start page: http://oozie.apache.org/docs/3.3.2/DG_QuickStart.html
My local environment:
- Apache Hadoop 1.1.1 pseudo setup
- Apache oozie 3.3.2, the latest release
- Java 1.7 u25
- Apache Maven 3.0.4
Oozie server installation:
- Download apache oozie 3.3.2 via: http://www.apache.org/dyn/closer.cgi/oozie/3.3.2
- Unpack the oozie-3.3.2.tar.gz under development directory: ~/dev/oozie-3.3.2
- The following two properties are required in hadoop core-site.xml:
<!-- OOZIE --> <property> <name>hadoop.proxyuser.[OOZIE_SERVER_USER].hosts</name> <value>[OOZIE_SERVER_HOSTNAME]</value> </property> <property> <name>hadoop.proxyuser.[OOZIE_SERVER_USER].groups</name> <value>[USER_GROUPS_THAT_ALLOW_IMPERSONATION]</value> </property>
Replaced [OOZIE_SERVER_USER] with gsun , [OOZIE_SERVER_HOSTNAME] with * and [OOZIE_GROUPS_THAT_ALLOW_IMPERSONATION] with * . TIP: in this blog post, the $OOZIE_HOME refers to ~/dev/oozie-3.3.2- Download extjs-2.2.zip, don't unzip it right now.
- Building a distro
cd $OOZIE_HOME ./bin/mkdistro.sh -DskipTests
It will take a while, go get a coffee, you'll be fine.- If the build successfully, we could go to the next step. In $OOZIE_HOME a directory called *hadooplibs/* directory will be created containing the Hadoop JARs for the versions of Hadoop that the Oozie distribution supports.
MacBookPro:oozie-3.3.2 gsun$ cd hadooplibs/ MacBookPro:hadooplibs gsun$ ls hadoop-1/ hadoop-3/ hadoop-distcp-2/ hadoop-test-1/ hadoop-test-3/ target/ hadoop-2/ hadoop-distcp-1/ hadoop-distcp-3/ hadoop-test-2/ pom.xml MacBookPro:hadooplibs gsun$ tree hadoop-1/ hadoop-1/ ├── pom.xml └── target ├── archive-tmp ├── hadooplibs │ └── hadooplib-1.1.1.oozie-3.3.2 │ ├── commons-beanutils-1.7.0.jar │ ├── commons-beanutils-core-1.8.0.jar │ ├── commons-codec-1.4.jar │ ├── commons-collections-3.2.1.jar │ ├── commons-configuration-1.6.jar │ ├── commons-digester-1.8.jar │ ├── commons-el-1.0.jar │ ├── commons-io-2.1.jar │ ├── commons-lang-2.4.jar │ ├── commons-logging-1.1.jar │ ├── commons-math-2.1.jar │ ├── commons-net-1.4.1.jar │ ├── hadoop-client-1.1.1.jar │ ├── hadoop-core-1.1.1.jar │ ├── hsqldb-1.8.0.7.jar │ ├── jackson-core-asl-1.8.8.jar │ ├── jackson-mapper-asl-1.8.8.jar │ ├── log4j-1.2.16.jar │ ├── oro-2.0.8.jar │ └── xmlenc-0.52.jar ├── maven-archiver │ └── pom.properties └── oozie-hadoop-1.1.1.oozie-3.3.2.jar 5 directories, 23 files MacBookPro:hadooplibs gsun$
- Create a directory named libext/ in $OOZIE_HOME, copy all of jars inside hadoop-1 into $OOZIE_HOME/libext/
- Building with ExtJS library for Oozie Web Console. Copy the extjs-2.2.zip into $OOZIE_HOME/webapp/src/main/webapp, then unzip it.
- Run oozie-setup.sh to create an oozie.war file.
> cd $OOZIE_HOME > ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin/oozie-setup.sh -extjs $OOZIE_HOME/webapp/src/main/webapp/ext-2.2.zip
If the build success, go to the next step, or refer to apache Oozie official docs for oozie-setup.sh usage. - Copy the newly minted oozie.war file to your Tomcat deployment directory.
> cd $OOZIE_HOME > vi ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf/oozie-site.xml
Change the property below to true as shown.<property> <name>oozie.service.JPAService.create.db.schema</name> <value>true</value> <description> Creates Oozie DB. If set to true, it creates the DB schema if it does not exist. If the DB schema exists is a NOP. If set to false, it does not create the DB schema. If the DB schema does not exist it fails start up. </description> </property>
- Run below command:
> cd $OOZIE_HOME > ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin/ooziedb.sh create -sqlfile oozie.sql -run
if the command is successfully run, the SQL has been created successfully.Start the oozie server.
> cd $OOZIE_HOME ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin/oozied.sh run
- From quick start in Oozie official site, we know that all Oozie server scripts (=oozie-setup.sh=, oozied.sh , oozie-start.sh , oozie-run.sh and oozie-stop.sh ) run only under the Unix user that owns the Oozie installation directory, if necessary use sudo -u OOZIE_USER when invoking the scripts.As of Oozie 3.3.2, use of oozie-start.sh , oozie-run.sh , and oozie-stop.sh has been deprecated and will print a warning. The oozied.sh script should be used instead; passing it start , run , or stop as an argument will perform the behaviors of oozie-start.sh , oozie-run.sh , and oozie-stop.sh respectively.Check the Oozie log file ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs/oozie.log to ensure Oozie started properly.
- Using the Oozie command line tool check the status of Oozie:
- $ ./distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin/oozie admin -oozie http://[ip address of namenode]:11000/oozie -statusIn my case, the Oozie server can not start due to below exception:
java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration java.lang.NoClassDefFoundError: org/apache/hadoop/util/ReflectionUtils
- You have to copy hadoop-core.jar and commons-configuration.jar under $HADOOP_HOME/lib into $OOZIE_HOME/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie/WEB-INF/lib . You should see the oozie server up and running. Enjoy oozie!