spark tachyon 搭建 配置
spark1.5.1 支持 tachyon0.7.1
jdk需要1.7
1.spark
下载spark source
http://spark.apache.org/downloads.html
编译spark
export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m"
mvn -Dhadoop.version=2.3.0 -DskipTests clean package
spark-env.sh
因为需要访问hdfs,hive,所以需要压缩lzo,和mysql
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/data/hadoop/hadoop-2.3.0-cdh5.1.0/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar export SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/data/hadoop/hadoop-2.3.0-cdh5.1.0/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar export SPARK_CLASSPATH=$SPARK_CLASSPATH:/data/hadoop/spark/spark-1.5.1-bin-hadoop2.3/lib/mysql-connector-java-5.1.20-bin.jar export SPARK_PID_DIR=/data/hadoop/spark/spark-1.5.1-bin-hadoop2.3
写slave文件
host136 host137 host138
添加配置文件到conf里:
core-site.xml
hdfs-site.xml
hive-site.xml
修改hive-site.xml
<property> <name>hive.metastore.warehouse.dir</name> <value>tachyon://host136:19998/warehouse</value> </property>
2.tachyon
下载tachyon,http://www.tachyon-project.org/
编译tachyon
mvn clean package -Djava.version=1.7 -Dhadoop.version=2.3.0 -DskipTests
修改配置:tachyon-env.sh
export TACHYON_RAM_FOLDER=/data/hadoop/spark/tachyon-0.7.1-2.3/data //内存数据mount目录 fi if [ -z "$JAVA_HOME" ]; then export JAVA_HOME="$(dirname $(which java))/.." fi export JAVA="${JAVA_HOME}/bin/java" export TACHYON_MASTER_ADDRESS=host136 #hdfs export TACHYON_UNDERFS_ADDRESS=hdfs://mycluster #-Dtachyon.master.journal.folder=hdfs://mycluster/tachyon/journal/ #export TACHYON_UNDERFS_ADDRESS=${TACHYON_UNDERFS_ADDRESS:-hdfs://localhost:9000} #使用内存大小 export TACHYON_WORKER_MEMORY_SIZE=20GB export TACHYON_UNDERFS_HDFS_IMPL=org.apache.hadoop.hdfs.DistributedFileSystem export TACHYON_WORKER_MAX_WORKER_THREADS=2048 export TACHYON_MASTER_MAX_WORKER_THREADS=2048 export TACHYON_SSH_FOREGROUND="yes" export TACHYON_WORKER_SLEEP="0.02"
可以配置多level数据
export TACHYON_JAVA_OPTS+=" -Dlog4j.configuration=file:$CONF_DIR/log4j.properties -Dtachyon.debug=false -Dtachyon.worker.tieredstore.level.max=1 -Dtachyon.worker.tieredstore.level0.alias=MEM -Dtachyon.worker.tieredstore.level0.dirs.path=$TACHYON_RAM_FOLDER -Dtachyon.worker.tieredstore.level0.dirs.quota=$TACHYON_WORKER_MEMORY_SIZE -Dtachyon.underfs.address=$TACHYON_UNDERFS_ADDRESS -Dtachyon.underfs.hdfs.impl=$TACHYON_UNDERFS_HDFS_IMPL -Dtachyon.data.folder=$TACHYON_UNDERFS_ADDRESS/tmp/tachyon/data -Dtachyon.worker.max.worker.threads=$TACHYON_WORKER_MAX_WORKER_THREADS -Dtachyon.workers.folder=$TACHYON_UNDERFS_ADDRESS/tmp/tachyon/workers -Dtachyon.worker.memory.size=$TACHYON_WORKER_MEMORY_SIZE -Dtachyon.worker.data.folder=/tachyonworker/ -Dtachyon.master.max.worker.threads=$TACHYON_MASTER_MAX_WORKER_THREADS -Dtachyon.master.worker.timeout.ms=60000 -Dtachyon.master.hostname=$TACHYON_MASTER_ADDRESS -Dtachyon.master.journal.folder=$TACHYON_HOME/journal/ -Dorg.apache.jasper.compiler.disablejsr199=true -Djava.net.preferIPv4Stack=true "
写slaves,workers
host136 host137 host138
添加配置文件到conf:
hdfs-site.xml
core-site.xml
在core-site.xml中添加
<property> <name>fs.tachyon.impl</name> <value>tachyon.hadoop.TFS</value> </property> <property> <name>fs.tachyon-ft.impl</name> <value>tachyon.hadoop.TFSFT</value> </property>
3启动服务:
tachyon:
需要用户有sudo权限,或是用root执行tachyon的mount操作
如果不想每次启动Tachyon都挂载一次RamFS,可以先使用命令 bin/tachyon-mount.sh Mount workers 或 bin/tachyon-mount.sh SudoMount workers 挂载好所有RamFS,然后使用 bin/tachyon-start.sh all NoMount 命令启动Tachyon。 bin/tachyon format bin/tachyon-mount.sh SudoMount workers bin/tachyon-start.sh all NoMount bin/tachyon-stop.sh
当添加一个tachyon节点
bin/tachyon formatWorker bin/tachyon-mount.sh SudoMount local bin/tachyon-start.sh worker NoMount
spark:
bin/spark-sql --master spark://host136:7077 bin/spark-shell --master spark://host136:7077
4.spark操作tachyon
用spark直接通过tachyon读取hdfs数据:
var textFile = sc.textFile("tachyon://host136:19998/hdfspath");
用spark在tachyon上建hive表:
CREATE EXTERNAL TABLE `test4`( `aaa` string, `ccc` string, `bbb` string) PARTITIONED BY ( `day_id` string, `hour_id` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'tachyon://host136:19998/tmp/storm2'
页面:
http://tachyonmasterIP:19999/home
http://sparkIP:8099/