Arch-03-19- NoSQL实践

虽然对新技术甚为谨慎，但目前大家所做的伪 sns 的 nosql 似乎也让一帮同事兴奋不已，没办法，试试吧。

不讨论nosql 的长短，直接实施吧，关键在于 nosql 表的设计和与 postgrsql 间数据的关系组织。

一、用 YCSB 测试候选的数据库

（1）候选：hbase,voldemort,mongodb

（2）ycsb 环境：centos5.4,12G,4x4核 服务器

（3）下载安装  ycsb 略过

（4）Hbase-0.9

    . 插入数据 100w

java -cp build/ycsb.jar:db/hbase/lib/* com.yahoo.ycsb.Client -load -db com.yahoo.ycsb.db.HBaseClient -P workloads/workloada -p columnfamily=family -p recordcount=10000000 -s > load.dat

10 sec: 73825 operations; 7380.29 current ops/sec; [INSERT AverageLatency(ms)=0.11]
 20 sec: 127145 operations; 5329.87 current ops/sec; [INSERT AverageLatency(ms)=0.16]
 30 sec: 217374 operations; 9020.19 current ops/sec; [INSERT AverageLatency(ms)=0.09]
...

 平均 insert 8k/sec

java -cp build/ycsb.jar:db/hbase/lib/* com.yahoo.ycsb.Client -t -db com.yahoo.ycsb.db.HBaseClient -P workloads/workloada -p columnfamily=family -p operationcount=1000000 -s -threads 10 -target 100 > transactions.dat

（5）Voldemort-0.9

    初始化插入 500w，0.7k/s

YCSB Client 0.1
Command line: -load -db com.yahoo.ycsb.db.VoldemortClient -p bootstrap_urls=tcp://localhost:6666 -P workloads/workloada -s -threads 10
[OVERALL], RunTime(ms), 655195.0
[OVERALL], Throughput(ops/sec), 7631.315867795084
[INSERT], Operations, 5000000
[INSERT], AverageLatency(ms), 1.2568772
[INSERT], MinLatency(ms), 0
[INSERT], MaxLatency(ms), 2289
[INSERT], 95thPercentileLatency(ms), 1
[INSERT], 99thPercentileLatency(ms), 3
[INSERT], Return=0, 5000000

    写/读各50w 次，写 0.83k/s,R1.2k/s

YCSB Client 0.1
Command line: -t -db com.yahoo.ycsb.db.VoldemortClient -p bootstrap_urls=tcp://localhost:6666 -P workloads/workloada -s
[OVERALL], RunTime(ms), 1026399.0
[OVERALL], Throughput(ops/sec), 974.2799827357587
[UPDATE], Operations, 501020
[UPDATE], AverageLatency(ms), 1.2046704722366373
[UPDATE], MinLatency(ms), 0
[UPDATE], MaxLatency(ms), 1887
[UPDATE], 95thPercentileLatency(ms), 5
[UPDATE], 99thPercentileLatency(ms), 8
[UPDATE], Return=0, 501020

....


[READ], Operations, 498980
[READ], AverageLatency(ms), 0.8289450478977113
[READ], MinLatency(ms), 0
[READ], MaxLatency(ms), 359
[READ], 95thPercentileLatency(ms), 5
[READ], 99thPercentileLatency(ms), 7
[READ], Return=0, 498980

（6）mongodb

    初始化插入 500w，0.1k/s

YCSB Client 0.1
Command line: -load -db com.yahoo.ycsb.db.MongoDbClient -p mongodb.database=mytestdbt100 -p mongodb.url=mongodb://localhost:27017 -P workloads/workloada -s -threads 100
new database url = localhost:27017/mytestdbt100
new database url = localhost:27017/mytestdbt100
...100行

mongo connection created with localhost:27017/mytestdbt100
mongo connection created with localhost:27017/mytestdbt100
...100行

[OVERALL], RunTime(ms), 4664799.0
[OVERALL], Throughput(ops/sec), 1071.857544129983
[INSERT], Operations, 5000000
[INSERT], AverageLatency(ms), 93.0971558
[INSERT], MinLatency(ms), 0
[INSERT], MaxLatency(ms), 34143
[INSERT], 95thPercentileLatency(ms), 220
[INSERT], Return=0, 5000000

    写/读各50w 次，写 0.13k/s,R3.7k/s

YCSB Client 0.1
Command line: -t -db com.yahoo.ycsb.db.MongoDbClient -p mongodb.database=mytestdb100 -p mongodb.url=mongodb://localhost:27017 -P workloads/workloada -s -threads 100
new database url = localhost:27017/mytestdb100
new database url = localhost:27017/mytestdb100
...100

mongo connection created with localhost:27017/mytestdb100
mongo connection created with localhost:27017/mytestdb100
...100

[OVERALL], RunTime(ms), 40395.0
[OVERALL], Throughput(ops/sec), 24755.539051862856
[UPDATE], Operations, 500052
[UPDATE], AverageLatency(ms), 7.622983209746186
[UPDATE], MinLatency(ms), 0
[UPDATE], MaxLatency(ms), 1344
[UPDATE], 95thPercentileLatency(ms), 18
[UPDATE], 99thPercentileLatency(ms), 27
[UPDATE], Return=1, 500052
...

READ], Operations, 499948
[READ], AverageLatency(ms), 0.27114819941273893
[READ], MinLatency(ms), 0
[READ], MaxLatency(ms), 1308
[READ], 95thPercentileLatency(ms), 1
[READ], 99thPercentileLatency(ms), 1
[READ], Return=1, 499948
...

（7）MongoDB Java 框架选择

• Kundera   https://github.com/impetus-opensource/Kundera-Examples  -印度人搞的，飘过。
• Morphia   http://code.google.com/p/morphia/    http://www.ibm.com/developerworks/cn/java/j-morphia/
• Spring-Data  http://www.springsource.org/spring-data/mongodb-现在的选择，基于 spring 风格，一切都很自然。跟 hibernate 有点接近，Query...

    . 参考 http://blog.csdn.net/fengzanfeng/article/details/6634433; 

安装

Mongodb: http://be-evil.org/install-mongodb-on-centos.html

然而关于 nosql 的讨论是没有休止的：

为什么说很多NoSQL的Benchmark是扯淡？ 

解读NoSQL数据库的四大家族 

微博架构与平台安全演讲稿

新浪微博在技术上是如何架构的-微博架构与平台安全

http://www.360doc.com/content/11/0710/04/1542811_132636642.shtml