Fluency +MongoDB搭建日志系统中的几个问题
1.运行时如下错误，
error="Unknown output plugin 'mongo'. Run 'gem search -rd fluent-plugin' to find plugins"
搜索mongodb插件
gem search -rd fluent-plugin-mongo
得到结果
*** REMOTE GEMS ***
fluent-plugin-mongo (0.6.7)
Author: Masahiro Nakagawa
Homepage: https://github.com/fluent/fluent-plugin-mongo
MongoDB plugin for Fluent event collector
安装这个插件
gem install fluent-plugin-mongo
2、运行的时候可能会得到如下提示
**Notice: C extension not loaded. This is required for optimum MongoDB Ruby driver performance.
You can install the extension as follows:
gem install bson_ext
If you continue to receive this message after installing, make sure that the
bson_ext gem is in your load path and that the bson_ext and mongo gems are of the same version.
按照上述提示执行 gem install bson_ext
3、flunt.conf
<source>
type tail
path /home/aircom/mongodb-linux-i686-2.0.6/bin/logs/1001.log
format /^*(?<message>.*)$/
#format /^(?<time>[^ ]* [^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?[^\:]*\: *(?<message>.*)$/
#time_format %b %d %H:%M:%S
tag mongo.apache
</source>
<match mongo.**>
type mongo
database apache
collection fmongo
host Server202
port 10001
flush_interval 10s
</match>
4、login primary node run command
PRIMARY> db.fmongo.find().sort({"_id":-1})
{ "_id" : ObjectId("5062d3700ce0ea43fc000387"), "message" : "Wed Sep 26 18:05:28 [initandlisten] connection accepted from 192.168.19.201:34114 #3204", "time" : ISODate("2012-09-26T10:05:28Z") }
{ "_id" : ObjectId("5062d3700ce0ea43fc000386"), "message" : "Wed Sep 26 18:05:28 [conn3201] end connection 192.168.19.201:34108", "time" : ISODate("2012-09-26T10:05:28Z") }
{ "_id" : ObjectId("5062d3700ce0ea43fc000385"), "message" : "Wed Sep 26 18:05:28 [initandlisten] connection accepted from 192.168.19.201:34113 #3203", "time" : ISODate("2012-09-26T10:05:28Z") }
{ "_id" : ObjectId("5062d3700ce0ea43fc000384"), "message" : "Wed Sep 26 18:05:28 [conn3200] end connection 192.168.19.201:34107",
5、fluentd 配置正常时的启动日志
fluentd -c fluent.conf -vv &
[1] 15847
[root@Server202 fluentd-0.10.6]# 2012-09-26 16:28:26 +0800: fluent/supervisor.rb:153:supervise: starting fluentd-0.10.25
2012-09-26 16:28:26 +0800: fluent/supervisor.rb:235:read_config: reading config file path="fluent.conf"
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered buffer plugin 'file'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered buffer plugin 'memory'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered input plugin 'exec'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered input plugin 'forward'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered input plugin 'gc_stat'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered input plugin 'http'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered input plugin 'object_space'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered input plugin 'status'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered input plugin 'tcp'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered input plugin 'unix'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered input plugin 'syslog'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered input plugin 'tail'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered output plugin 'copy'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered output plugin 'exec'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered output plugin 'exec_filter'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered output plugin 'file'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered output plugin 'forward'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered output plugin 'null'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered output plugin 'roundrobin'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered output plugin 'stdout'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered output plugin 'tcp'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered output plugin 'unix'
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered output plugin 'test'
2012-09-26 16:28:26 +0800: fluent/engine.rb:63:block in configure: adding source type="forward"
2012-09-26 16:28:26 +0800: fluent/engine.rb:63:block in configure: adding source type="http"
2012-09-26 16:28:26 +0800: fluent/engine.rb:63:block in configure: adding source type="tail"
2012-09-26 16:28:26 +0800: plugin/in_tail.rb:49:configure: 'pos_file PATH' parameter is not set to a 'tail' source.
2012-09-26 16:28:26 +0800: plugin/in_tail.rb:50:configure: this parameter is highly recommended to save the position to resume tailing.
2012-09-26 16:28:26 +0800: fluent/engine.rb:79:block in configure: adding match pattern="mongo.**" type="mongo"
2012-09-26 16:28:26 +0800: fluent/plugin.rb:85:register_impl: registered output plugin 'mongo'
2012-09-26 16:28:27 +0800: plugin/out_mongo.rb:70:configure: Setup mongo configuration: mode = normal
2012-09-26 16:28:27 +0800: fluent/engine.rb:79:block in configure: adding match pattern="debug.**" type="stdout"
2012-09-26 16:28:27 +0800: plugin/in_forward.rb:60:listen: listening fluent socket on 0.0.0.0:24224
2012-09-26 16:28:27 +0800: plugin/in_http.rb:74:start: listening http on 0.0.0.0:8888
2012-09-26 16:28:27 +0800: plugin/in_tail.rb:262:initialize: following tail of /home/aircom/mongodb-linux-i686-2.0.6/bin/logs/1001.log
2012-09-26 16:28:49 +0800: fluent/parser.rb:38:call: pattern not match: Wed Sep 26 16:28:49 [FileAllocator] allocating new datafile /home/aircom/mongodb-linux-i686-2.0.6/bin/db/apache.ns, filling with zeroes...
2012-09-26 16:28:49 +0800: fluent/parser.rb:38:call: pattern not match: Wed Sep 26 16:28:49 [FileAllocator] creating directory /home/aircom/mongodb-linux-i686-2.0.6/bin/db/_tmp
2012-09-26 16:28:49 +0800: fluent/parser.rb:38:call: pattern not match: Wed Sep 26 16:28:49 [FileAllocator] allocating new datafile /home/aircom/mongodb-linux-i686-2.0.6/bin/db/apache.0, filling with zeroes...
2012-09-26 16:28:50 +0800: fluent/parser.rb:38:call: pattern not match: Wed Sep 26 16:28:50 [FileAllocator] allocating new datafile /home/aircom/mongodb-linux-i686-2.0.6/bin/db/apache.1, filling with zeroes...
2012-09-26 16:28:50 +0800: fluent/parser.rb:38:call: pattern not match: Wed Sep 26 16:28:50 [conn2616] build index done 0 records 0.001 secs