菜鸟请教logstash与ElasticSearch使用问题

Question

想请教大家，如何使用logstash 跟 elasticSearch，我看官网上流程有点混乱，不知如何将logstash跟elasticSearch结合，整个流程是如何配置的？？

官网上也没有说ElasticSearch是如何安装的，然后可以在Logstash中使用

Answer 1

经测试elasticsearch1.1.1 logstash1.4.2 redis2.8.13 不会有兼容性问题

Answer 2

redis client 连接上去..能看到logstash put 过来数据.. ------ 我的问题 解决了..是因为 logstash 版本与elasticsearch版本不兼容...

Answer 3

请问你怎么知道logstash 收集日志输出到redis ...没问题？谢谢

Answer 4

output换成elasticsearch_http试试

Answer 5

回复
我的问题 解决了..是因为 logstash 版本与elasticsearch版本不兼容...

Answer 6

回复
版本各是多少，我换了几个版本还是出现问题

Answer 7

搭车问个问题 ..

logstash 收集日志输出到redis ...没问题 .

logstash 输出到 elasticsearch 时报错..

[root@localhost logstash]# java -jar logstash-1.3.3-flatjar.jar agent -f indexer.conf 
You are using a deprecated config setting "format" set in redis. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. You should use the newer 'codec' setting instead. If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"format", :plugin=><LogStash::Inputs::Redis --->, :level=>:warn}
Using milestone 2 input plugin 'redis'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.3/plugin-milestones {:level=>:warn}
Using milestone 2 codec plugin 'oldlogstashjson'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.3/plugin-milestones {:level=>:warn}
log4j, [2014-03-21T13:20:20.465]  WARN: org.elasticsearch.discovery.zen.ping.unicast: [Psycho-Man] failed to send ping to [[#zen_unicast_1#][inet[/192.168.58.11:9300]]]
org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream
Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream
	at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:169)
	at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:123)
	at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
	at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
	at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
	at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
	at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
	at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
	at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
	at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
	at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
	at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
	at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
	at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.InvalidClassException: failed to read class descriptor
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1603)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
	at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:167)
	... 23 more

indexer.conf

[root@localhost logstash]# cat indexer.conf 
input{ 
	redis{ 
		host=>"192.168.58.11" 
		type=>"redis-input" 
		data_type=>"list" 
		key=>"logstash" 
		format=>"json_event" 
	} 
} 
output{ 
	stdout {
		debug => true
	}
	elasticsearch{ 
		host=>"192.168.58.11" 
	}	 
}