Questions tagged [logstash]

Logstash is a tool for managing events and logs. You can use it to collect logs, parse them and send them to storage for later use (such as searching).

0
votes
0answers
10 views

how to index two mongodb collections in Elasticsearch

I have two mongodb collections, Transactions and Users,this is transaction example { "_id" : ObjectId("5cdd391e1e4b8f0cb8e17d0f"), "txId" : "...
1
vote
0answers
14 views

Avoid joins by multiple select statements - Logstash

I'm using Logstash to migrate data from mysql to elasticsearch. My mysql database has a primary table called product that has many relations the query to select it contains around 46 left outer join ...
0
votes
1answer
17 views

Sync MongoDB with ElasticSearch [Losgtach]

I want to sync my MongoDB data to ElasticSearch, I read a lot of posts talking about elasticsearch river plugin and mongo connector, but all of them are deprecated for mongo 4 and elasticsearch 7! As ...
1
vote
0answers
12 views

Not geeting the desired output in logstash

i am not able to get any output on the command prompt screen E:\kibana\logstash-7.1.1\logstash-7.1.1>bin\logstash -f E:\kibana\logstash-7.1.1\logstash-7.1.1\config\pipeline.conf --config.reload....
1
vote
1answer
18 views

How to set kibana index pattern from filebeat?

I am using elk stack with a node application. I am sending logs from host to logstash with filebeat, logsstash formats and send data to elastic and kibana reads from elastic. In kibana i see default ...
0
votes
1answer
13 views

Getting _jsonparsefailure with valid json in logstash

I currently and attempting to send a json through logstash to elasticsearch. However, I am getting an error that there is a jsonparsefailure even though I have validated my json as being in the proper ...
0
votes
0answers
17 views

How to put a json file into elastic search using logstash

I currently and attempting to place a json file into elastic-search using logstash. Im not sure if I am doing it correctly and I am not really sure how to check my elastic-search instance to make sure ...
0
votes
1answer
12 views

Postfix Logs + Logstash + Aggregate

I'm having trouble configuring Logstash properly. There are two lines in the postfix logs that I care about: Jun 14 09:06:22 devmailforwarder postfix/smtp[1994]: A03CA9F532: to=<[email protected]&...
1
vote
1answer
17 views

Is there a way to find out if load on Elastic stack is growing?

I have just started learning Elastic stack and I already have to diagnose production issue. Our setup from time to time has problems with pulling messages from ActiveMq to Elastic Search using ...
0
votes
2answers
56 views

Java stack trace regex

I need to match Java stack trace if it is a stack trace like this one Exception in thread "main" java.lang.IllegalStateException: A book has a null property at com.example.myproject.Author....
1
vote
1answer
19 views

Ignoring the 'pipelines.yml' file because modules or command line options are specified

I have set up elasticsearch with password protected, and i am successfully able to work with elastic search by entering username=elastic and password=mypassword but now I am trying to import mysql ...
0
votes
0answers
11 views

Receive SIGTERM on logstash startup version 7.1.1

I am trying to use logstash to send data from kafka to s3 via logstash, and I am getting an SIGTERM in the logstash process with no apparent error messages. I am using the following helm template ...
0
votes
2answers
22 views

grok parse optional field pattern doesn't work

I've got a log like this: ERROR_MESSAGE:Invalid Credentials,THROTTLED_OUT_REASON:API_LIMIT_EXCEEDED I'm trying to parse it with grok using grok debugger: ERROR_MESSAGE:%{GREEDYDATA:errorMassage},...
0
votes
1answer
14 views

How to escape keys[ ] while filtering JSON fields using Logstash?

I'm trying to configure Logstash to write some JSON fields into a database. Source JSON is as follows: "latencies":{"request":0,"kong":0,"proxy":-1},"service":{"host":"127.0.0.1","created_at":...
0
votes
0answers
27 views

how to denormalize mongo data for elasticsearch and how to push updates to ES

My first architechture was mongodb => logstash=> elasticsearch I then realized that the mongodb input logstash plugin i was using does not push updates, only new inserts with new $oid So, I've ...
0
votes
0answers
19 views

kibana - how to search unique urls

In kibana, I have below type of log entries in the "message" field. I want to search for all the unique/distinct URLs from the log. My URL format is like web.cluster.test.com/api/* 123.456.78.90 - ...
1
vote
1answer
53 views

Facing issue while decrypting logstash encrypted data in ES

I'm encrypting some data using logstash cipher filter & store the encrypted data in elasticsearch. The code snippet used for the encryption is as follow: filter { cipher { algorithm => "aes-...
-1
votes
0answers
18 views

How to parse the message field in ingress-nginx from kubernetes with splitting into an error code, etc.?

There are kubernetes. It has ingress-nginx, which logs nginx logs. Installed filebeat. But it does not parse the message field. How to parse the message field in ingress-nginx from kubernetes with ...
0
votes
0answers
23 views

logstash configuration pipeline

I have a log file looking like this 116.50.181.5 - - [18/May/2015:19:05:32 +0000] "GET /images/web/2009/banner.png HTTP/1.1" 200 52315 "http://www.semicomplete.com/style2.css" "Mozilla/5.0 (X11; ...
0
votes
1answer
29 views

ILM using Logstash Elasticsearch output plugin doesn't work

I'm trying to implement ILM for an index to properly use hardware, using the Elasticsearch output plugin. Looks like I misunderstand how Logstash manages ILM. I have ELK stack version 7.1.0 in docker....
0
votes
1answer
39 views

Elastic Stack - REST API logging with full JSON request and response

Background We have a web server written in Java that communicates with thousands of mobile apps via HTTPS REST APIs. For investigation purposes we have to log all API calls - currently this is ...
1
vote
0answers
20 views

Filter text based on two-and-conditions for an if-condition for logstash configuration

I have a log file. The log file contains many db-exceptions . I am interested in fetching all exceptions using logstash and index in ElasticSearch Exception : 05-06-19 05:34:42.794 6a5ba226-da6a-...
0
votes
0answers
11 views

How to disable specific logstash log output using log4j.properties?

I am seeing a ton of org.logstash.beats.BeatHandler logs in logstash's debug out, but I would love to disable that so I can better debug our pipeline. How would I go about that in the log4j....
0
votes
0answers
11 views

Plot cumulative delta using ELK

We have 2 sample log lines as below: [LOGSTASH]time=May 28,2019 07:41:20;logging-level=INFO;request-id=xxxxxxx;task-type=xxxxxxxxx; [LOGSTASH]time=May 28,2019 07:41:20;logging-level=INFO;response-...
0
votes
1answer
14 views

Does logstash downloads logs from s3 or it reads without downloading?

I'm using logstash to process logs to our centralized logging and the inputs are at s3 in gz format. I need to create a cost projection regarding this process and does logstash download the s3 object ...
0
votes
1answer
21 views

Need to map certain fields from the log

I received an exercise: I have a log examples "Critical Machine_5 I have a really severe issue 42" "Medium Machine_3 everything is fine 244" I need to apply Grok patterns in order to map 4 fields "...
0
votes
0answers
14 views

How do I import log4j logs from an application onto Kibana via Filebeat and Logstash?

Trying to import logs from an application to Kibana so that I can filter through them. I've tried importing logs onto Filebeat and send those to Logstash, and then to Elasticsearch to finally have ...
0
votes
1answer
21 views

Logstash not populating ElasticSearch from MySQL

I'm currently in the early stages of setting up some Kibana dashboards using the ELK stack and a MySQL database. According to the logstash config check utility, I have a valid .conf file, but nothing ...
0
votes
0answers
19 views

How to use ssl_tcp to send log from jboss to logstash

I have an ELK stack/docker on a server and another server with my app using wildlfy/docker who sent logs to ELK so I need to ensure that the connection is using ssl between the 2 servers (My app -> ...
0
votes
0answers
27 views

Get JSON substring from log message

I have log messages like this message:2019-06-07 15:49:10.110 INFO 5632 --- [http-nio-9026-exec-1] *****************.jdbc.audit.AuditHandler : AUDITHANDLER:{"USER":"0c27e6ef- b615-4328-...
0
votes
1answer
14 views

How to turn off pre-check of how many rows are in the resultset in logstash output

I'm trying to turn off the pre-select logstash does to determine the count of rows, but ExaSol DB does not support any limits in any aggregation, is there any way to turn it off in logstash? input { ...
0
votes
1answer
22 views

Logstash output to Iothub using https

Any one using IoT hub as an output in logstash. I know that there is a plugin logstash-output-iothub plugin to leverage logstash to ship to IoT hub. output { iothub { connection_string => "{...
0
votes
0answers
12 views

Extended Stats Bucket not returning sum of bucket - Using Top Hits Aggregation

I am struggling to use a "Top Hits" aggregation and "extended_stats_bucket". From the buckets returned my "extended_stats_bucket" path cannot read the metric value count_status. Is this is a draw ...
0
votes
1answer
31 views

Remove parameters from SQL Query in with Regex / LogStash

A 3rd party system I use logs all SQL queries along with rowcount & response time which I then send to Logstash/Elastic to calculate metrics. As this system doesn't use bind variables, and there ...
0
votes
0answers
31 views

Configuring the new logstash version - issues on the output Elasticsearch template for the mapping of my logs

I'm trying to configure the new version logstash (7.x) Everything is working as prefect, logstash is running and I can see my logs on kibana :slight_smile: . BUT the mapping/parsing elasticsearch isn'...
0
votes
0answers
16 views

Setup infrastructure with ELK and Filebeat in docker

I'm quite a newbie in ELK and I'm struggling to understand what is the best way to organize my logs shipping to logstash. What I have: Many REST API services running Many Services that perform a ...
0
votes
1answer
12 views

file data from start not passed to Elasticsearch

I have an existing log file and I am setting up ELK stack now and I am able to connect everything together and anything I add does reflect in Elasticsearch, But data that was already there in the log ...
1
vote
0answers
17 views

Logstash Error: (GemNotFound) Could not find gem 'logstash-core-plugin-api'

I have installed logstash using NSSM in my windows environment and when I start the service I get the following error in the output log: D:\Logstash\bin>logstash.bat agent -f logstash.conf [ERROR]...
1
vote
0answers
15 views

Multiple translate filters on the same event field in Logstash

I am using the following translate filters in my logstash configuration file. I have two separate YAML files that serve as the lookup dictionaries. The logstash event field referenced is the same for ...
0
votes
0answers
21 views

Multiline.pattern for xml file in filebeat

What would be the multiline pattern to print all the lines under for the below xml in filebeat <?xml-stylesheet alternate="yes" href="file://c:/drive/bin/event_log.xsl" type="text/xsl"?> <...
0
votes
0answers
19 views

Logstash pull server date as config variable

As part of my logstash config I want to pull the current date from the server, which it uses as part of it's API query using http_poller. Is there any way to do that? I've tried something along the ...
0
votes
1answer
37 views

Remote data store processing with ElasticSearch 7.1 and log4j2.11.1

I am using ElasticSearch 7.1. It comes with log4j2.11.1.jar. The problem comes when I am trying to setup a remote data store with log4j2 running as a TcpSocketServer. I would then use log4j logging ...
0
votes
1answer
61 views

What is the best way to send and display logs for easy access and monitoring?

I would love to know the best way to display logs for a system of mine. I receive logs once a day in bulk (the logs are not needed in real time) and I was wondering, what is the most efficient way to ...
0
votes
1answer
33 views

How can I filter a keyword in logstash message field in new field

Can I filter key word in message field? For example, this is log: " session opened for user root by (uid=0)"--> And I want create a new field to keep data like "root", I wanna filter user name when ...
0
votes
0answers
43 views

multiline pattern for nested xml files in filebeat - logstash 7.0.1

I was trying to parse a typical xml log file from my windows machine using filebeat to logstash. below is my xml example. <?xml-stylesheet alternate="yes" href="file://c:/drive/bin/event_log.xsl" ...
0
votes
0answers
5 views

logstash: remove plugins completely

The logstash command logstash-plugin remove <plugin name> only disables the plugin. The code/binaries/dependencies remain in the installation. How to completely remove code/binaries/dependencies ...
0
votes
0answers
46 views

Elasticsearch crashes after a while with 'all shards failed' message

I have an application that is sending a maximum amount of 600 log messages every 2 minutes in packages of 100 (sends 100 logs, 2 seconds pause...) This app is sending the logs to Logstash -> ...
0
votes
0answers
19 views

Import Heartbeat Dashboard is trying to connect Elasticsearch client on http://127.0.0.1:9200

I am trying to import Heartbeat Dashboard. Steps performed: Installed Elasticsearch and Kibana on Host1. This is working Fine. Installed Heartbeat5.6 on Host2 . Heartbeat monitoring is working fine. ...
0
votes
0answers
28 views

Compare ElasticSearch index data with my database records

I am streaming messages from the source into ElasticSearch, then process data with Logstash and want to compare keywords with my own database records. My question is how to compare keywords from ...
0
votes
0answers
16 views

ELK : One or more required cgroup files or directories not found: /proc/self/cgroup

I have configured the logstash configuration as below to read the logs generated by spring boot to push into elastic search, but index is not creating even the logstash started successfully, Conf ...