Not receiving logs after fresh install.

[Johnny111125]

I just ran through the install step by step, however it looks like I am not receiving logs. Dashboards are there, just not getting any info. Please help pinpoint where this is failing. Thanks in advance!

System Specs...
16 gig ram
4 cores
200 gb
Ubuntu Server 22.04.4

[a3ilson]

Can you provide logs?

[Johnny111125]

Which logs would you like me to post?

[a3ilson]

Check and provide logstash logs.

[Johnny111125]

Here is the log. Thank you!

logstash-plain-2024-08-22-1.log.gz

[a3ilson]

Thanks! Currently traveling and will take a look this weekend. If you tail the last 15 logs and paste here, we should be able to determine further troubleshooting steps.

[a3ilson]

@Johnny111125 - appears to be invalid credentials based on the provided logs. Please confirm you updated/configured the credentials within step i10a

[Johnny111125]

That was certainly an issue. I corrected it, but am still not seeing anything on the dashboards.

Here is more data from the log. Let me know if you want the entire file. Thanks!

[2024-08-26T00:54:10,830][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-26T00:54:11,149][INFO ][logstash.filters.geoip.databasemanager][pfelk] By not manually configuring a database path with database =>, you accepted and agreed MaxMind EULA. For more details please visit https://www.maxmind.com/en/geolite2/eula
[2024-08-26T00:54:11,150][INFO ][logstash.filters.geoip ][pfelk] Using geoip database {:path=>"/var/lib/logstash/geoip_database_management/1724633059/GeoLite2-City.mmdb"}
[2024-08-26T00:54:11,168][INFO ][logstash.filters.geoip.databasemanager][pfelk] By not manually configuring a database path with database =>, you accepted and agreed MaxMind EULA. For more details please visit https://www.maxmind.com/en/geolite2/eula
[2024-08-26T00:54:11,169][INFO ][logstash.filters.geoip ][pfelk] Using geoip database {:path=>"/var/lib/logstash/geoip_database_management/1724633059/GeoLite2-ASN.mmdb"}
[2024-08-26T00:54:11,171][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-26T00:54:11,218][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-26T00:54:11,265][INFO ][logstash.filters.json ][pfelk] ECS compatibility is enabled but target option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the target option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2024-08-26T00:54:11,266][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-26T00:54:11,324][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-26T00:54:11,363][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-26T00:54:11,393][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-26T00:54:11,468][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-26T00:54:11,495][INFO ][logstash.filters.geoip.databasemanager][pfelk] By not manually configuring a database path with database =>, you accepted and agreed MaxMind EULA. For more details please visit https://www.maxmind.com/en/geolite2/eula
[2024-08-26T00:54:11,496][INFO ][logstash.filters.geoip ][pfelk] Using geoip database {:path=>"/var/lib/logstash/geoip_database_management/1724633059/GeoLite2-City.mmdb"}
[2024-08-26T00:54:11,497][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-26T00:54:11,519][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-26T00:54:11,546][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-26T00:54:11,573][ERROR][logstash.javapipeline ][pfelk] Pipeline error {:pipeline_id=>"pfelk", :exception=>#<Grok::PatternError: pattern %{OVPN_PINFO4} not defined>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in block in compile'", "org/jruby/RubyKernel.java:1722:in loop'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in compile'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-grok-4.4.3/lib/logstash/filters/grok.rb:282:in block in register'", "org/jruby/RubyArray.java:1981:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-grok-4.4.3/lib/logstash/filters/grok.rb:276:in block in register'", "org/jruby/RubyHash.java:1610:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-grok-4.4.3/lib/logstash/filters/grok.rb:271:in register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:237:in block in register_plugins'", "org/jruby/RubyArray.java:1981:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:236:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:611:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:249:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:194:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146:in block in start'"], "pipeline.sources"=>["/etc/pfelk/conf.d/01-inputs.pfelk", "/etc/pfelk/conf.d/02-firewall.pfelk", "/etc/pfelk/conf.d/05-apps.pfelk", "/etc/pfelk/conf.d/30-geoip.pfelk", "/etc/pfelk/conf.d/49-cleanup.pfelk", "/etc/pfelk/conf.d/50-outputs.pfelk"], :thread=>"#<Thread:0x1d4f4e93 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2024-08-26T00:54:11,574][INFO ][logstash.javapipeline ][pfelk] Pipeline terminated {"pipeline.id"=>"pfelk"}
[2024-08-26T00:54:11,582][ERROR][logstash.agent ] Failed to execute action {:id=>:pfelk, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
[2024-08-26T00:54:11,591][INFO ][logstash.runner ] Logstash shut down.
[2024-08-26T00:54:11,600][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:921) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:880) ~[jruby.jar:?]
at usr.share.logstash.lib.bootstrap.environment.

(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]

[a3ilson]

Thank you and my apologizes... There was an undefined GROK pattern in the openvpn.grok file...it was corrected.

Please delete the /etc/pfelk/patterns/openvpn.grok and download the latest update:
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/patterns/openvpn.grok -P /etc/pfelk/patterns/

[Johnny111125]

Did this, but dashboard is still not populating.

Here is what the log is saying now...

[2024-08-27T00:48:15,442][WARN ][logstash.outputs.elasticsearch][pfelk] Restored connection to ES instance {:url=>"https://elastic:xxxxxx@localhost:9200/"}
[2024-08-27T00:48:15,448][INFO ][logstash.outputs.elasticsearch][pfelk] Elasticsearch version determined (8.14.3) {:es_version=>8}
[2024-08-27T00:48:15,449][WARN ][logstash.outputs.elasticsearch][pfelk] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8}
[2024-08-27T00:48:15,465][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-27T00:48:15,800][INFO ][logstash.filters.geoip.databasemanager][pfelk] By not manually configuring a database path with database =>, you accepted and agreed MaxMind EULA. For more details please visit https://www.maxmind.com/en/geolite2/eula
[2024-08-27T00:48:15,801][INFO ][logstash.filters.geoip ][pfelk] Using geoip database {:path=>"/var/lib/logstash/geoip_database_management/1724716844/GeoLite2-City.mmdb"}
[2024-08-27T00:48:15,819][INFO ][logstash.filters.geoip.databasemanager][pfelk] By not manually configuring a database path with database =>, you accepted and agreed MaxMind EULA. For more details please visit https://www.maxmind.com/en/geolite2/eula
[2024-08-27T00:48:15,820][INFO ][logstash.filters.geoip ][pfelk] Using geoip database {:path=>"/var/lib/logstash/geoip_database_management/1724716844/GeoLite2-ASN.mmdb"}
[2024-08-27T00:48:15,822][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-27T00:48:15,870][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-27T00:48:15,916][INFO ][logstash.filters.json ][pfelk] ECS compatibility is enabled but target option was not specified. This may cause fields to be set at the top-level of the event where they are likely to clash with the Elastic Common Schema. It is recommended to set the target option to avoid potential schema conflicts (if your data is ECS compliant or non-conflicting, feel free to ignore this message)
[2024-08-27T00:48:15,917][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-27T00:48:15,973][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-27T00:48:16,004][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-27T00:48:16,032][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-27T00:48:16,101][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-27T00:48:16,141][INFO ][logstash.filters.geoip.databasemanager][pfelk] By not manually configuring a database path with database =>, you accepted and agreed MaxMind EULA. For more details please visit https://www.maxmind.com/en/geolite2/eula
[2024-08-27T00:48:16,141][INFO ][logstash.filters.geoip ][pfelk] Using geoip database {:path=>"/var/lib/logstash/geoip_database_management/1724716844/GeoLite2-City.mmdb"}
[2024-08-27T00:48:16,143][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-27T00:48:16,175][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-27T00:48:16,203][WARN ][logstash.filters.grok ][pfelk] ECS v8 support is a preview of the unreleased ECS v8, and uses the v1 patterns. When Version 8 of the Elastic Common Schema becomes available, this plugin will need to be updated
[2024-08-27T00:48:16,238][ERROR][logstash.javapipeline ][pfelk] Pipeline error {:pipeline_id=>"pfelk", :exception=>#<Grok::PatternError: pattern %{OVPN_PINFO4} not defined>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in block in compile'", "org/jruby/RubyKernel.java:1722:in loop'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in compile'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-grok-4.4.3/lib/logstash/filters/grok.rb:282:in block in register'", "org/jruby/RubyArray.java:1981:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-grok-4.4.3/lib/logstash/filters/grok.rb:276:in block in register'", "org/jruby/RubyHash.java:1610:in each'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-grok-4.4.3/lib/logstash/filters/grok.rb:271:in register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:237:in block in register_plugins'", "org/jruby/RubyArray.java:1981:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:236:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:611:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:249:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:194:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146:in block in start'"], "pipeline.sources"=>["/etc/pfelk/conf.d/01-inputs.pfelk", "/etc/pfelk/conf.d/02-firewall.pfelk", "/etc/pfelk/conf.d/05-apps.pfelk", "/etc/pfelk/conf.d/30-geoip.pfelk", "/etc/pfelk/conf.d/49-cleanup.pfelk", "/etc/pfelk/conf.d/50-outputs.pfelk"], :thread=>"#<Thread:0x71a43eb6 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2024-08-27T00:48:16,239][INFO ][logstash.javapipeline ][pfelk] Pipeline terminated {"pipeline.id"=>"pfelk"}
[2024-08-27T00:48:16,251][ERROR][logstash.agent ] Failed to execute action {:id=>:pfelk, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
[2024-08-27T00:48:16,260][INFO ][logstash.runner ] Logstash shut down.
[2024-08-27T00:48:16,269][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:921) ~[jruby.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:880) ~[jruby.jar:?]
at usr.share.logstash.lib.bootstrap.environment.

(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]

[a3ilson]

@Johnny111125 - logs are showing the same issue...

Try the following:

1. stop logstash (systemctl stop logstash.service)
2. remove all files within the /etc/pfelk/patterns folder (sudo rm /etc/pfelk/patterns/*)
3. download updated patterns files

sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/patterns/pfelk.grok -P /etc/pfelk/patterns/
sudo wget https://raw.githubusercontent.com/pfelk/pfelk/main/etc/pfelk/patterns/openvpn.grok -P /etc/pfelk/patterns/

4. start logstash (systemctl start logstash.service)