I am working on a solution where I have to ingest logs from a Linux server to New Relic Logs. I am using the document to configure log ingestion. There are various solutions to configure to send your logs to New Relic Logs. I choose the New Relic infrastructure agent log forwarding method, since it is listed first in the solutions. You need to have a New Relic license to forward logs to New Relic Logs.
To start with, I choose a linux server and installed New Relic agent. The New Relic infrastructure agent installation is described here. Before proceeding further, you need to verify the compatibility requirements. New Relic infrastructure agent version should be 1.11.4 or higher.
After installing the New Relic Infrastructure agent, you need to configure the log forwarding. Follow these steps to configure log forwarding. But before configuring the log forwarding, it is a good practice to test with a sample file. So I choose the following steps to test the log forwarding to New Relic logs.
Login to the linux machine and Navigate to /etc/newrelic-infra/logging.d/.
Create any file with .yml extension, say askm.yml and update the following information.
logs:- name: "test_log"file: /tmp/askm/askm.log
You don't need to restart the agent. You should be able to test it immediately.
We need to populate the logfile /tmp/askm/askm.log with some content to verify if that log is being forwarded to New Relic logs.
Use the following command to populate the file with some text.
echo "This is a test message from nrpd-askmlabs01 on `date`" >> /tmp/askm/askm.log
Now, We are ready to verify the logs in New Relic logs. Access New Relic logs and verify the logs. If there are no issues, you should be able to see the logs. But ......
I can't see logs in New Relic logs.
Where to look for logs?
NR infrastructure logs are stored in location "/var/log/newrelic-infra". Verify if you can find anything in the logs. No, Because our New Relic Infrastructure agent is installed without verbose parameter.
We need logs to understand why the Infrastructure agent is not able to forward logs to New Relic Logs.
We can enable logging using the document "Generate logs for troubleshooting Infrastructure".
All the following two entries to the /etc/newrelic-infra.yml file to enable debug.
log_file: /var/log/newrelic-infra/newrelic-infra.logverbose: 3
Now restart the New Relic infrastructure agent. Again populate the input logfile that we have created "/tmp/askm/askm.log" with some data using echo command. Make sure to disable the logging immediately after this test. Remove the log_file and verbose entries and restart the New Relic infrastructure agent again.
Analyzing the Log for issue......
At this point, you should have the log file with debug messages. Search for errors. With the same setup as described above, I observed the following error message which are noted for furthre analysis.
level=debug msg="Launching process." component=integrations.Supervisor process=log-forwarderlevel=debug msg="Running command." command=/var/db/newrelic-infra/newrelic-integrations/logging/fluent-bit component=integrations.Executor env="[TERM=linux PATH=/usr/local/sbin:/usr/local/bin:/usr/bin :/usr/sbin:/sbin:/bin PWD=<HIDDEN> SHLVL=0 UPSTART_INSTANCE= UPSTART_JOB=newrelic-infra DBUS_SYSTEM_BUS_ADDRESS=unix:path=/var/run/dbus/system_bus_socket]" path=/var/db/newrelic-infra/newrelic-integrations/logging/fluent-bitlevel=debug msg="/var/db/newrelic-infra/newrelic-integrations/logging/fluent-bit: error while loading shared libraries: libsystemd.so.0: cannot open shared object file: No such file or directory" comp onent=integrations.Supervisor output=stderr process=log-forwarderlevel=error msg="Error occurred while handling the process" component=integrations.Supervisor error="exit status 127" process=log-forwarderlevel=debug msg="Supervisor backOff." backOff duration=7.037440062s component=integrations.Supervisor process=log-forwarderlevel=debug msg="Sending events to metrics-ingest." component=MetricsIngestSender key=650267471925810761 numEvents=5 postCount=3 timestamps="[2020-07-29 11:30:02 -0400 EDT 2020-07-29 11:30:03 -0400 ED T]"
The error that caught to my attention is "error while loading shared libraries: libsystemd.so.0". Why is this happening?
I tried to locate this on linux OS using 'locate libsystemd.so.0", but nothing listed. I then started thinking why we don't have this? I tried to find, if systemmd is running on my system using "pidof systemd", I don't have this process running. Then I slowly started realizing.... What is the Linux OS that I am using. Is this OS version supported to run systemmd ?
The Linux OS that I am using is RHEL 6.9 and the fact that RHEL 6.x does not use the systemd. On this version systemd is not the init system used. Then started verifying the first basic check ( Ofcourse we will forget most of the time), compatibiliy? Missed the fact that New Relic Infrastructure agent log forwarding will work only for RHEL 7.x and later. So, my next option is to pick one of the other options available for log forwarding.
Other troubleshooting links that may help you for New Relic Logs ingestion are
Hope this information helps you. If you have any questions, please post them in the comments. I will revert as soon as I can.
Thanks
SRI
Posts
Post a Comment
Thank you for visiting our site and leaving your valuable comment.