Top 4 Common Issues when Configuring Splunk Connect for Syslog (SC4S)
By: Greg Becker | Senior Splunk Consultant
Splunk Connect for Syslog (SC4S) is a containerized version of syslog-ng that has out-of-the-box configurations for many known Sources, and it’s also supported by Splunk. The implementation is different than a classic syslog-ng install that is typically installed on a host which receives syslog traffic and then it writes out logs to a filesystem to then be picked up by a Splunk Universal Forwarder and then sent to a Splunk indexing tier. SC4S uses HTTP Event Collector (HEC) to send events directly to the receiving Splunk instance without the need to write the logs to disk. We help customers with issues when configuring Splunk Connect for syslog (SC4S) implementations and have noticed a few common areas where missteps can occur.
Here are the top 4 issues when configuring Splunk Connect for syslog:
- 1. Read through the entire set of docs
- 2. Decide if you’ll use out-of-the-box indexes
- 3. Important HEC token settings
- 4. What to do if your Source isn’t listed
1. Read the docs
The most common issue that I see is that users don’t read the docs that are available before attempting to install and configure SC4S. There is a ton of great information out there. Sometimes it takes a few passes through the docs to understand various concepts, but the developers have done a great job of getting excellent material out there. Nearly 10/10 times when I’m helping a customer this is the first place I’ll check to see (or confirm) that the necessary configs are spelled out well. There are document sections for based configurations (for various container flavors), quick start guides, all the OOTB sources, troubleshooting, FAQ and much more.
Check out the docs here and make sure that you select the version that you are currently using:
https://splunk.github.io/splunk-connect-for-syslog/main/
2. Out-of-the-box or custom indexes
SC4S is pre-configured to map each sourcetype to a typical index. For new installations, it is best practice to create the indexes in Splunk when using the SC4S defaults. SC4S can be easily customized to use different indexes if desired.
When you decide to use custom indexes or maybe you’ve already been adding data to custom indexes prior to implementing SC4S, you can easily override the default index values for the source by editing a configuration file called splunk_metadata.csv which contains a mapping using the key value for a source and then specifies what metadata field to overwrite, index in this case and then the value for index. Here is a typical example entry below.
juniper_netscreen,index,ns_index
See the configuration pages (in the docs that you’re supposed to read) here:
https://splunk.github.io/splunk-connect-for-syslog/main/configuration/#sc4s-metadata-configuration
3. Important HEC token settings
It is very important that you understand the configuration choices that you make when creating your HEC token on the Splunk receiving instance. There is a lot of info in the docs regarding endpoints and loadbalancer settings but the settings that are most often set incorrectly are for the indexes. Ensure the token has access to place events in main, em_metrics, and all indexes used as event destinations.
NOTE: It is recommended that the “Selected Indexes” on the token configuration page be left blank so that the token has access to all indexes, including the lastChanceIndex. If this list is populated, extreme care must be taken to keep it up to date, as an attempt to send data to an index not in this list will result in a 400 error from the HEC endpoint. Furthermore, the lastChanceIndex will not be consulted in the event the index specified in the event is not configured on Splunk. Keep in mind just one bad message will “taint” the whole batch (by default 1000 events) and prevent the entire batch from being sent to Splunk.
Check the docs here:
4. What to do if your source isn’t listed
SC4S comes pre-configured to support over 50 different Vendors and many more Sources including data formats such as CEF and LEEF. You can refer to the docs to check if the Source you want to onboard is already included, however if it’s not, you can simply head over to the GitHub project for SC4S and create a new “Issue” and provide a redacted packet capture (PCAP) of the source as well as any information that you have about the Vendor and then Splunk will develop the appropriate parser for the Source and include it in the next release. This process not only helps you, but it also potentially helps other customers that may want to onboard that Source as well.
Sources:
https://splunk.github.io/splunk-connect-for-syslog/main/sources/
Create a new Issue:
https://github.com/splunk/splunk-connect-for-syslog/issues
SC4S can be a very straightforward and simple way to implement syslog data collection. If you can follow these guidelines and avoid the gotchas you won’t have any issues when configuring Splunk Connect for syslog.
If you have any additional needs or questions please contact TekStream and we’d be happy to assist you.