Operations

Sending logs to a syslog server

Guide · Administrators · Applies to 5.0.175 and later; log search from 5.0.177

Every Eldric node can send its own log lines to a central syslog server. This page shows how to set up such a server with rsyslog, how to connect each Eldric node to it, and what to check when nothing arrives.

On this page

What Eldric sends Set up the syslog server Extended example Connect Eldric Check that it works Pitfalls Searching logs in Eldric

What Eldric sends

Each node sends its own log lines. Configure every node you want to see.
Format: syslog RFC 5424, application name eldric-kernel, facility local0 (16) unless you change it. Errors arrive as severity err, warnings as warning, normal lines as info, debug lines as debug.
Transport: UDP port 514 by default; TCP (octet-counted framing) or TLS on request.
Sending never blocks Eldric: an unreachable server costs at most a short timeout on TCP, and UDP is fire-and-forget. Lines that cannot be sent are counted (see Check that it works).

Set up the syslog server

The example uses Fedora and rsyslog. Any syslog server that accepts RFC 5424 over UDP or TCP works.

Install rsyslog (on Fedora it is usually already there):
sudo dnf install rsyslog
Give the logs their own disk. Logs grow without limit; a full root file system stops the server. Mount a separate disk, for example at /data/syslog.

Receive on port 514 and write one file per sender and day. Create /etc/rsyslog.d/50-eldric.conf:

module(load="imudp")
module(load="imtcp")
input(type="imudp" port="514")
input(type="imtcp" port="514")

template(name="PerHostDay" type="string"
         string="/data/syslog/%HOSTNAME%/%$YEAR%-%$MONTH%-%$DAY%.log")

if $fromhost-ip != "127.0.0.1" then {
    action(type="omfile" dynaFile="PerHostDay" createDirs="on")
    stop
}

Open the port only for your Eldric nodes, not for everyone. With firewalld, for a management network 10.0.0.0/24:
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.0.0.0/24" port port="514" protocol="udp" accept'
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.0.0.0/24" port port="514" protocol="tcp" accept'
sudo firewall-cmd --reload
Check the configuration before you restart:
sudo rsyslogd -N1
Only when it reports no errors: sudo systemctl restart rsyslog.

rsyslog does not reload its configuration; a change always needs rsyslogd -N1 and then a restart.

To undo it: remove /etc/rsyslog.d/50-eldric.conf, restart rsyslog, and remove the two firewall rules.

TLS: to receive over TLS, configure rsyslog's TLS listener on port 6514 with your certificate, and connect Eldric with protocol tls and your CA certificate (below).

Extended example: JSON lines, two disks, classes

This is the configuration of our own log server. It writes every line as one JSON object (the format Eldric's log module reads), into hourly files per class in UTC, on two disks mounted at /data and /data2. Each copy has its own disk-assisted queue, so one stuck disk does not stop the other. The classes and the firewall block are an example: 192.0.2.1 stands for your own firewall, and which lines you keep or drop is your decision.

Before you use it: sudo mkdir -p /var/lib/eldric/logs (rsyslog's statistics are written there every 60 seconds). If SELinux is enforcing, the two log directories also need a file context rsyslog may write to; we have not tested that case.

module(load="imudp")
module(load="imtcp")
module(load="impstats" interval="60" format="json" log.syslog="off" log.file="/var/lib/eldric/logs/rsyslog-stats.json" resetCounters="off")

# One JSON object per line; the field names are what the Eldric logs module reads.
template(name="eldric_json" type="list" option.jsonf="on") {
  property(outname="uuid"        name="$.uuid"          format="jsonf")
  property(outname="ts"          name="timereported"    dateFormat="rfc3339" format="jsonf")
  property(outname="rcvd"        name="timegenerated"   dateFormat="rfc3339" format="jsonf")
  property(outname="host"        name="hostname"        format="jsonf")
  property(outname="fromhost_ip" name="fromhost-ip"     format="jsonf")
  property(outname="app"         name="$.app"           format="jsonf")
  property(outname="severity"    name="syslogseverity"  format="jsonf" datatype="number")
  property(outname="facility"    name="syslogfacility"  format="jsonf" datatype="number")
  property(outname="class"       name="$.class"         format="jsonf")
  property(outname="msg"         name="$.msg"           format="jsonf")
}
# Hourly files per class, in UTC, on two disks.
template(name="eldric_path_a" type="string" string="/data/%$now-utc%/%$hour-utc%-%$.class%.jsonl")
template(name="eldric_path_b" type="string" string="/data2/%$now-utc%/%$hour-utc%-%$.class%.jsonl")

ruleset(name="eldric_in") {
  # Create the line id once, before both outputs (otherwise the two outputs race on it).
  set $.uuid = $uuid;
  set $.class = "other";
  set $.app = $programname;
  set $.msg = $msg;
  # EXAMPLE: a FortiGate at 192.0.2.1. Its lines have no syslog header, so keep the raw text.
  if ($fromhost-ip == "192.0.2.1") then {
    set $.class = "firewall";
    set $.app = "fortigate";
    set $.msg = $rawmsg-after-pri;
  }
  # No DNS query logging.
  if ($programname == "named" and ($msg contains "query:")) then { stop }
  # Allowed forward traffic: archive only (class firewall_allowed is kept in the files, not indexed).
  if ($.app == "fortigate" and $.msg contains "type=\"traffic\"" and $.msg contains "subtype=\"forward\""
      and not ($.msg contains "action=\"deny\"" or $.msg contains "action=\"blocked\""
               or $.msg contains "action=\"dropped\"" or $.msg contains "action=\"ip-conn\"")) then {
    set $.class = "firewall_allowed";
  }
  # Two copies, each with its own disk-assisted queue: one stuck disk does not stop the other.
  action(type="omfile" dynaFile="eldric_path_a" template="eldric_json" createDirs="on" dirCreateMode="0750" fileCreateMode="0640"
         dynaFileCacheSize="64" asyncWriting="on" flushOnTXEnd="on"
         name="eldric_copy_a" queue.type="LinkedList" queue.filename="eldric_copy_a" queue.saveOnShutdown="on"
         queue.maxDiskSpace="10g" action.resumeRetryCount="-1")
  action(type="omfile" dynaFile="eldric_path_b" template="eldric_json" createDirs="on" dirCreateMode="0750" fileCreateMode="0640"
         dynaFileCacheSize="64" asyncWriting="on" flushOnTXEnd="on"
         name="eldric_copy_b" queue.type="LinkedList" queue.filename="eldric_copy_b" queue.saveOnShutdown="on"
         queue.maxDiskSpace="10g" action.resumeRetryCount="-1")
  stop
}
input(type="imudp" port="514" ruleset="eldric_in")
input(type="imtcp" port="514" ruleset="eldric_in")

We checked this configuration with rsyslog 8.2604 on Fedora 44: rsyslogd -N1 accepts it, lines arriving over UDP and TCP land as JSON in both copies, and both copies carry the same line id for every line. Creating the id once, before the two outputs, is what keeps the ids equal; without that, rsyslog can give the two copies different ids under load.

Connect Eldric

Permanently, on each node

Add the server to /etc/eldric/eldric-aios.env on every node, then restart the service on that node (sudo systemctl restart eldric-aios):

ELDRIC_SYSLOG_HOST=logs.example.internal
ELDRIC_SYSLOG_PORT=514
ELDRIC_SYSLOG_PROTOCOL=udp        # udp (default), tcp or tls
# Only for TLS:
# ELDRIC_SYSLOG_CA_CERT=/etc/eldric/pki/ca.pem
# ELDRIC_SYSLOG_CLIENT_CERT=/etc/eldric/pki/client.pem
# ELDRIC_SYSLOG_CLIENT_KEY=/etc/eldric/pki/client.key

For TLS, set the CA certificate: without it the node does not verify the server's certificate. Client certificate and key are needed only if your server requires them. The older single setting ELDRIC_SYSLOG_SERVER=host:port still works; ELDRIC_SYSLOG_HOST wins when both are set.

At runtime, through the API

An administrator can set the destination on a running node with PUT /api/v1/system/syslog/config, for example {"host": "logs.example.internal", "port": 514, "protocol": "tcp"}; {"enabled": false} switches forwarding off. GET on the same path shows the current setting. Each call changes only the node you send it to.

From 5.0.177 the setting is saved on that node and survives a restart; a saved setting wins over the ELDRIC_SYSLOG_* environment. GET shows where it came from (source: saved or environment), and DELETE /api/v1/system/syslog/config removes the saved setting and applies the environment again at once. Invalid values are refused with 400 (invalid_protocol, invalid_port, invalid_facility, invalid_tenant_route) and change nothing. If saving fails, the answer is 500 syslog_config_not_saved: applied, but lost at the next restart. The saved setting includes the node's host name; a copied file keeps the old name until you change it.
In 5.0.175 the setting lasts only until that node restarts; use the environment file above for a lasting setting.

Logs of one tenant can go to a different server: PUT /api/v1/system/syslog/config/tenant/<tenant> with host, port and protocol (udp, tcp or tls; TLS uses the node's own certificates set above, there are none per tenant), and DELETE on the same path removes it. Tenant routes follow the same rule as the main setting: saved from 5.0.177, until restart in 5.0.175.

Check that it works

Send a test line: POST /api/v1/system/syslog/test (administrator). The answer shows where it was sent; 503 syslog_not_configured means this node has no destination.
Look for it on the server: a line containing eldric-aios syslog test in the file for that node.
Read the counters: GET /api/v1/system/syslog/stats shows lines queued, sent, dropped and send errors. Rising send_errors_total or dropped_total means the server is unreachable or too slow.

Pitfalls

Nothing arrives over UDP: check the firewall on the server first; UDP gives the sender no error.
Other devices on the same server: some firewalls and appliances send lines without a standard syslog header. A parser expecting one cuts off the first field. Give such sources their own rule.
Not all log lines are valid UTF-8: lines from some devices contain bytes that are not UTF-8. Tools that read the files strictly as UTF-8 fail on them.
Host names: the name a line is filed under can be a resolver's name for the sending address, not the device's own name. Check how your server resolves sender addresses.

Searching logs in Eldric (from 5.0.177)

From 5.0.177 Eldric can index and search what your syslog server collects. Everything in this section is for administrators only, and needs no licence feature.

Where it runs: on one node you give the logs role, which also runs rsyslog. Eldric does not receive syslog itself; rsyslog on that node does.
rsyslog's configuration is written by Eldric: you set allowed networks, sources and their classes with PUT /api/v1/logs/rsyslog/config, and POST /api/v1/logs/rsyslog/apply writes the file, has rsyslog check it with rsyslogd -N1 and restarts it. If rsyslog rejects it, the previous file is put back and the answer is 422 rsyslog_config_invalid with rsyslog's message. The hand-written configuration above is then only needed for a syslog server outside Eldric.
Two copies: lines are written to two disks, and the index switches to the other copy on its own when one disk disappears or stops being written; GET /api/v1/logs/status shows which copy is read and whether both are in step.
Retention and secrets: you set per class how long lines stay searchable and how long the files are kept; secrets are removed before a line is indexed (the files keep the original, on the logs node only).
Searching and following: in the admin console's Syslog dashboard, with GET /api/v1/logs/search (time range, class, severity, full text) and live with GET /api/v1/logs/tail. The chat can search and summarise logs for administrators.
Detection: Sigma detection rules raise alerts, which can be grouped into incidents; matches against the Spamhaus DROP list are marked, with Spamhaus's credit; learned baselines show what is usual for a signal.
Network flows: NetFlow and IPFIX are received and kept as aggregates; the volumes are as reported by the exporter, not exact.

The built-in help topic logs.overview on your installation describes every setting and route.

Admin guide Install & troubleshooting