Three Ways to Splunk Your Docker Containers
Logging from Docker is becoming less of a total pain in the butt. EnerNOC uses Splunk Enterprise for its central logging solution, and when we started building apps with Docker and AWS ECS, getting our logs into Splunk was a real puzzle.
I’ve set up a couple of different methods to log out to Splunk with varying success.
Option 1: Splunk Agent on the Host and Shared Log Volumes
The first thing I tried was provisioning our host ECS machines to run the Splunk forwarder and share volumes from the running containers.
This was a quick and easy solution since I’d done a lot of Chef-style setup of the Splunk forwarder before and it wasn’t difficult to get going by having applications in Docker put their logs in a directory that is mounted to the host. The Splunk forwarder reads from those logs and sends along.
The drawbacks here:
- It’s running a process on the host, which breaks convention of having your application stack managed by Docker / ECS.
- The container process is responsible for logging to a file, but if that process crashes, the logs don’t capture it.
Option 2: Use the Splunk Log Driver in Docker
With version 1.10, Docker included a splunk
driver option for its engine.
It took quite some time for AWS ECS to support this option, during which time
I would regularly go to the ECS Agent documentation page to refresh hoping for
an update.
This requires the HTTP Event Collector, which is token based. When ECS finally
supported it, I updated our task definitions to use the driver and inject
tokens and settings. I like how easy it is to set metatags like source
and
index
, but the big gotcha here is that the logs show up in Splunk as
{
"line": "logger=output foo=bar",
"source": "stdout",
"tag": "container-tag"
}
This makes simple querying within the line
attribute quite difficult,
requiring rex
-style matching to do field extractions or stats. The
next version of Docker (1.13) will support a splunk-format
log-driver option
to let the splunk driver send JSON or even just raw strings. Until then, I
need to find a better way to send the raw events to Splunk.
Option 3: Use a Splunk Forwarding Container
On instance start (in your Launch Configuration user_data
script) spin up
a Splunk forwarder container to listen to syslog
and forward the events to
Splunk. Then use the syslog
log driver to send container log messages there.
version: '2'
services:
# Splunk forwarder to send all syslogs to Splunk
forwarder:
image: 'splunk/universalforwarder:6.5.0'
environment:
SPLUNK_FORWARD_SERVER: 'YOUR_SPLUNK_HOST:9997'
SPLUNK_FORWARD_SERVER_ARGS: '-method clone'
SPLUNK_START_ARGS: '--accept-license --answer-yes'
SPLUNK_ADD: 'udp 1514 -index INDEX_NAME -sourcetype STACK_NAME'
SPLUNK_USER: 'root'
ports:
- '514:1514/udp'
restart: 'always'
# Log producer
app:
image: 'alpine'
command: 'sh -c "while true; do echo \"hi `date`\"; sleep 1; done;"'
logging:
driver: 'syslog'
options:
tag: 'CONTAINER_NAME_OR_TAG'
syslog-address: 'udp://127.0.0.1'
This gets us raw logs, which are more easily searchable until Docker 1.13 is released.