At my day job, my team makes heavy use of Elasticsearch. I have not run an Elastic stack for my own personal use, but after my former boss left the organization several months back, it made sense to finally light something up for myself so I could do my own Elastic stack testing (with relevant data).
For those that don’t know Elasticsearch, it is “a distributed, RESTful search and analytics engine capable of solving a growing number of use cases.” It is the core piece of the Elastic stack, whereby the Logstash application can take (un)structured data and (re)structure it before sending it to Elasticsearch, and is then viewed via Kibana (web UI).
Elasticsearch, Logstash, and Kibana are free. You only pay for support and some advanced features.
Anyway, I ended up creating an ELK stack (Elasticsearch, Kibana, Logstash) on a standalone Linode Virtual Private Server (VPS) and started feeding my Apache, OSSEC, and other syslogs from the various servers I run for my own use (and for others).
But I decided to add the Elastic X-Pack plugins. X-Pack adds several paid / premium features, but does offer one free feature (need to request a basic license which is free): Elastic Monitoring. I ended up installing that and then disabled the premium X-Pack plugins (e.g Security, Alerting) because I was not going to pay for a Gold license.
When I disabled everything except Elastic Monitoring, Kibana would fail to start. I checked my logs and discovered the following log entries:
Mar 8 07:03:32 elk systemd: Started Kibana.
Mar 8 07:03:32 elk systemd: Starting Kibana...
Mar 8 07:03:36 elk kibana: {"type":"log","@timestamp":"2017-03-08T15:03:36Z","tags":["fatal"],"pid":31648,"level":"fatal","message":"EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'","error":{"message":"EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'","name":"Error","stack":"Error: EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'\n at Error (native)","code":"EACCES"}}
Mar 8 07:03:36 elk kibana: FATAL { Error: EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'
Mar 8 07:03:36 elk kibana: at Error (native)
Mar 8 07:03:36 elk kibana: cause:
Mar 8 07:03:36 elk kibana: { Error: EACCES: permission denied, open '/usr/share/kibana/optimize/bundles/monitoring.entry.js'
Mar 8 07:03:36 elk kibana: at Error (native)
Mar 8 07:03:36 elk kibana: errno: -13,
Mar 8 07:03:36 elk kibana: code: 'EACCES',
Mar 8 07:03:36 elk kibana: syscall: 'open',
Mar 8 07:03:36 elk kibana: path: '/usr/share/kibana/optimize/bundles/monitoring.entry.js' },
Mar 8 07:03:36 elk kibana: isOperational: true,
Mar 8 07:03:36 elk kibana: errno: -13,
Mar 8 07:03:36 elk kibana: code: 'EACCES',
Mar 8 07:03:36 elk kibana: syscall: 'open',
Mar 8 07:03:36 elk kibana: path: '/usr/share/kibana/optimize/bundles/monitoring.entry.js' }
Mar 8 07:03:37 elk systemd: kibana.service: main process exited, code=exited, status=1/FAILURE
Mar 8 07:03:37 elk systemd: Unit kibana.service entered failed state.
Mar 8 07:03:37 elk systemd: kibana.service failed.
Mar 8 07:03:37 elk systemd: kibana.service holdoff time over, scheduling restart.
A Google search for Kibana EACCES: permission denied resulted in a few interesting results, including a thread on the Elastic discussion forums.
The main problem is that for some reason, whenever plugins are disabled, a re-optimization step takes place within Kibana where files get owned by root, but Kibana runs as a non-root user (typically ‘kibana’). So when Kibana tries to start, it results in the EACCES: permission denied error.
The only workaround I found that made sense was to execute:
chown -R kibana:kibana /usr/share/kibana/optimize/
This gave the Kibana application access to the optimize directory to complete the optimization process, and then finally launch Kibana.
Another workaround I saw was to make Kibana run as root, which is not ideal.
Anyway, if you come across this issue in your own Elastic stack, try the workaround above.
The comments on Elastic threads seem to indicate there is no clear fix for this problem with existing installs. If you installed plugins as root, this will continue to be an issue. You have to install plugins as the respective application user (e.g. sudo -u <user>) from the beginning to avoid this issue.
Comment on this post