Earlier today Graylog stopped processing messages but I didn’t realise for a good few hours. Eventually i found it wasn’t “Outputting” messages per the indicator in the top right.
It was ingesting them, but not outputting them. The disk had run out of space.
With the primary disk space issue resolved Elasticsearch showed all shards as green, but still not messages were being processed.
I rebooted the server, no change.
After some digging in the logs I found this message
WARN [Messages] Retrying 76 messages, because their indices are blocked with status [read-only / allow delete]
I searched the web for some more hints and found this curl command I needed to run on the elasticsearch host
curl -XPUT -H "Content-Type: application/json" http://127.0.0.1:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'
Immediately after running the above command messages started flowing.
Moral of the day :- Check your retention settings to make sure they are sufficient to not allow your disk to be filled
