MinIO Bucket Notification Guide

Events occurring on objects in a bucket can be monitored using bucket event notifications.

NOTE: Gateway mode does not support bucket notifications (except NAS gateway).

Various event types supported by MinIO server are

Use client tools like mc to set and listen for event notifications using the event sub-command. MinIO SDK's BucketNotification APIs can also be used. The notification message MinIO sends to publish an event is a JSON message with the following structure.

Bucket events can be published to the following targets:

Prerequisites

• Install and configure MinIO Server from here.
• Install and configure MinIO Client from here.

$ mc admin config get myminio | grep notify
notify_webhook        publish bucket notifications to webhook endpoints
notify_amqp           publish bucket notifications to AMQP endpoints
notify_kafka          publish bucket notifications to Kafka endpoints
notify_mqtt           publish bucket notifications to MQTT endpoints
notify_nats           publish bucket notifications to NATS endpoints
notify_nsq            publish bucket notifications to NSQ endpoints
notify_mysql          publish bucket notifications to MySQL databases
notify_postgres       publish bucket notifications to Postgres databases
notify_elasticsearch  publish bucket notifications to Elasticsearch endpoints
notify_redis          publish bucket notifications to Redis datastores

NOTE:

• '*' at the end of arg means its mandatory.
• '*' at the end of the values, means its the default value for the arg.
• When configured using environment variables, the :name can be specified using this format MINIO_NOTIFY_WEBHOOK_ENABLE_<name>.

Publish MinIO events via AMQP

Install RabbitMQ from here.

Step 1: Add AMQP endpoint to MinIO

The AMQP configuration is located under the sub-system notify_amqp top-level key. Create a configuration key-value pair here for your AMQP instance. The key is a name for your AMQP endpoint, and the value is a collection of key-value parameters described in the table below.

KEY:
notify_amqp[:name]  publish bucket notifications to AMQP endpoints

ARGS:
url*           (url)       AMQP server endpoint e.g. `amqp://myuser:mypassword@localhost:5672`
exchange       (string)    name of the AMQP exchange
exchange_type  (string)    AMQP exchange type
routing_key    (string)    routing key for publishing
mandatory      (on|off)    quietly ignore undelivered messages when set to 'off', default is 'on'
durable        (on|off)    persist queue across broker restarts when set to 'on', default is 'off'
no_wait        (on|off)    non-blocking message delivery when set to 'on', default is 'off'
internal       (on|off)    set to 'on' for exchange to be not used directly by publishers, but only when bound to other exchanges
auto_deleted   (on|off)    auto delete queue when set to 'on', when there are no consumers
delivery_mode  (number)    set to '1' for non-persistent or '2' for persistent queue
queue_dir      (path)      staging dir for undelivered messages e.g. '/home/events'
queue_limit    (number)    maximum limit for undelivered messages, defaults to '100000'
comment        (sentence)  optionally add a comment to this setting

Or environment variables

KEY:
notify_amqp[:name]  publish bucket notifications to AMQP endpoints

ARGS:
MINIO_NOTIFY_AMQP_ENABLE*        (on|off)    enable notify_amqp target, default is 'off'
MINIO_NOTIFY_AMQP_URL*           (url)       AMQP server endpoint e.g. `amqp://myuser:mypassword@localhost:5672`
MINIO_NOTIFY_AMQP_EXCHANGE       (string)    name of the AMQP exchange
MINIO_NOTIFY_AMQP_EXCHANGE_TYPE  (string)    AMQP exchange type
MINIO_NOTIFY_AMQP_ROUTING_KEY    (string)    routing key for publishing
MINIO_NOTIFY_AMQP_MANDATORY      (on|off)    quietly ignore undelivered messages when set to 'off', default is 'on'
MINIO_NOTIFY_AMQP_DURABLE        (on|off)    persist queue across broker restarts when set to 'on', default is 'off'
MINIO_NOTIFY_AMQP_NO_WAIT        (on|off)    non-blocking message delivery when set to 'on', default is 'off'
MINIO_NOTIFY_AMQP_INTERNAL       (on|off)    set to 'on' for exchange to be not used directly by publishers, but only when bound to other exchanges
MINIO_NOTIFY_AMQP_AUTO_DELETED   (on|off)    auto delete queue when set to 'on', when there are no consumers
MINIO_NOTIFY_AMQP_DELIVERY_MODE  (number)    set to '1' for non-persistent or '2' for persistent queue
MINIO_NOTIFY_AMQP_QUEUE_DIR      (path)      staging dir for undelivered messages e.g. '/home/events'
MINIO_NOTIFY_AMQP_QUEUE_LIMIT    (number)    maximum limit for undelivered messages, defaults to '100000'
MINIO_NOTIFY_AMQP_COMMENT        (sentence)  optionally add a comment to this setting

MinIO supports persistent event store. The persistent store will backup events when the AMQP broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in queue_dir field and the maximum limit of events in the queue_dir in queue_limit field. For eg, the queue_dir can be /home/events and queue_limit can be 1000. By default, the queue_limit is set to 100000.

To update the configuration, use mc admin config get notify_amqp command to get the current configuration for notify_amqp.

$ mc admin config get myminio/ notify_amqp
notify_amqp:1 delivery_mode="0" exchange_type="" no_wait="off" queue_dir="" queue_limit="0"  url="" auto_deleted="off" durable="off" exchange="" internal="off" mandatory="off" routing_key=""

Use mc admin config set command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like SQS ARNs: arn:minio:sqs::1:amqp at start-up if there were no errors.

An example configuration for RabbitMQ is shown below:

mc admin config set myminio/ notify_amqp:1 exchange="bucketevents" exchange_type="fanout" mandatory="off" no_wait="off"  url="amqp://myuser:mypassword@localhost:5672" auto_deleted="off" delivery_mode="0" durable="off" internal="off" routing_key="bucketlogs"

MinIO supports all the exchanges available in RabbitMQ. For this setup, we are using fanout exchange.

MinIO also sends with the notifications two headers: minio-bucket and minio-event. An exchange using the type "headers" can use this information to route the notifications to proper queues.

Note that, you can add as many AMQP server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the AMQP instance and an object of per-server configuration parameters.

Step 2: Enable RabbitMQ bucket notification using MinIO client

We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted images bucket on myminio server. Here ARN value is arn:minio:sqs::1:amqp. To understand more about ARN please follow AWS ARN documentation.

mc mb myminio/images
mc event add myminio/images arn:minio:sqs::1:amqp --suffix .jpg
mc event list myminio/images
arn:minio:sqs::1:amqp s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg”

Step 3: Test on RabbitMQ

The python program below waits on the queue exchange bucketevents and prints event notifications on the console. We use Pika Python Client library to do this.

#!/usr/bin/env python
import pika

connection = pika.BlockingConnection(pika.ConnectionParameters(
        host='localhost'))
channel = connection.channel()

channel.exchange_declare(exchange='bucketevents',
                         exchange_type='fanout')

result = channel.queue_declare(exclusive=False)
queue_name = result.method.queue

channel.queue_bind(exchange='bucketevents',
                   queue=queue_name)

print(' [*] Waiting for logs. To exit press CTRL+C')

def callback(ch, method, properties, body):
    print(" [x] %r" % body)

channel.basic_consume(callback,
                      queue=queue_name,
                      no_ack=False)

channel.start_consuming()

Execute this example python program to watch for RabbitMQ events on the console.

python rabbit.py

Open another terminal and upload a JPEG image into images bucket.

mc cp myphoto.jpg myminio/images

You should receive the following event notification via RabbitMQ once the upload completes.

python rabbit.py
'{"Records":[{"eventVersion":"2.0","eventSource":"aws:s3","awsRegion":"","eventTime":"2016–09–08T22:34:38.226Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"minio"},"requestParameters":{"sourceIPAddress":"10.1.10.150:44576"},"responseElements":{},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"images","ownerIdentity":{"principalId":"minio"},"arn":"arn:aws:s3:::images"},"object":{"key":"myphoto.jpg","size":200436,"sequencer":"147279EAF9F40933"}}}],"level":"info","msg":"","time":"2016–09–08T15:34:38–07:00"}'

Publish MinIO events MQTT

Install an MQTT Broker from here.

Step 1: Add MQTT endpoint to MinIO

The MQTT configuration is located as notify_mqtt key. Create a configuration key-value pair here for your MQTT instance. The key is a name for your MQTT endpoint, and the value is a collection of key-value parameters described in the table below.

KEY:
notify_mqtt[:name]  publish bucket notifications to MQTT endpoints

ARGS:
broker*              (uri)       MQTT server endpoint e.g. `tcp://localhost:1883`
topic*               (string)    name of the MQTT topic to publish
username             (string)    MQTT username
password             (string)    MQTT password
qos                  (number)    set the quality of service priority, defaults to '0'
keep_alive_interval  (duration)  keep-alive interval for MQTT connections in s,m,h,d
reconnect_interval   (duration)  reconnect interval for MQTT connections in s,m,h,d
queue_dir            (path)      staging dir for undelivered messages e.g. '/home/events'
queue_limit          (number)    maximum limit for undelivered messages, defaults to '100000'
comment              (sentence)  optionally add a comment to this setting

or environment variables

KEY:
notify_mqtt[:name]  publish bucket notifications to MQTT endpoints

ARGS:
MINIO_NOTIFY_MQTT_ENABLE*              (on|off)    enable notify_mqtt target, default is 'off'
MINIO_NOTIFY_MQTT_BROKER*              (uri)       MQTT server endpoint e.g. `tcp://localhost:1883`
MINIO_NOTIFY_MQTT_TOPIC*               (string)    name of the MQTT topic to publish
MINIO_NOTIFY_MQTT_USERNAME             (string)    MQTT username
MINIO_NOTIFY_MQTT_PASSWORD             (string)    MQTT password
MINIO_NOTIFY_MQTT_QOS                  (number)    set the quality of service priority, defaults to '0'
MINIO_NOTIFY_MQTT_KEEP_ALIVE_INTERVAL  (duration)  keep-alive interval for MQTT connections in s,m,h,d
MINIO_NOTIFY_MQTT_RECONNECT_INTERVAL   (duration)  reconnect interval for MQTT connections in s,m,h,d
MINIO_NOTIFY_MQTT_QUEUE_DIR            (path)      staging dir for undelivered messages e.g. '/home/events'
MINIO_NOTIFY_MQTT_QUEUE_LIMIT          (number)    maximum limit for undelivered messages, defaults to '100000'
MINIO_NOTIFY_MQTT_COMMENT              (sentence)  optionally add a comment to this setting

MinIO supports persistent event store. The persistent store will backup events when the MQTT broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in queue_dir field and the maximum limit of events in the queue_dir in queue_limit field. For eg, the queue_dir can be /home/events and queue_limit can be 1000. By default, the queue_limit is set to 100000.

To update the configuration, use mc admin config get command to get the current configuration.

$ mc admin config get myminio/ notify_mqtt
notify_mqtt:1 broker="" password="" queue_dir="" queue_limit="0" reconnect_interval="0s"  keep_alive_interval="0s" qos="0" topic="" username=""

Use mc admin config set command to update the configuration for the deployment. Restart the MinIO server to put the changes into effect. The server will print a line like SQS ARNs: arn:minio:sqs::1:mqtt at start-up if there were no errors.

mc admin config set myminio notify_mqtt:1 broker="tcp://localhost:1883" password="" queue_dir="" queue_limit="0" reconnect_interval="0s"  keep_alive_interval="0s" qos="1" topic="minio" username=""

MinIO supports any MQTT server that supports MQTT 3.1 or 3.1.1 and can connect to them over TCP, TLS, or a Websocket connection using tcp://, tls://, or ws:// respectively as the scheme for the broker url. See the Go Client documentation for more information.

Note that, you can add as many MQTT server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the MQTT instance and an object of per-server configuration parameters.

Step 2: Enable MQTT bucket notification using MinIO client

We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted images bucket on myminio server. Here ARN value is arn:minio:sqs::1:mqtt.

mc mb myminio/images
mc event add  myminio/images arn:minio:sqs::1:mqtt --suffix .jpg
mc event list myminio/images
arn:minio:sqs::1:amqp s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg”

Step 3: Test on MQTT

The python program below waits on mqtt topic /minio and prints event notifications on the console. We use paho-mqtt library to do this.

#!/usr/bin/env python3
from __future__ import print_function
import paho.mqtt.client as mqtt

# This is the Subscriber

def on_connect(client, userdata, flags, rc):
  print("Connected with result code "+str(rc))
  # qos level is set to 1
  client.subscribe("minio", 1)

def on_message(client, userdata, msg):
    print(msg.payload)

# client_id is a randomly generated unique ID for the mqtt broker to identify the connection.
client = mqtt.Client(client_id="myclientid",clean_session=False)

client.on_connect = on_connect
client.on_message = on_message

client.connect("localhost",1883,60)
client.loop_forever()

Execute this example python program to watch for MQTT events on the console.

python mqtt.py

Open another terminal and upload a JPEG image into images bucket.

mc cp myphoto.jpg myminio/images

You should receive the following event notification via MQTT once the upload completes.

python mqtt.py
{“Records”:[{“eventVersion”:”2.0",”eventSource”:”aws:s3",”awsRegion”:”",”eventTime”:”2016–09–08T22:34:38.226Z”,”eventName”:”s3:ObjectCreated:Put”,”userIdentity”:{“principalId”:”minio”},”requestParameters”:{“sourceIPAddress”:”10.1.10.150:44576"},”responseElements”:{},”s3":{“s3SchemaVersion”:”1.0",”configurationId”:”Config”,”bucket”:{“name”:”images”,”ownerIdentity”:{“principalId”:”minio”},”arn”:”arn:aws:s3:::images”},”object”:{“key”:”myphoto.jpg”,”size”:200436,”sequencer”:”147279EAF9F40933"}}}],”level”:”info”,”msg”:””,”time”:”2016–09–08T15:34:38–07:00"}

Publish MinIO events via Elasticsearch

Install Elasticsearch server.

This notification target supports two formats: namespace and access.

When the namespace format is used, MinIO synchronizes objects in the bucket with documents in the index. For each event in the MinIO, the server creates a document with the bucket and object name from the event as the document ID. Other details of the event are stored in the body of the document. Thus if an existing object is over-written in MinIO, the corresponding document in the Elasticsearch index is updated. If an object is deleted, the corresponding document is deleted from the index.

When the access format is used, MinIO appends events as documents in an Elasticsearch index. For each event, a document with the event details, with the timestamp of document set to the event's timestamp is appended to an index. The ID of the documented is randomly generated by Elasticsearch. No documents are deleted or modified in this format.

The steps below show how to use this notification target in namespace format. The other format is very similar and is omitted for brevity.

Step 1: Ensure Elasticsearch minimum requirements are met

MinIO requires a 5.x series version of Elasticsearch. This is the latest major release series. Elasticsearch provides version upgrade migration guidelines here.

Step 2: Add Elasticsearch endpoint to MinIO

The Elasticsearch configuration is located in the notify_elasticsearch key. Create a configuration key-value pair here for your Elasticsearch instance. The key is a name for your Elasticsearch endpoint, and the value is a collection of key-value parameters described in the table below.

KEY:
notify_elasticsearch[:name]  publish bucket notifications to Elasticsearch endpoints

ARGS:
url*         (url)                Elasticsearch server's address, with optional authentication info
index*       (string)             Elasticsearch index to store/update events, index is auto-created
format*      (namespace*|access)  'namespace' reflects current bucket/object list and 'access' reflects a journal of object operations, defaults to 'namespace'
queue_dir    (path)               staging dir for undelivered messages e.g. '/home/events'
queue_limit  (number)             maximum limit for undelivered messages, defaults to '100000'
username     (string)             username for Elasticsearch basic-auth
password     (string)             password for Elasticsearch basic-auth
comment      (sentence)           optionally add a comment to this setting

or environment variables

KEY:
notify_elasticsearch[:name]  publish bucket notifications to Elasticsearch endpoints

ARGS:
MINIO_NOTIFY_ELASTICSEARCH_ENABLE*      (on|off)             enable notify_elasticsearch target, default is 'off'
MINIO_NOTIFY_ELASTICSEARCH_URL*         (url)                Elasticsearch server's address, with optional authentication info
MINIO_NOTIFY_ELASTICSEARCH_INDEX*       (string)             Elasticsearch index to store/update events, index is auto-created
MINIO_NOTIFY_ELASTICSEARCH_FORMAT*      (namespace*|access)  'namespace' reflects current bucket/object list and 'access' reflects a journal of object operations, defaults to 'namespace'
MINIO_NOTIFY_ELASTICSEARCH_QUEUE_DIR    (path)               staging dir for undelivered messages e.g. '/home/events'
MINIO_NOTIFY_ELASTICSEARCH_QUEUE_LIMIT  (number)             maximum limit for undelivered messages, defaults to '100000'
MINIO_NOTIFY_ELASTICSEARCH_USERNAME     (string)             username for Elasticsearch basic-auth
MINIO_NOTIFY_ELASTICSEARCH_PASSWORD     (string)             password for Elasticsearch basic-auth
MINIO_NOTIFY_ELASTICSEARCH_COMMENT      (sentence)           optionally add a comment to this setting

For example: http://localhost:9200 or with authentication info http://elastic:MagicWord@127.0.0.1:9200.

MinIO supports persistent event store. The persistent store will backup events when the Elasticsearch broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in queue_dir field and the maximum limit of events in the queue_dir in queue_limit field. For eg, the queue_dir can be /home/events and queue_limit can be 1000. By default, the queue_limit is set to 100000.

If Elasticsearch has authentication enabled, the credentials can be supplied to MinIO via the url parameter formatted as PROTO://USERNAME:PASSWORD@ELASTICSEARCH_HOST:PORT.

To update the configuration, use mc admin config get command to get the current configuration.

$ mc admin config get myminio/ notify_elasticsearch
notify_elasticsearch:1 queue_limit="0"  url="" format="namespace" index="" queue_dir=""

Use mc admin config set command to update the configuration for the deployment. Restart the MinIO server to put the changes into effect. The server will print a line like SQS ARNs: arn:minio:sqs::1:elasticsearch at start-up if there were no errors.

mc admin config set myminio notify_elasticsearch:1 queue_limit="0"  url="http://127.0.0.1:9200" format="namespace" index="minio_events" queue_dir="" username="" password=""

Note that, you can add as many Elasticsearch server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the Elasticsearch instance and an object of per-server configuration parameters.

Step 3: Enable Elastichsearch bucket notification using MinIO client

We will now enable bucket event notifications on a bucket named images. Whenever a JPEG image is created/overwritten, a new document is added or an existing document is updated in the Elasticsearch index configured above. When an existing object is deleted, the corresponding document is deleted from the index. Thus, the rows in the Elasticsearch index, reflect the .jpg objects in the images bucket.

To configure this bucket notification, we need the ARN printed by MinIO in the previous step. Additional information about ARN is available here.

With the mc tool, the configuration is very simple to add. Let us say that the MinIO server is aliased as myminio in our mc configuration. Execute the following:

mc mb myminio/images
mc event add  myminio/images arn:minio:sqs::1:elasticsearch --suffix .jpg
mc event list myminio/images
arn:minio:sqs::1:elasticsearch s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg”

Step 4: Test on Elasticsearch

Upload a JPEG image into images bucket.

mc cp myphoto.jpg myminio/images

Use curl to view contents of minio_events index.

$ curl  "http://localhost:9200/minio_events/_search?pretty=true"
{
  "took" : 40,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "minio_events",
        "_type" : "event",
        "_id" : "images/myphoto.jpg",
        "_score" : 1.0,
        "_source" : {
          "Records" : [
            {
              "eventVersion" : "2.0",
              "eventSource" : "minio:s3",
              "awsRegion" : "",
              "eventTime" : "2017-03-30T08:00:41Z",
              "eventName" : "s3:ObjectCreated:Put",
              "userIdentity" : {
                "principalId" : "minio"
              },
              "requestParameters" : {
                "sourceIPAddress" : "127.0.0.1:38062"
              },
              "responseElements" : {
                "x-amz-request-id" : "14B09A09703FC47B",
                "x-minio-origin-endpoint" : "http://192.168.86.115:9000"
              },
              "s3" : {
                "s3SchemaVersion" : "1.0",
                "configurationId" : "Config",
                "bucket" : {
                  "name" : "images",
                  "ownerIdentity" : {
                    "principalId" : "minio"
                  },
                  "arn" : "arn:aws:s3:::images"
                },
                "object" : {
                  "key" : "myphoto.jpg",
                  "size" : 6474,
                  "eTag" : "a3410f4f8788b510d6f19c5067e60a90",
                  "sequencer" : "14B09A09703FC47B"
                }
              },
              "source" : {
                "host" : "127.0.0.1",
                "port" : "38062",
                "userAgent" : "MinIO (linux; amd64) minio-go/2.0.3 mc/2017-02-15T17:57:25Z"
              }
            }
          ]
        }
      }
    ]
  }
}

This output shows that a document has been created for the event in Elasticsearch.

Here we see that the document ID is the bucket and object name. In case access format was used, the document ID would be automatically generated by Elasticsearch.