Minio shared objects operator

In this story I’m going to present an application called Minio URL Operator. I created this service in order to manage objects in Minio cluster that we want to share. This operator helps you to config and manage these urls.

Problem

Minio is an object storage server that implements the same public API as Amazon S3. This means that applications that can be configured to talk to Amazon S3 can also be configured to talk to Minio. MinIO optimizes storage of objects across available pools by writing new objects (that is, objects with no existing versions) to the server pool with the most free space compared total amount of free space on all available server pools.

MinIO Object Storage uses buckets to organize objects. A bucket is similar to a top-level drive, folder, or directory in a filesystem ( /mnt/data or C:\ ), where each bucket can hold an arbitrary number of objects. With the example structure, an administrator would create the /images , /videos and /articles buckets.

You can export these objects by creating a unique resource locator. This url will get you the requested object from the cluster. But this url is not accessible in outside of your cluster. And it’s not human readable at all. You can see an example of these urls below.

https://min.io/docs/minio/linux/developers/javascript/API.html#presignedUrl:%7E:text=getObject(bucketName%2C%20objectName%2C%20getOpts%5B%2C%20callback%5D)

Moreover, it’s not even permanent. Minio shared objects urls expire after 7 days and you need to regenerate them.

Method

In order to fix these problems, I created an operator to manage shared objects urls. I build a HTTP server using Flask and MySQL in order to control Minio urls. We map each object in a bucket to an address in this operator. When you want to access an object, this system acts like a proxy and redirects your request to Minio cluster.

Minio url operator schema

When you connect to Minio cluster, it will get objects metadata and generates a maped item in our MySQL database. After that it uses these data to manage and config urls. The system gets objects metadata from Minio using a python code like this:

def get_objects_metadata(self, bucket, prefix="") -> list:
        """get objects of a bucket based on prefix

        :param bucket: minio bucket
        :param prefix: objects prefix
        :return: list of objects metadata
        """
        client = self.minio_connection.get_connection()

        objects = client.list_objects(bucket, prefix=prefix)

        objects_list = []

        for item in objects:
            tmp = self.__get_object__(bucket, item.object_name)

            object_pack = {
                'name': item.object_name,
                'address': tmp.address,
                'status': tmp.status,
                'created_at': tmp.createdAt,
                'updated_at': tmp.updatedAt,
                'expires_at': tmp.expiresAt
            }

            objects_list.append(object_pack)

        return objects_list

Features

System supports the following list of features:

• Generating HRU (human read URL) for objects
• Set expire time (no 7 days limit)
• Block object access when needed

Operator

In this section we are going to talk about how you can use this operator in order to manage your objects.

How to use?

You can use docker image of MUP in order to setup the operator on Docker or Kubernetes.

docker pull amirhossein21/muo:v0.4.0

Run

In order to setup our container, you can use the following command:

docker run -d -it \
 -e HTTP_PORT=80 -e HTTP_DEBUG=0 -e MINIO_HOST=localhost:9000 \
 -e MINIO_SECURE=0 -e MINIO_ACCESS=9iWKawYzq68iNMN7MsiU \
 -e MINIO_SECRET=zWwZlmTX56Hr8NYBOpN4ga2zV8oO2ECIjjPHPF20 \
 -e HTTP_HOST=localhost -e HTTP_PRIVATE=1 \
 amirhossein21/muo:v0.4.0

Envs

This operator has several environmental variables in order to manage your server.

• HTTP_PORT
• HTTP_DEBUG
• HTTP_HOST
• HTTP_PRIVATE
• MYSQL_HOST
• MYSQL_PORT
• MYSQL_USER
• MYSQL_PASSWORD
• MYSQL_DB
• MYSQL_MIGRATE
• MINIO_HOST
• MINIO_SECURE
• MINIO_ACCESS
• MINIO_SECRET

Conclusion

By using this operator for Minio shared objects links you can help yourself in managing urls. Since Minio object storage provides urls for sharing objects with a maximum 7 days limit, you can use this operator in order to have live urls for every object that you want. With this operator, you can create a persistent HRU for your objects in Minio.

More documents can be found in project repository.

Thank you