IBM Cloud Object Storage access using Boto

IBM Cloud Object Storage access using Boto


IBM Cloud Object Storage (or COS) is a highly scalable cloud storage service, designed for high durability, resiliency and security. It is available on-premises and in the cloud (public and private). It can be self managed, or fully managed by IBM. There is a free plan (Lite), which provides up to 25 GB of free storage a month, and if you haven't got a COS account you can sign up for free here – https://www.ibm.com/cloud/object-storage.

In this quick post I will show you how to connect and access data in COS from Python.

By default COS IBM Cloud Identity and Access Management for authentication and authorization, which doesn't work with traditional S3 clients. However, as stated in the official documentation you can enable support for the S3 API by adding a set of HMAC credentials. This will add the Access Key / Secret Key pair, which is needed by the S3-compatible tools and libraries.

Here is a screenshot of one of my COS instances, that has a test bucket (test-bucket-1234) with some sample contents.

To add the HMAC keys we first go to Service credentials, and then we click the New Credential button.

We set a name and select an access role for the credentials set, and then we pass the {"HMAC":true} parameter in the configuration parameters box.

You can see that my credentials now have an additional element, which contains the e Access Key / Secret Key pair.

We can now put some Python code together. First, we connect to the COS instance and list all the available buckets:

import boto
import boto.s3.connection
access_key = "ACCESS_KEY_HERE"   # Change to match your setup
secret_key = "SECRET_KEY_HERE"   # Change to match your setup
bucket = "test-bucket-1234"      # Change to match your setup

host = "s3-api.us-geo.objectstorage.softlayer.net"

conn = boto.connect_s3(
        aws_access_key_id = access_key,
        aws_secret_access_key = secret_key,
        host = host,
        calling_format = boto.s3.connection.OrdinaryCallingFormat(),
        )

for bucket in conn.get_all_buckets():
        print "{name}\t{created}".format(
                name = bucket.name,
                created = bucket.creation_date,
        )
test-bucket-1234        2018-01-16T10:58:29.276Z

We can also look at the contents of the test bucket.

mybucket = conn.get_bucket(bucket) 

for key in mybucket.list():
        print "{name}\t{size}\t{modified}".format(
                name = key.name,
                size = key.size,
                modified = key.last_modified,
                )
the_moon_and_sixpence.txt       423657  2018-01-16T11:01:55.960Z

We can use the get_contents_to_filename() method to download the data like this:

key = bucket.get_key("the_moon_and_sixpence.txt")
key.get_contents_to_filename("/PATH_TO_LOCAL_DIRECTORY/the_moon_and_sixpence.txt")

Or we can use all the other methods available in the Boto package to interact with the COS stored data.