Uploading images to AWS S3 in Golang

12 Jan 2020 ⏱️ 6 min

Get latest articles directly in your inbox

Recently I came across a use case where I had to store and manage images on backend. AWS (Amazon Web Service) provides with S3 storage for this purpose. AWS is one of the most popular cloud services provider and is used by a large companies like Netflix, Uber, etc.

In this tutorial I’ll cover how to upload, fetch and manage other operations for objects on AWS S3 in Golang. Let’s get started!

Setting up AWS S3

AWS comes with a free trial to help you gain hands-on experience with the AWS platform, products, and services for free. You can signup for AWS here.

Next you need to create a s3 bucket. Since there are a couple of tutorials explaining this, I’ll skip this. You can follow this for setting up s3 account.

I am assuming you have a basic Golang app in place. So next step is to fetch the official AWS SDK for Golang.

Run the following command -

go get -u github.com/aws/aws-sdk-go/...

Configuring environment variables

AWS S3 requires to create a session before we make any further request. This is basically for authentication for calls that will be made in this session.

While you setup your S3, you would have saved your secret key and ID. Also, AWS has multiple regions, you can check your region here.

AWS_REGION = ap-south-1
AWS_ACCESS_KEY_ID = <YOUR ACCESS ID> #AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY = <YOUR ACCESS KEY> #wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

You can add configs either in you .env file, yaml or other formats.

Coding out S3 operations

So I use applications.yml to manage configs for environment variables. Here is the AWSConfig struct which contains all the information related to AWS required for fetching S3 operations.

type AWSConfig struct {
    AccessKeyID     string
    AccessKeySecret string
    Region          string
    BucketName      string
    UploadTimeout   int
    BaseURL         string
}

Creating a session

AWS requires to create a session before making API calls. This acts as an authentication layer. Create a new file s3.go . Here is a method to setup session -

// s3.go
package s3

import (
    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/credentials"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3"
)

// This function creates session and requires AWS credentials
func CreateSession(awsConfig config.AWSConfig) *session.Session {
    sess := session.Must(session.NewSession(
        &aws.Config{
            Region: aws.String(awsConfig.Region),
            Credentials: credentials.NewStaticCredentials(
                awsConfig.AccessKeyID,
                awsConfig.AccessKeySecret,
                "",
            ),
        },
    ))
    return sess
}

Creating S3 Session

In the previous step we create a AWS session. But for using S3 we need to create s3 session. Following is a function to create that. You can combine it with previous method too.

func CreateS3Session(sess *session.Session) *s3.S3 {
    s3Session := s3.New(sess)
    return s3Session
}

Uploading images to S3

Once we have s3 session, we can start with uploading images now. The method below creates new Uploader instance that uses the Upload method to upload the object to S3.

// upload_objects_s3.go
package s3

import (
    "s3_blog/internal/config"
    "s3_blog/pkg/logger"
    "fmt"
    "os"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/awserr"
    "github.com/aws/aws-sdk-go/aws/request"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3/s3manager"
)

// bucket: name of your s3 bucket
// filePath: path to file you want to upload
// fileName: name with which it will be present on s3
func UploadObject(bucket string, filePath string, fileName string, sess *session.Session, awsConfig config.AWSConfig) error {

    // Open file to upload
    file, err := os.Open(filePath)
    if err != nil {
        logger.Error("Unable to open file %q, %v", err)
        return err
    }
    defer file.Close()

    // Upload to s3
    uploader := s3manager.NewUploader(sess)
    _, err = uploader.Upload(&s3manager.UploadInput{
        Bucket: aws.String(bucket),
        Key:    aws.String(fileName),
        Body:   file,
    })

    if err != nil {
        logger.Error(os.Stderr, "failed to upload object, %v\n", err)
        return err
    }

    fmt.Printf("Successfully uploaded %q to %q\n", fileName, bucket)
    return nil
}

Downloading images from S3

We can download the image we have uploaded using the method below. Here we create new Downloader instance to fetch objects from S3. Then we use the Download method that downloads the object.

// download_objects_s3.go
package s3

import (
    "s3_blog/internal/config"
    "s3_blog/pkg/logger"
    "fmt"
    "os"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3"
    "github.com/aws/aws-sdk-go/service/s3/s3manager"
)

// bucket: name of your s3 bucket
// item: name of object on s3 that you want to download
// fileName: name of file where this object will be downloaded
func DownloadObject(bucket string, item string, fileName string, sess *session.Session, awsConfig config.AWSConfig) error {
    // open file to save contents to
    file, err := os.Open(fileName)
    if err != nil {
        logger.Error("Unable to open file %q, %v", err)
        return err
    }
    defer file.Close()

    // Download from s3
    downloader := s3manager.NewDownloader(sess)
    numBytes, err := downloader.Download(file,
        &s3.GetObjectInput{
            Bucket: aws.String(bucket),
            Key:    aws.String(item),
        },
    )
    if err != nil {
        logger.Error("Unable to download item %q, %v", item, err)
        return err
    }

    fmt.Println("Downloaded", file.Name(), numBytes, "bytes")
    return nil
}

Listing objects on S3

You can list out objects present on s3 using ListObjectsV2. This will list down all the objects in specific bucket.

// list_objects_s3.go
package s3

import (
    "s3_blog/internal/config"
    "s3_blog/pkg/logger"
    "fmt"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/service/s3"
)

// bucket: name of your s3 bucket
// svc is the s3 session we created
func ListObjects(svc *s3.S3, bucket string, awsConfig config.AWSConfig) error {

    resp, err := svc.ListObjectsV2(&s3.ListObjectsV2Input{
        Bucket: aws.String(bucket),
    })

    if err != nil {
        logger.Error("Unable to list items in bucket %q, %v", bucket, err)
        return err
    }

    for _, item := range resp.Contents {
        fmt.Println("Name:         ", *item.Key)
        fmt.Println("Last modified:", *item.LastModified)
        fmt.Println("Size:         ", *item.Size)
        fmt.Println("Storage class:", *item.StorageClass)
        fmt.Println("")
    }
    return nil
}

Additional Notes

There are a lot more methods you can explore based on your usecase. You can create more methods like above. Some common ones can be -
- ListBuckets
- DeleteObject
- RestoreObject

The code I presented above is good for starting out. For production I would recommend to use context based methods. So the download object method would change like this -

  ctx := CreateContextForTimeout(awsConfig)
  downloader := s3manager.NewDownloader(sess)

  numBytes, err := downloader.DownloadWithContext(ctx, file,
      &s3.GetObjectInput{
          Bucket: aws.String(bucket),
          Key:    aws.String(item),
      },
  )
  if err != nil {
      logger.Error("Unable to download item %q, %v", item, err)
      return err
  }