Vishal Sharma's Blog
Amazon S3 Simplified: A Beginner's Guide

Amazon S3 Simplified: A Beginner's Guide

Vishal Sharma's photo
Vishal Sharma
·

9 min read

In today's digital era, managing and storing vast amounts of data efficiently has become crucial for businesses and individuals alike. Amazon Simple Storage Service (S3) stands as a cornerstone in cloud storage solutions, offering scalable, reliable, and secure storage for a wide array of use cases. Whether you're an enterprise handling massive datasets or a small-scale developer building applications, S3 provides the foundation for storing and retrieving data seamlessly.

What is Amazon S3?

Amazon S3, short for Simple Storage Service, stands as a cornerstone within the Amazon Web Services (AWS) ecosystem, offering a highly scalable, durable, and secure object storage solution. Its flexibility and reliability have made it a go-to choice for businesses and developers worldwide.

With Amazon S3, users can store virtually any type of data, ranging from text files to multimedia content like audio, video, and images. This data is stored as objects, known as Binary Large Objects (BLOBs), organized within buckets, which act as top-level containers for data storage.

Key Features of S3

Here are a few key features that make the S3 unbeatable in the market are as follows:

  1. Scalability: S3 is designed to scale effortlessly, allowing users to store and retrieve virtually unlimited amounts of data without worrying about capacity constraints. Whether you're storing a few gigabytes or petabytes of data, S3 can handle it with ease.

  2. Durability: Data stored in S3 is replicated across multiple storage devices and facilities within a chosen AWS region, ensuring high durability and resilience against hardware failures, errors, and even natural disasters.

  3. Availability: S3 provides high availability for data access, with service level agreements (SLAs) guaranteeing 99.99% uptime. This ensures that your data is accessible whenever you need it, with minimal downtime or interruptions.

  4. Security: S3 offers robust security features to protect your data, including encryption at rest and in transit using SSL/TLS, access control lists (ACLs), and bucket policies. It also integrates seamlessly with AWS Identity and Access Management (IAM), allowing you to manage access permissions and control who can access your data.

  5. Lifecycle Management: S3 allows you to define lifecycle policies to automate the management of your objects over time. You can configure rules to automatically transition objects to different storage classes (e.g., from Standard to Infrequent Access or Glacier) or delete objects after a specified period, helping you optimize storage costs and meet compliance requirements.

How to use AWS S3 correctly?

At first, you have to set up the s3 bucket for the project and we also have to get the secret key, access key, and s3 region. So let's start

  1. Go to the AWS console and search S3 and then click on Create Bucket.

  2. Add the credentials like the name of the bucket which should be unique and then

    uncheck the block all public access click on the create Bucket.

  3. Now navigate to the Permission section after your bucket is ready and paste this permission in the bucket policy it will keep you users to only get the object stored in the bucket and delete them and the delete one you can edit as per your need.

      {
     "Version": "2012-10-17",
     "Statement": [
         {
             "Sid": "Statement1",
             "Effect": "Allow",
             "Principal": "*",
             "Action": [
                 "s3:GetObject",
                 "s3:DeleteObject"
             ],
             "Resource": "YOUR_BUCKET_ARN"
         }
     ]
 }

  4. In real-world scenarios, it's essential to provide different levels of access to various users or teams within a company, depending on the services they need to access. Similarly, we will implement access control for our photo app's S3 bucket using AWS Identity and Access Management (IAM). IAM allows us to define granular permissions for different users or groups, ensuring secure and controlled access to resources.

    Create a user

  5. Now after creating the user you have to create and group in which we will add this user and set their permission about how much access they have and what are services they can use.

  6. No we just need two more credentials , first access key and the second is secret_key that we will generate.

Keep this secret key and access key handy because we gonna use this in our app to access the S3 and also remember to keep your environment variable in the .env file.

Disclaimer: Currently we are keeping the S3 bucket public but what if we keep it private? Then we have to introduct the concept of preSigned URL to access the bucket which we will talk in some next sort of blog.

Let's create a simple Photo Gallery App in MERN using S3 to store the pictures and perform some CRUD operations over time yes it would be informative for you so let's get started...

  1. Name a Folder with the name PhotoApp or whatever you like and Open it in your favorite code editor.

  2. In that make two separate folder for the backend and frontend setup, start your normal empty node app and install the following packages.

     npm i nodemon express dotenv mongoose cors multer-s3 @aws-sdk/client-s3 
 multer

  3. Now let's write a basic boilerplate code to start the express app in the node.js and to make the code look clean, we will use the different folders for different purposes, like all the routes in the route folder, all the database model in the db folder. Please also make the.env file to handle the environment variable.

     //.env
 MONGO_URL='MONGO URI, either use the localhost string or the mongodb atlas string'
 PORT=3000
 BUCKET='Your Bucket Name'
 AWS_ACCESS_KEY='Your Access Key that you hae to make'
 AWS_SECRET_KEY='A secrret from  AWS IAM' 
 AWS_REGION='region of S3'

//server.js
const express=require('express')
const app = express()
const cors = require('cors')
require('dotenv').config();
const photoRouter=require('./routes/route');
const port=process.env.PORT
app.use(cors());
app.use(express.json());
app.use(express.urlencoded({ extended: true }));


app.listen(port,()=>{
console.log(`Server Running on port ${port}`)
})

//router.js
const express = require('express');
require('dotenv').config();
const {getImage , uploadImage ,deleteImage ,updateImage} = require('../controller/controller')
const {upload}=require('../middleware/upload');
const router= express.Router();

router.route('/').get(getImage);
router.route('/upload').post(upload.single('image'),uploadImage);
router.route('/delete/:id').delete(deleteImage);
router.route('/update/:id').put(updateImage);  // This is nothing but delete and then upload


module.exports = router;

//db.js
const mongoose = require('mongoose');
require('dotenv').config();
mongoose.connect(process.env.MONGO_URL, {
    useNewUrlParser: true,
    useUnifiedTopology: true,
  })
  .then(() => {
    console.log('Connected to MongoDB');
  })
  .catch((error) => {
    console.error('Error connecting to MongoDB:', error.message);
  });
const photoSchema = new mongoose.Schema({
  picUrl: { type: String, required: true },
  tag: { 
    type: String, 
    enum: ['holidays', 'events', 'nature'], 
    required: true 
  }, 
}, { timestamps: true }); 
const Photo = mongoose.model('Photo', photoSchema);

module.exports = Photo;

//controller.js
const photo = require('../db/db'); // Assuming this is your photo model
const {deleteFromS3}=require('../services/service');
async function getImage(req, res) {
  try {
    const images = await photo.find(); // Fetch all images from the database
    res.json(images);
  } catch (error) {
    console.error(error);
    res.status(500).json({ error: 'Internal server error' });
  }
}

async function uploadImage(req, res) {
  try {

    const { location } = req.file;
    const newPhoto = new photo({
      picUrl: location, 
      tag: req.body.tag 
    });
    await newPhoto.save(); 
    res.status(201).json(newPhoto);
  } catch (error) {
    console.error(error);
    res.status(500).json({ error: 'Internal server error' });
  }
}

async function deleteImage(req, res) {
  try {
    const { id } = req.params; 
    const deletedPhoto = await photo.findByIdAndDelete(id);
    const picUrl = deletedPhoto.picUrl;
    await deleteFromS3(picUrl);
    res.sendStatus(204);
  } catch (error) {
    console.error(error);
    res.status(500).json({ error: 'Internal server error' });
  }
}

async function updateImage(req, res) {
  try {
    const { id } = req.params;
    const { tag } = req.body;
    const updatedPhoto = await photo.findByIdAndUpdate(id, { tag }, { new: true });
    res.json(updatedPhoto);
  } catch (error) {
    console.error(error);
    res.status(500).json({ error: 'Internal server error' });
  }
}

module.exports = {
  getImage,
  deleteImage,
  updateImage,
  uploadImage
};

I know there is a lot of code and the thing that might captured your eyes is why we used a upload('image') middleware in /uploadImage route.So whenever you have to upload a file then you have to handle where this file would be stored in the storage and how we will gain access that , what would be the name and handle these all we have multer library which handles multi-part form data.

For the same reason in S3 we are using the same multer-S3 and multer package to do the same.I am not saying that this is the only thing but this is good to go approach.

//upload.js
const multer = require('multer');
const { multerS3 } = require('multer-s3');
const { AWS } = require('../services/service');
const upload = multer({
  storage: multerS3({
    s3: AWS,
    bucket: process.env.AWS_BUCKET,
    key: function (req, file, cb) {
      cb(null, `upload/Pictures/${file.originalname}`);//where we will store -path
    },
  }),
  fileFilter: function (req, file, cb) { //checks the file type

    if (
      file.mimetype === "image/jpeg" ||
      file.mimetype === "image/png" ||
      file.mimetype === "image/jpeg"
    ) {
      cb(null, true);
    } else {
      cb(new Error("Only images are allowed!"), false);
    }
  },
});

module.exports = upload;

//services.js
const { S3Client, DeleteObjectCommand } = require("@aws-sdk/client-s3");

const AWS = new S3Client({
  region: process.env.AWS_REGION,
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY,
    secretAccessKey: process.env.AWS_SECRET_KEY,
  },
});

async function deleteFromS3(url) {
  const urlParts = new URL(url);
  const bucket = urlParts.hostname.split('.')[0];
  let key = urlParts.pathname.substring(1);
  key = decodeURIComponent(key);

  const params = {
    Bucket: bucket,
    Key: key,
  };

  await AWS.send(new DeleteObjectCommand(params));
}

module.exports = {
  AWS,
  deleteFromS3
};

Now these have all the crux of Uploading and deleting from the S3 let's break the code and look into it. To Perform operation from the nodejs we have to make an S3Client in our nodejs in behalf of which we will perform the tasks in our S3 bucket.

const { S3Client, DeleteObjectCommand } = require("@aws-sdk/client-s3");

const AWS = new S3Client({
  region: process.env.AWS_REGION,
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY,
    secretAccessKey: process.env.AWS_SECRET_KEY,
  },
});

Now to perform the upload functionality, we utilize Multer-S3, an extension of Multer designed specifically for integrating with AWS S3 for file storage. Multer-S3 seamlessly interfaces with Amazon S3 to handle the storage of uploaded files.

In the provided code snippet, we configure Multer to use Multer-S3 as the storage engine. This configuration includes specifying the AWS S3 client (s3) and the destination bucket (bucket). Additionally, we define a custom key function to determine the storage path of the uploaded file within the S3 bucket.

The fileFilter function ensures that only files with specified MIME types (in this case, JPEG and PNG images) are allowed for upload. If a file with an unsupported MIME type is attempted to be uploaded, Multer will throw an error, preventing the upload process from proceeding.

Now for the DeleteObject we just have to give the bucket and the key. Bucket is the name of our bucket and the key we will extract from the picUrl which will look similar to this

async function deleteFromS3(url) {
  const urlParts = new URL(url);
  const bucket = urlParts.hostname.split('.')[0];
  let key = urlParts.pathname.substring(1);
  key = decodeURIComponent(key);

  const params = {
    Bucket: bucket, //productbase12
    Key: key, //1704889883421-Screenshot+2023-09-25+122851.png
  };

  await AWS.send(new DeleteObjectCommand(params));
}

Frontend Side

So now we are on the last phase of this amazing blog and if you know the frontend bit then it is very good otherwise you can also refer to the GitHub repo of this project.

Make your version and tag me on the X and it would be great. You can also make some changes in my code.

I hope you are loving this series in the next sort of blog we will going to deploy this same application using EC2 service and if you don't know what is EC2 then you can check out my previous blog where we made our first instance and connected that with our windows machine using SSH.

If you are loving this series then please share this with your friends who are still confused.So keep moving and keep learning. Happy Coding ...🚀👨‍💻

Did you find this article valuable?

Support Vishal Sharma by becoming a sponsor. Any amount is appreciated!

S3AWS100xdevsWeb DevelopmentCloud Computing