S3 Bucket Mount to EC2-Centos6.4

This post is about how to mount s3 bucket on Ec2 instance.EC2 Instance is Linux Centos 6.4 version.

https://documents.lucidchart.com/documents/d17b1447-4bca-4850-8e47-886ae7070f0a/pages/0_0?a=726&x=244&y=430&w=792&h=220&store=1&accept=image%2F*&auth=LCA%209a67d2b87fc0330ef686d9284828b2301314aa8a-ts%3D1481267198

S3 Bucket Mount to EC2-Centos6.4

  • Requirements to mount S3 bucket:
    • Access Credentials
      • AWS Access Key ID
      • Secret Access Key
  • Name of bucket you want mount
    Read/Write permissions to bucket
    s3fs-fuse
  1. First You have to install AWS-CLI into the server.
  2. Configure – “aws configure” to Access and Secret Key.
  3. If you like to access S3 buckets without mounting on system, use s3cmd command line utility to manage s3 buckets. s3cmd is also provides faster speed for data upload and download rather than s3fs. To work with s3cmd use next articles to install s3cmd.
    1. Install Latest S3cmd using Source:
      1. wget http://ufpr.dl.sourceforge.net/project/s3tools/s3cmd/1.6.1/s3cmd-1.6.1.tar.gz
      2. tar xzf s3cmd-1.6.1.tar.gz
    2. Now install it using below command with source files.
      1. cd s3cmd-1.6.1
      2. sudo python setup.py install
    3. Configure s3cmd Environment:
      1. S3cmd –configure

Enter new values or accept defaults in brackets with Enter.

Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3

Access Key: xxxxxxxxxxxxxxxxxxxxxx

Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Encryption password is used to protect your files from reading

by unauthorized persons while in transfer to S3

Encryption password: xxxxxxxxxx

Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3

servers is protected from 3rd party eavesdropping. This method is

slower than plain HTTP and can’t be used if you’re behind a proxy

Use HTTPS protocol [No]: Yes

New settings:

Access Key: xxxxxxxxxxxxxxxxxxxxxx

Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Encryption password: xxxxxxxxxx

Path to GPG program: /usr/bin/gpg

Use HTTPS protocol: True

HTTP Proxy server name:

HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] Y

Please wait, attempting to list all buckets…

Success. Your access key and secret key worked fine 🙂

Now verifying that encryption works…

Success. Encryption and decryption worked fine 🙂

Save settings? [y/N] y

Configuration saved to ‘/root/.s3cfg’

Use command:

List allthe buckets: “S3cmd ls”

How to Mount S3 Bucket on CentOS using S3FS

  • S3FS is FUSE (File System in User Space) based solution to mount an Amazon S3 buckets, We can use system commands with this drive just like as another Hard Disk in system. On s3fs mounted files systems we can simply use cp, mv and ls the basic Unix commands similar to run on locally attached disks.
  1. Install Required Packages:
    1. yum install gcc libstdc++-devel gcc-c++ curl-devel libxml2-devel openssl-devel mailcap automake fuse-devel git libcurl-devel libxml2-devel make
  2. Download and Compile Latest Fuse:
    1. cd /usr/src/
    2. wget https://github.com/libfuse/libfuse/releases/download/fuse-2.9.7/fuse-2.9.7.tar.gz
    3. tar xzf fuse-2.9.3.tar.gz
  3. Change to the extracted directory:
    1. cd fuse-2.9.3
  4. Configure package:
    1. ./configure –prefix=/usr/local
  5. Compile and Install package:
    1. make && make install
    2. export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
    3. ldconfig
    4. modprobe fuse
  6. Download and compile Latest S3FS:
    1. Cd/usr/src/
    2. Wget https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/s3fs/s3fs-1.74.tar.gz
    3. Tar xzf s3fs-1.74.tar.gz
    4. Cd s3fs-1.74
    5. ./configure –prefix=/usr/local
    6. Make && make install
  7. Setup Security Access:
    1. Vim /etc/passwd-s3fs
      1. BucketName:AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY
      2. Note: Change AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with your actual key values.
    2. chmod 600 /etc/passwd-s3fs
  8. Setup Auto Mount (Fstab):
    1. mkdir /tmp/cache
    2. mkdir /mnt/s3mnt
    3. chmod 777 /tmp/cache /mnt/s3mnt
  9. Edit /etc/fstab:
    1. vim /etc/fstab
  10. Add a line at the end, in the following format:
    1. s3fs#edris-demo /mnt/edrismnt fuse _netdev,allow_other 0 0
  11. Almost Done, Time to Active:
    1. mount –afv or mount -a
      1. If you don’t get any errors, you’re good to go.
      2. s3fs mybucket /path/to/mountpoint -o passwd_file=/path/to/passwd
        1. For debug the error use this command:
        2. s3fs mybucket /path/to/mountpoint -o passwd_file=/path/to/passwd -d -d -f -o f2 -o curldbg
        3. Note: If you get any error like “nonempty” just use this command
          1. s3fs -o nonempty <bucket-name> </mount/point/>
  12. Upload the file and Check s3 mount/point.
  13. Troubleshooting:
    1. If the fuse mount fails for some reason you’ll receive the message ‘Transport endpoint is not connected’. To correct this remount the drive and fuse. Here’s the simple commands:
      1. sudo umount /mnt/aws_3s
      2. sudo mount -va /etc/fstab
        1. If that fails, try:
          1. sudo fusermount -u /mnt/aws_s3
          2. sudo mount –va
  14. Check the directory where you mount the bucket:
    1. If it’s shows the all objects then done
  15. Successfully S3 Bucket Mount to EC2-Centos-6.4

Anshul Modi.
AWS-team,
Electromech Corporation.

img_20150731_120617-copy

Leave a Reply

Your email address will not be published. Required fields are marked *

*