Monday, July 31, 2017

How to push local file to s3 using AWS CLI ?

Pre Steps :

- Install AWS CLI , Please visit the following post if you haven't installed already. (

Steps :

1) Create an user with access to s3 using aws IAM service.

i) Go to AWS console and search for service IAM

ii) Visit the Users section and click on "Add User"

iii) Enter the user name (for ex: s3-user or demo in my example) and check the Access Type as Programming Access and click "Next Permission"

iv) Attach Policy (so that user can access s3) by clicking "Attach Existing Policies directly".
Note : If you want you can create a group and attach user to the specific group having necessary permission.

v) Choose the policy "Amazons3FullAccess"

vi) Review and Click "Create User"

vii) Copy the access Key Id and Secret Access Key

2) Come to the Mac console and type "aws configure"and enter the keys details.

$ aws configure
AWS Access Key ID [****************QQNA]:
AWS Secret Access Key [****************GIel]:
Default region name [None]:
Default output format [None]:

3) s3 related commands -

Type the following command to see all the buckets created in s3

$ aws s3 ls
2017-07-30 20:09:09 raja-lamda-trigger-demo
2017-07-31 01:19:13 serverless-hello-world-d-serverlessdeploymentbuck-bib12jl57p6m

Create bucket

$ aws s3 mb s3://fst-demo-test
make_bucket: fst-demo-test
$ aws s3 ls | grep fst
2017-07-31 14:35:50 fst-demo-test

Check the AWS s3 console.

Remove Bucket

$ aws s3 rb s3://fst-demo-test
remove_bucket: fst-demo-test
$ aws s3 ls | grep fst

4) Copy local file to s3 bucket.

$ ls test.csv
$ aws s3 cp test.csv s3://fst-demo-test
upload: ./test.csv to s3://fst-demo-test/test.csv

List files in console

$ aws s3 ls s3://fst-demo-test
2017-07-31 14:42:30         24 test.csv

Check the aws s3 console

How to install AWS CLI in Mac ?


1) Install latest version of Python (

$ python3

Python 3.6.2 (v3.6.2:5fd33b5926, Jul 16 2017, 20:11:06) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.


2) Install PIP (Python Package Index)

i) curl -O

$ curl -O
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

100 1558k  100 1558k    0     0   419k      0  0:00:03  0:00:03 --:--:--  419k
$ ls

ii) python3 --user

$ python3 --user
Requirement already up-to-date: pip in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages
Collecting wheel
  Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
    100% |████████████████████████████████| 71kB 544kB/s 
Installing collected packages: wheel

Successfully installed wheel-0.29.0
$ pip3 --version
pip 9.0.1 from /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (python 3.6)

3) Install AWS CLI using PIP

pip3 install --user --upgrade awscli

$ pip3 install --user --upgrade awscli
Collecting awscli
  Downloading awscli-1.11.127-py2.py3-none-any.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 

Building wheels for collected packages: PyYAML
  Running bdist_wheel for PyYAML ... done
  Stored in directory: /Users/raja/Library/Caches/pip/wheels/2c/f7/79/13f3a12cd723892437c0cfbde1230ab4d82947ff7b3839a4fc
Successfully built PyYAML
Installing collected packages: docutils, colorama, jmespath, six, python-dateutil, botocore, pyasn1, rsa, s3transfer, PyYAML, awscli

Successfully installed PyYAML-3.12 awscli-1.11.127 botocore-1.5.90 colorama-0.3.7 docutils-0.13.1 jmespath-0.9.3 pyasn1-0.3.1 python-dateutil-2.6.1 rsa-3.4.2 s3transfer-0.1.10 six-1.10.0

4) Verify AWS CLI installation

$ aws
-bash: aws: command not found

Here in my case executable is not found, so need to add the aws executable to OS's PATH environment variable.

Following is the path.
$ ~/Library/Python/3.6/bin/
__pycache__/    pyrsa-encrypt-bigfile  
aws                            pyrsa-keygen         wheel
aws.cmd                pyrsa-decrypt          pyrsa-priv2pub               
aws_bash_completer     pyrsa-decrypt-bigfile  pyrsa-sign              
aws_completer          pyrsa-encrypt          pyrsa-verify                 

In this example i am adding the path to my "bash_profile" (~/.bash_profile)


$ source ~/.bash_profile
Type aws as command to verify.

$ aws
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help

aws: error: the following arguments are required: command

** AWS CLI is installed successfully **