How to push local file to s3 using AWS CLI ?

Pre Steps :

- Install AWS CLI , Please visit the following post if you haven't installed already. (http://www.fullstacktechnos.com/2017/07/how-to-install-aws-cli-in-mac.html)

Steps :

1) Create an user with access to s3 using aws IAM service.

i) Go to AWS console and search for service IAM



ii) Visit the Users section and click on "Add User"




iii) Enter the user name (for ex: s3-user or demo in my example) and check the Access Type as Programming Access and click "Next Permission"

iv) Attach Policy (so that user can access s3) by clicking "Attach Existing Policies directly".
Note : If you want you can create a group and attach user to the specific group having necessary permission.



v) Choose the policy "Amazons3FullAccess"



vi) Review and Click "Create User"













vii) Copy the access Key Id and Secret Access Key



2) Come to the Mac console and type "aws configure"and enter the keys details.

$ aws configure
AWS Access Key ID [****************QQNA]:
AWS Secret Access Key [****************GIel]:
Default region name [None]:
Default output format [None]:

3) s3 related commands -

Type the following command to see all the buckets created in s3

$ aws s3 ls
2017-07-30 20:09:09 raja-lamda-trigger-demo
2017-07-31 01:19:13 serverless-hello-world-d-serverlessdeploymentbuck-bib12jl57p6m

Create bucket

$ aws s3 mb s3://fst-demo-test
make_bucket: fst-demo-test
$ aws s3 ls | grep fst
2017-07-31 14:35:50 fst-demo-test

Check the AWS s3 console.



Remove Bucket

$ aws s3 rb s3://fst-demo-test
remove_bucket: fst-demo-test
$ aws s3 ls | grep fst
$

4) Copy local file to s3 bucket.

$ ls test.csv
test.cs
$ aws s3 cp test.csv s3://fst-demo-test
upload: ./test.csv to s3://fst-demo-test/test.csv

List files in console

$ aws s3 ls s3://fst-demo-test
2017-07-31 14:42:30         24 test.csv
$

Check the aws s3 console

How to install AWS CLI in Mac ?

Steps:

1) Install latest version of Python (https://www.python.org/downloads/)

$ python3

Python 3.6.2 (v3.6.2:5fd33b5926, Jul 16 2017, 20:11:06) 

[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> 

2) Install PIP (Python Package Index)

i) curl -O https://bootstrap.pypa.io/get-pip.py

$ curl -O https://bootstrap.pypa.io/get-pip.py

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100 1558k  100 1558k    0     0   419k      0  0:00:03  0:00:03 --:--:--  419k

$ ls get-pip.py 

get-pip.py

ii) python3 get-pip.py --user

$ python3 get-pip.py --user

Requirement already up-to-date: pip in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages

Collecting wheel

  Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)

    100% |████████████████████████████████| 71kB 544kB/s 

Installing collected packages: wheel

Successfully installed wheel-0.29.0

$ pip3 --version

pip 9.0.1 from /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (python 3.6)

3) Install AWS CLI using PIP

pip3 install --user --upgrade awscli

$ pip3 install --user --upgrade awscli

Collecting awscli

  Downloading awscli-1.11.127-py2.py3-none-any.whl (1.2MB)

    100% |████████████████████████████████| 1.2MB 
....
....

Building wheels for collected packages: PyYAML

  Running setup.py bdist_wheel for PyYAML ... done

  Stored in directory: /Users/raja/Library/Caches/pip/wheels/2c/f7/79/13f3a12cd723892437c0cfbde1230ab4d82947ff7b3839a4fc

Successfully built PyYAML

Installing collected packages: docutils, colorama, jmespath, six, python-dateutil, botocore, pyasn1, rsa, s3transfer, PyYAML, awscli

Successfully installed PyYAML-3.12 awscli-1.11.127 botocore-1.5.90 colorama-0.3.7 docutils-0.13.1 jmespath-0.9.3 pyasn1-0.3.1 python-dateutil-2.6.1 rsa-3.4.2 s3transfer-0.1.10 six-1.10.0

4) Verify AWS CLI installation

$ aws

-bash: aws: command not found

$ 

Here in my case executable is not found, so need to add the aws executable to OS's PATH environment variable.

Following is the path.

$ ~/Library/Python/3.6/bin/

__pycache__/           aws_zsh_completer.sh   pyrsa-encrypt-bigfile  rst2html.py            rst2odt_prepstyles.py  rstpep2html.py

aws                    jp.py                  pyrsa-keygen           rst2html5.py           rst2pseudoxml.py       wheel

aws.cmd                pyrsa-decrypt          pyrsa-priv2pub         rst2latex.py           rst2s5.py              

aws_bash_completer     pyrsa-decrypt-bigfile  pyrsa-sign             rst2man.py             rst2xetex.py        

aws_completer          pyrsa-encrypt          pyrsa-verify           rst2odt.py             rst2xml.py             

In this example i am adding the path to my "bash_profile" (~/.bash_profile)

PATH="~/Library/Python/3.6/bin:$PATH"

$ source ~/.bash_profile

Type aws as command to verify.

$ aws

usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]

To see help text, you can run:

  aws help

  aws <command> help

  aws <command> <subcommand> help

aws: error: the following arguments are required: command

** AWS CLI is installed successfully **