Uploading Files To S3 Using Python

I have a list of file URLs which are download links. I have written Python code to download the files to my computer. Here's the problem, there are about 500 files in the list and

Solution 1:

Its quite simple using python3 and boto3 (AWS SDK), eg.:

import boto3

s3 = boto3.client('s3')
with open('filename.txt', 'rb') asdata:
    s3.upload_fileobj(data, 'bucketname', 'filenameintos3.txt')

for more information you can read boto3 documentation here: http://boto3.readthedocs.io/en/latest/guide/s3-example-creating-buckets.html

Enjoy

Solution 2:

If you have the aws cli installed on your system you can make use of subprocess library. For example:

import subprocess
defcopy_file_to_s3(source: str, target: str, bucket: str):
   subprocess.run(["aws", "s3" , "cp", source, f"s3://{bucket}/{target}"])

Similarly you can use that logics for all sort of AWS client operations like downloading or listing files etc. This way there is no need to import Boto3. I guess its use is not intended that way but in practice I find it quite convenient that way. This way you also get the status of the upload displayed in your console - for example:

Completed 3.5 GiB/3.5 GiB (242.8 MiB/s) with 1 file(s) remaining

To modify the method to your wishes I recommend having a look into the subprocess reference as well as to the AWS Cli reference.