How to download a file from a public (no credentials) AWS S3 bucket with python ?

Created October 12, 2022 | Edited November 14, 2022 | Viewed 341 | by Benjamin Edit

Example of how to download a file from a public, no credentials required here, AWS S3 bucket with python ?

Find an open data project

Let's consider the NOAA public s3 bucket. We want to download a file from the following folder SNPP/VIIRS/SNPP_AF_I-Band_EDR_NRT/2022/09/09/.

Install boto3

To download a file from a aw S3 bucket with python, a solution is to use boto3. To install boto3 with anaconda:

conda install -c anaconda boto3

Find all files in a S3 folder

A first step is often to list all files in a S3 bucket folder. To do that a solution is to do

import boto3

from botocore import UNSIGNED
from botocore.config import Config

s3 = boto3.resource('s3', config=Config(signature_version=UNSIGNED))

bucket = "noaa-jpss"
folder = "SNPP/VIIRS/SNPP_AF_I-Band_EDR_NRT/2022/09/09/"

s3_bucket = s3.Bucket(bucket)

files_in_s3 = [f.key.split(folder + "/")[0] for f in s3_bucket.objects.filter(Prefix=folder).all()]

Note that UNSIGNED has been used here since no credentials are required.

returns here:


Download a given file

Now to download a file, a solution is to use the boto3 function download_file()

s3 = boto3.client('s3', config=Config(signature_version=UNSIGNED))

bucket_name = "noaa-jpss"
bucket_dir = "SNPP/VIIRS/SNPP_AF_I-Band_EDR_NRT/2022/09/10/"

filename = ''

s3.download_file(Filename=filename,Bucket=bucket_name,Key=bucket_dir + filename)

See also

Links |
------------- | -------------
How to get NOAA VIIRS Active Fires products ? | NOAA Class