Using AWS CloudFront Path-Based behaviour for uploading to multiple S3 Buckets with CF and S3 resigned URLs

TLDR


In a nutshell, this allows you to: 
  • Replace multiple S3 endpoint with a single custom CloudFront domain (e.g., for branding reasons)
  • Use Path-Based behaviour in CloudFront to handle destination bucket (e.g., for app compatibility reasons)
  • Combine the following:
    • CloudFront Presigned URL - an "outer auth" for the request to be let in
    • S3 Presigned Post - an "inner auth" against S3 for to POST request to be processed

Design Diagram



Points to note: 

  • S3 Presigned Post URL
    • The S3 endpoint generated is discarded
    • The POST field containing signature (S3 Auth), policy, bucket name and key related to the upload is retained
  • CloudFront Signed URL
    • The URL is made up of CloudFront custom domain + Path of each bucket's associated CloudFront behaviour
    • CloudFront signer adds query string to the URI with signatures, expirations, etc
    • The Signed URL forms the outer credentials (CloudFront auth) to access the CloudFront behaviour
  • These two are then combined and sent to the client, and the client POST the content to the CloudFront distribution
    • See flow above for how the request is combined
  • On the CloudFront side, a Javascript based CloudFront function is placed before the origin to strip out the URI (e.g., `/upload/bucket1`). 
    This is necessary so that S3 endpoint process the request as a S3 REST API call, instead of acting as a Website Endpoint.

Considerations

The main purpose of this design is mostly enterprise branding and for endpoint to be consolidated.
Performance might take a small impact, but bandwidth cost will be a major consideration. 

During my super non-scientific testing, upload through CloudFront will never exceed that of direct to S3 endpoint, except for endpoints that is on the other side of the planet. (Tested from a domestic FTTP multi-gig connection.)

You will have to pay for all the bandwidth of the upload, which might be significant depending on your use case. 


Comments

Popular posts from this blog

Android Samsung Print Service Plugin - unable to print

UK Virgin Media Hub 4 Link Aggregation is very broken (balance-rr)

How to check Nexus 6 Manufacture Date