Event Driven Cloud Native Architecture

Posted on by

This blog provides utility function in the context of event driven cloud native architecture

AWS space:

  • EMR creation using lambda
  • EMR shutdown using lambda
  • Use of SSM to run remote code in EC2 instance

Solution using above functions act together,creates event driven cloud native architecture in aws.

Event based data cloud native architecture in aws

GCP Space

Big-query extraction service providing standard sql query and aws/gcp bucket name

  • external system publish sql query and aws s3 path with delimiter “|” to gcp pub/sub
  • Pub/sub triggers cloud function
  • cloud function gets authenticated and call data-flow rest api
  • cloud function extract data from big query
  • compress and write data to csv files based on underline shard design in aws S3
Big-query extraction service by providing sql query and bucket name

Github repository for all utility function

AWS:

EMR creation using lambda : https://github.com/soumyaraja/on-demand-exe-hadoop.git

EMR shutdown using lambda: https://github.com/soumyaraja/Terminate-EMR-Cluster.git

Use of SSM to run remote code in EC2 instance : https://github.com/soumyaraja/AWS-SSM-through-Lambda.git

GCP:

Big-query extraction service https://github.com/soumyaraja/BigqueryExtract-Service-to-s3.git

Author

Leave a Reply