This blog provides utility function in the context of event driven cloud native architecture
AWS space:
- EMR creation using lambda
- EMR shutdown using lambda
- Use of SSM to run remote code in EC2 instance
Solution using above functions act together,creates event driven cloud native architecture in aws.
GCP Space
Big-query extraction service providing standard sql query and aws/gcp bucket name
- external system publish sql query and aws s3 path with delimiter “|” to gcp pub/sub
- Pub/sub triggers cloud function
- cloud function gets authenticated and call data-flow rest api
- cloud function extract data from big query
- compress and write data to csv files based on underline shard design in aws S3
Github repository for all utility function
AWS:
EMR creation using lambda : https://github.com/soumyaraja/on-demand-exe-hadoop.git
EMR shutdown using lambda: https://github.com/soumyaraja/Terminate-EMR-Cluster.git
Use of SSM to run remote code in EC2 instance : https://github.com/soumyaraja/AWS-SSM-through-Lambda.git
GCP:
Big-query extraction service https://github.com/soumyaraja/BigqueryExtract-Service-to-s3.git
