Event Based Runtime Infrastructure As a Service on Public Cloud
What and How Space
Cloud infrastructure provides the easy way to manage vms including creation/termination and provides language specific api — but what about cost effective way to use these infrastructure- isn’t it good enough if infrastructure is only available for processing data and goes down when not required and the process can be managed by user/group
Data processing lake can be created based on a single notification or serious of notification
Strategic Architecture Diagram
The above architecture diagram provides the combination/all possible view to create and manage execution environment.There could be many use cases that architecture can support
Use case: On Demand cost effective data analytics or ETL tool platform and send the result to different target system
- Create object storage adapter on on prem/cloud ( plug and )
- Create notification to cloud function to create hadoop cluster -function will return cluster id as response
- Hadoop cluster can retrieve data from on prem/object storage, process the data and sends output to object storage/on prem location
- Cluster can send notification to create cloud vm/initiate container managed platform or run code on vm
- Cluster termination can be optional — all cloud provider provides auto termination wrapper api to terminate cluster
other popular use cases:
- On Demand Container task based on notification
AWS Specific view:
EMR: wrapper service provide by AWS for managing hadoop cluster with support of language specific api
System Manager : can manage on prem and cloud vm and can able to run remote code execution/ if possible code can be resides in object storage/s3
Lambda function :can be used to create/terminate cluster and even start ec2 instance initiate ssm client to run code on and can shutdown/terminate ec2 and the same can be used for initiate container task as well.
Conclusion:
On demand execution platform can be used only on specific use cases -it is cost effective way to use cloud infrastructure and event driven provides the flexibility to run on any time rather using a scheduler
Centralised log solution shall be used in the view of both cloud and on prem systems
