Event Based Runtime migration platform As a Service on Public Cloud

Event Based Runtime Infrastructure As a Service on Public Cloud

What and How Space

Cloud infrastructure provides the easy way to manage vms including creation/termination and provides language specific api — but what about cost effective way to use these infrastructure- isn’t it good enough if infrastructure is only available for processing data and goes down when not required and the process can be managed by user/group

Data processing lake can be created based on a single notification or serious of notification

Strategic Architecture Diagram

The above architecture diagram provides the combination/all possible view to create and manage execution environment.There could be many use cases that architecture can support

Use case: On Demand cost effective data analytics or ETL tool platform and send the result to different target system

1. Create object storage adapter on on prem/cloud ( plug and )
2. Create notification to cloud function to create hadoop cluster -function will return cluster id as response
3. Hadoop cluster can retrieve data from on prem/object storage, process the data and sends output to object storage/on prem location
4. Cluster can send notification to create cloud vm/initiate container managed platform or run code on vm
5. Cluster termination can be optional — all cloud provider provides auto termination wrapper api to terminate cluster

other popular use cases:

1. On Demand Container task based on notification

AWS Specific view:

 

EMR: wrapper service provide by AWS for managing hadoop cluster with support of language specific api

System Manager : can manage on prem and cloud vm and can able to run remote code execution/ if possible code can be resides in object storage/s3

Lambda function :can be used to create/terminate cluster and even start ec2 instance initiate ssm client to run code on and can shutdown/terminate ec2 and the same can be used for initiate container task as well.

Conclusion:

On demand execution platform can be used only on specific use cases -it is cost effective way to use cloud infrastructure and event driven provides the flexibility to run on any time rather using a scheduler

Centralised log solution shall be used in the view of both cloud and on prem systems