Distributed Analytic Engine
Table of Contents
🧾 Description
Distributed computing engine that enable clients to submit jobs for processing via simple HTTP requests. The system uses a Master-Worker model, with the Master node receiving job requests and distributing them to Worker nodes for processing.
Once a Worker node receives a job, it utilizes multithreading and parallelization to execute the job quickly and efficiently. The system supports a range of normal operations, such as FlatMap, FoldByKey, Join, and more, allowing clients to perform complex data processing tasks with ease.
To support distributed computing, the engine relies on the webserver and key-value store I built earlier. Making it suitable for processing large datasets.
✨ Features
- ✅ Client could submit jobs using HTTP requests
- ✅ Employed Master-Workers model
- ✅ Once worker received request, It will utilize multithread and parllelize operations.
- ✅ Support normal operations (Ex. FlatMap, FoldByKey, Join, …etc)