Conclusion
Table of contents
Congratulations on finishing the assignment! Here are some optional, not-for-credit things you can do to improve your MapReduce system.
These are only for fun; there is no extra credit provided.
Freeing intermediate results
At the moment, workers buffer map task outputs indefinitely. This means that they will eventually run out of memory. Implement logic to free intermediate map buffers once a job has completed (or fails).
You can add additional RPCs to handle this, or modify your existing RPCs.
MapReduce applications
Try implementing your own MapReduce application! You can get ideas for what to implement from the MapReduce paper.
Look at src/app/grep.rs and src/app/wc.rs to see how to fit a MapReduce application into the framework we provide. You’ll also need to modify src/app/mod.rs to allow workers to find your new application.