Code

This might look like a lot of code, but much of it is already written for you. You will only need to understand and modify around 3 files to complete this assignment.

.
├── Cargo.lock
├── Cargo.toml
├── build.rs
├── data
│   ├── alphabet2
│   │   └── ...
│   ├── graph-edges-medium
│   │   └── ...
│   └── gutenberg
│       └── ...
├── proto
│   ├── coordinator.proto
│   └── worker.proto
├── rust-toolchain.toml
└── src
    ├── app
    │   ├── grep.rs
    │   ├── mod.rs
    │   ├── vertex_degree.rs
    │   └── wc.rs
    ├── autograder
    │   └── mod.rs
    ├── bin
    │   ├── autograder.rs
    │   ├── client.rs
    │   ├── coordinator.rs
    │   └── worker.rs
    ├── client.rs
    ├── codec
    │   └── mod.rs
    ├── coordinator
    │   ├── args.rs
    │   └── mod.rs
    ├── lib.rs
    ├── log.rs
    ├── proto
    │   ├── coordinator.rs
    │   └── worker.rs
    ├── task
    │   └── mod.rs
    ├── tests
    │   └── mod.rs
    ├── utils.rs
    └── worker
        ├── args.rs
        └── mod.rs

data/

Data files for testing your MapReduce implementation. The alphabet2 and gutenberg folders contain data that can be used with the wc (word count) and grep applications, while the graph-edges-medium folder contains data for the vertex-degree application.

proto/

Protocol buffers for the coordinator and worker gRPC servers. The necessary protocols have been defined for you, so you may not modify these files. However, you are encouraged to look through the provided data structures.

src/app/

MapReduce applications such as word count and distributed grep. You may not modify any of the existing files in this folder, although you are free to test out your own MapReduce apps if you like.

src/bin/

Code for the binaries that are generated by cargo build. You may not modify the code in this folder, but feel free to take a look at the client if you are curious how it is implemented.

client.rs

Helper functions for the MapReduce client.

src/codec/

Helper functions for serialization, which will come in handy when storing and retrieving data. You may not modify the code in this folder as our tests depend on it. The doc comments describe how to use the code in this module.

src/coordinator/

Code for the MapReduce coordinator. The majority of your changes will be likely be made in the mod.rs file, but you are free to create additional files to organize your code as you see fit.

src/lib.rs

Helper functions, types, and constants that may come in useful for your implementation. You may not modify this file, but you are encouraged to take a look at its contents.

src/log.rs

Logging utilities to help you debug your code.

src/proto/

Autogenerated code for the coordinator and worker gRPC stubs. The coordinator and worker stub modules are included in the map_reduce crate structure in src/lib.rs here:

/// Automatically generated utilities for RPCs.
pub mod rpc {
    /// Includes RPC definitions for the coordinator.
    pub mod coordinator {
        include!("proto/coordinator.rs");
    }
    /// Includes RPC definitions for the worker.
    pub mod worker {
        include!("proto/worker.rs");
    }
}

You can thus import their contents with crate::rpc::coordinator and crate::rpc::worker, respectively.

src/task

Module for encapsulating logic related to task and job management. You are not required to use it, but it may help keep your code clean.

src/tests

Local MapReduce tests. We provide you with one basic test case, but you are free to add your own here. These tests are not run by the autograder.

src/utils.rs

Miscellaneous helper functions.

src/worker/

Code for MapReduce workers. The worker has been fully implemented for you, so you may not modify these files.