# 大数据代写 | COSC 2637/2633 Big Data Processing Assignment 2

## Overview

Write advanced MapReduce programs which give your chance to develop in-depth understanding of
principles when solving complex problems on Hadoop execution platform, and analyze solutions by applying
the knowledge learned in this course to achieve the optimal outcome.

## Learning Outcomes

The key course learning outcomes are:

CLO 1: model and implement efficient big data solutions for various application areas using appropriately
selected algorithms and data structures.

CLO 2: analyse methods and algorithms, to compare and evaluate them with respect to time and space
requirements and make appropriate design choices when solving real-world problems.

CLO 3: motivate and explain trade-offs in big data processing technique design and analysis in written and
oral form.

CLO 4: explain the Big Data Fundamentals, including the evolution of Big Data, the characteristics of Big
Data and the challenges introduced.

CLO 6: apply the novel architectures and platforms introduced for Big data, i.e. Hadoop, MapReduce and
Spark.

## Assessment details

### Task 1 – Count word co-occurrence frequency (10 marks)

Write a MapReduce program that uses pairs approach and outputs the frequency of word pairs.

– Given “(a, b)” and word pair “(b, a)”, they are considered as different word pairs,
– Do not output count the pair of same words, e.g., “(a, a)”,
– The words are considered co-occurred if they are in the same line and the number of words between them <=3.

### Task 2 – Count word pair relative frequency (10 marks)

Write a MapReduce program that uses pairs approach and outputs the relative frequency of word pairs.

– Given “(a, b)” and word pair “(b, a)”, they are considered as different word pairs,
– Do not output count the pair of same words, e.g., “(a, a)”,
– The words are considered co-occurred if they are in the same line and the number of words between them <=3.

### Task 3 – Implement PAM algorithm with a MapReduce Program (20 marks)

The most common realization of k-medoid clustering is the Partitioning Around Medoids (PAM) algorithm
which is described below:

(a) Your program must correct implement PAM. In your code, provide detailed comments to specify
where each step is implemented. For example

//Step 2 start.

Block of code;

//Step 2 end.

Run your PAM MapReduce program to cluster a point dataset NYTaxiLC1000
(with 1000 points in
longitude and latitude from line 1 to line 1000) where 1 ≤ 𝑘𝑘 ≤ 6. Note the initial medoids are always points
at line 100, 200, 300, 400, 500 and 600 (i.e., 𝑘𝑘 = 1, the initial medoid is point at line 100; 𝑘𝑘 = 2, the initial
medoids are points at line 100, 200; and so on for k=3, 4, 5 and 6).

(b) Visualize the clustering results. The points belonging to the same cluster are with the same color.
The medoid of each cluster is highlighted.

(c) Analyse what is the best setting of 𝑘𝑘 (3 ≤ 𝑘𝑘 ≤ 6) and explain why.

step 1. Initialize: randomly select 𝑘𝑘 of the 𝑛𝑛 data points as the medoids

step 2. Assignment step: Associate each data point to the closest medoid.

step 3. Update step: For each medoid m and each data point 𝑜𝑜 associated to 𝑚𝑚 , swap 𝑚𝑚 and 𝑜𝑜, and compute the
total cost of the configuration (that is, the average dissimilarity of 𝑜𝑜 to all the data points associated
to 𝑚𝑚). Select the medoid o with the lowest cost of the configuration.

step 4. Repeat alternating steps 2 and 3 until there is no change in the assignments. E-mail: itcsdx@outlook.com  微信:itcsdx 