2023年11月25日

大数据代写 | COSC 2637/2633 Big Data Processing Assignment 2

本次澳洲代写是B+大数据Map Reduce编程的一个assignment

Overview

Write advanced MapReduce programs which give your chance to develop in-depth understanding of
principles when solving complex problems on Hadoop execution platform, and analyze solutions by applying
the knowledge learned in this course to achieve the optimal outcome.

Learning Outcomes

The key course learning outcomes are:

CLO 1: model and implement efficient big data solutions for various application areas using appropriately
selected algorithms and data structures.

CLO 2: analyse methods and algorithms, to compare and evaluate them with respect to time and space
requirements and make appropriate design choices when solving real-world problems.

CLO 3: motivate and explain trade-offs in big data processing technique design and analysis in written and
oral form.

CLO 4: explain the Big Data Fundamentals, including the evolution of Big Data, the characteristics of Big
Data and the challenges introduced.

CLO 6: apply the novel architectures and platforms introduced for Big data, i.e. Hadoop, MapReduce and
Spark.

Assessment details

Task 1 – Count word co-occurrence frequency (10 marks)

Write a MapReduce program that uses pairs approach and outputs the frequency of word pairs.

– Given “(a, b)” and word pair “(b, a)”, they are considered as different word pairs,
– Do not output count the pair of same words, e.g., “(a, a)”,
– The words are considered co-occurred if they are in the same line and the number of words between them <=3.

Task 2 – Count word pair relative frequency (10 marks)

Write a MapReduce program that uses pairs approach and outputs the relative frequency of word pairs.

– Given “(a, b)” and word pair “(b, a)”, they are considered as different word pairs,
– Do not output count the pair of same words, e.g., “(a, a)”,
– The words are considered co-occurred if they are in the same line and the number of words between them <=3.

Task 3 – Implement PAM algorithm with a MapReduce Program (20 marks)

The most common realization of k-medoid clustering is the Partitioning Around Medoids (PAM) algorithm
which is described below:

(a) Your program must correct implement PAM. In your code, provide detailed comments to specify
where each step is implemented. For example

//Step 2 start.
…
Block of code;
…
//Step 2 end.

Run your PAM MapReduce program to cluster a point dataset NYTaxiLC1000
(with 1000 points in
longitude and latitude from line 1 to line 1000) where 1 ≤ ?? ≤ 6. Note the initial medoids are always points
at line 100, 200, 300, 400, 500 and 600 (i.e., ?? = 1, the initial medoid is point at line 100; ?? = 2, the initial
medoids are points at line 100, 200; and so on for k=3, 4, 5 and 6).

(b) Visualize the clustering results. The points belonging to the same cluster are with the same color.
The medoid of each cluster is highlighted.

(c) Analyse what is the best setting of ?? (3 ≤ ?? ≤ 6) and explain why.

step 1. Initialize: randomly select ?? of the ?? data points as the medoids

step 2. Assignment step: Associate each data point to the closest medoid.

step 3. Update step: For each medoid m and each data point ?? associated to ?? , swap ?? and ??, and compute the
total cost of the configuration (that is, the average dissimilarity of ?? to all the data points associated
to ??). Select the medoid o with the lowest cost of the configuration.

step 4. Repeat alternating steps 2 and 3 until there is no change in the assignments.