本次Python代写是通过大数据分析来对电影评论进行情感分类，把特征提取到分类

301110 Applications of Big Data
Assignment
3 Task 1. Feature extraction (15 points)
Use the MapReduce model to convert all text data into matrices. Convert ratings
to vectors. Œese will be used for classi€cation in Task 2. Use TF-IDF to vectorise
the text €les. See previous practical classes and lectures materials for TF-IDF.
One step further though is to represent each text €le (review) as a very long
and sparse vector as the following. Assume wordslist is the €nal list of distinct
words contained in all reviews and its length is N. Œen each review will be a
vector of length N, with each position associated with the word in wordlist and
the value being either 0, if the corresponding word is absent in the review, or the
word’s TF-IDF. For example, if wordlist = [‘word1’, ‘word2’, ‘word3’, ‘word4’] and
review 1 contains word1 and word4, then the vector representation of review 1
is [0.1, 0, 0, 0.4] assuming TF-IDF of word 1 and word 4 in review 1 is 0.1 and 0.4
respectively. Note that TF is calculated from one single document while IDF is
obtained from all documents in the collection.
3.1 Requirements:
3.1.1 Req. 1
A Map-reduce model is a must. Implement it using Hadoop streaming. All data
are available on SCEM HDFS. Œe recommendation is to work on the tiny version
of the data to make the code work. You may try your code on the full version.
However, the application to full version is not required.
3.1.2 Req. 2
Generate two matrices: training data, training targets, and two vectors:
test data, test targets. training data should have N rows and D columns
with each row corresponding to each review in the training set (N is the total
number of reviews in the training set and D is the total number of words). N
and D vary depending on which version of the data you use. training targets
should have N elements each of which is the rating of the review. test data
and test targets are similarly de€ned.
Notes:
1. If feature extraction is too dicult for you, you can use pre-computed
bag of words features included in this data set. Refer to the appendix and
README €le for details. If pre-computed features are used, a 60% penalty
will be incurred for this task, i.e. the maximum marks you can get from
this task is 6 if you do so.
2
2. Using a map-reduce model to extract TF-IDF is mandatory. If not used, a
20% penalty for this task will be incurred. Œere is no constraint on how to
form the training and test matrices and vectors. Œere are many versions
of TF-IDF. Œere is no preference for which version to use.
3. You can use data frame (using pandas package) instead of matrices and
vectors to store training and test data and targets.
Marking scheme for task 1:
• Text €le reading (1pt): read the text €les for TF-IDF extraction.
• Rating scores extraction (3pts): parse the name of text €les to extract ratings.
• TF-IDF extraction (8pts): use MapReduce class to extract TF-IDF for each
text €le.
• Forming matrices and target vectors (or data frames) (3pts): collect TF-IDFs
to form training and test data for task 2.
4 Task 2. Classi€cation (15 points)
Construct a classi€cation model for review sentiment prediction, meaning that:
given a customer movie review (taken from the test set), your program should be
able to predict whether it is positive or negative.
Œere is no limitation on how many classi€ers and what speci€c model you
should use. You can simply pick one that works for you for this task, either
from those covered in lecture and practical class materials or any other classi-
€ers from any python packages. A good starting point is the scikit-learn (i.e.
sklearn) package.
A few things you need to address in your python program are listed as requirements below.
4.1 Requirements:
4.1.1 Req. 1
Data pre-processing. In task 1, you extracted the ratings vectors for training
and test. Œese are raw ratings. As we are interested in sentiment prediction,
i.e. to predict either the review is positive or negative, you need to convert all
ratings >5 as positive class and all ratings <=5 as negative class. Choose a
coding scheme, e.g. 0 for positive, 1 for negative.
3
4.1.2 Req. 2
Normalisation. Apply at least one normalisation scheme and compare the performance of the classi€er(s) with and without normalisation.
4.1.3 Req. 3
Training and model selection. Use cross validation to select the best parameters
for your classi€er. Œere may be many parameters to tune in some classi€ers
(such as random forest classi€er — RFC). You can focus on the most important
one(s) such as max depth and n estimators in RFC. Refer to the scikit-learn
package documentation for details.
Hint: you can start with a small subset of the training set to test a few parameters
to get a feel of what range the parameters should be that make the model perform
well in terms of prediction accuracy. Šen turn on large scale cross validation on
the whole training set.
4.1.4 Req. 4
Test on test data. A‰er model selection, apply the best model, i.e. the model with
the parameters that produce the best cross validation scores, to test data, make
a prediction for each review, and record prediction accuracy
. Note:
1. Always train your classi€er(s) ONLY on training data including cross validation. A‰er model selection, apply the best model on test data to evaluate
the performance.
2. Good performance, i.e. higher accuracy on test data, is not essential for
this task. However, if your classi€er has accuracy lower than about 60%, it
usually means that there are some mistakes somewhere in your code. So
try to score as high an accuracy as possible.
3. You are encouraged to try many classi€ers. If the coding is right, this
should not be too dicult.
Marking scheme for task 2:
• Data pre-processing (1pts): convert ratings to positive and negative coding
scheme.
• Normalisation and comparison (3pts): apply normalisation and compare
performance di‚erence with and without it.