bigdata project

In this project, you will be asked to work on the ETL concept. ETL stands for Extract, Transform, and Load process. The extract phase may refer to the process of obtaining and mine the data required for the analysis project. This may include some cleaning and combining of the data. The transform phase is the process to make the acquired data comply with the format you are planning to use in the future steps. The load phase is about shipping the data into the enterprise systems (e.g. the database). In this project our focus is on the first two stages, namely Extract and Transform. We may see the last stage in a future assignment. You will be given a group of files. These files represent the customer reviews of a group of products. These products are: Canon G3 camera, Dvd player, Jukebox, Nikon Coolpix, and Nokia 6610. These review files are semi-structured in a format very specific to the website that generated this data. Usually this format limits the gains that can be attained from this data. To overcome this, you need to change this format into a popular format which is the JSON format. Please see below: 1. General Information: Data Description a. You will be given a folder that has the review data files. Each file is fully about one product. b. Each file is semi-structured in the following manner: i. The top of the file has a block of information related to the source of the data. This part is important but it is not of an interest for us in the moment. 1. This block will be ignored. ii. Any review starts with [t] followed by the review title 1. Ex: [t]great camera iii. Any positive aspects will start with [+n], where n can be 1, 2, 3….or any number of points representing a good score. Followed by ## and then the text of the review. 1. Ex: [+2]##i have only had this camera for one full day and i have to say that it is wonderful . iv. Any negative aspects will start with [-m], where m can be 1, 2, 3….or any number of points representing a bad score. Followed by ## and then the text of the review. 1. Ex: [-1]##* main dial is not backlit . 2. First part: Extract, Clean, and Combine a. In this part you will extract the data from the files given in point 1 above. b. In the extract process, you have to identify the product that is being reviewed. c. After identifying the product, each review has to be identified by the title. d. In a review you will identify the positive and negative aspects. To distinguish both, you can look and point 1 above. i. Extract the positive aspects of a review ii. Extract the negative aspects of a review. e. You should know that the text of any review, title and content, will not be clean. The cleaning should be done in the following manner: i. The text should not contain any special characters (!,@,#,$,%,^,&,”,’,<,>,/,? and *) ii. Also speech punctuation symbols have to be removed (, ; : \t) iii. The reviews are written in English. If for any reason a review has a language other than English, you can ignore such a review and do not include it in the output. f. After cleaning all the positive aspects, these will be combined in one single text block of positive criticism. g. The same will be done to the negative aspects. 3. Second part: Data Transformation a. After extracting, cleaning, and combining every review, the data will be put into the outpu