two projects about data mining strategies and some ML methods

The final project has 7 questions. Each question has 1 readme file. The question requirement are in the word.

Background: Detecting falls in pets using a wearable device that records signals corresponding to various variables.

Requirements: Handle data imbalance and prioritize recall rate for class 1 (falls).

Solving Approach:

• Data exploration to understand the variables and the distribution.
• Use of outlier detection algorithms, potentially Isolation Forest or One-Class SVM, considering the imbalance.

Background:

Developing an image classifier for different weather conditions based on raw image data.

Requirements:

Design a dataloader and train a deep learning model for at least 50 epochs.

Solving Approach:

• Image preprocessing including resizing and one-hot encoding of labels.
• Selection of a CNN architecture and training with early stopping and checkpointing.

Background:

Classifying short videos into categories using an automatic algorithm.

Requirements:

Implement a classification algorithm without using pre-trained models directly.

Solving Approach:

• Feature extraction from video frames using techniques like CNN.
• Training a machine learning model with extracted features and categorical labels.

Background:

Analyzing retail transaction records to provide business insights and sales suggestions.

Requirements:

Create visual methods for insights and use association rule analysis for recommendations.

Solving Approach:

• Data visualization for trends and patterns.
• Implementation of association rule mining using algorithms like Apriori or FP-Growth.

Background:

Predicting a patient's smoking status from biometric information.

Requirements:

Implement a binary classification algorithm.

Solving Approach:

• Exploratory data analysis and preprocessing.
• Application of classification algorithms, such as logistic regression or random forest.

Background:

Clustering bank customers for operational efficiency and tailored services.

Requirements:

Use at least three different clustering algorithms.

Solving Approach:

• Data exploration and visualization to understand customer attributes.
• Application of clustering algorithms like K-Means, DBSCAN, and Hierarchical Clustering.

Background:

Analyzing social network data to understand social relationships.

Requirements:

Conduct various graph analyses and community detection.

Solving Approach:

• Use of network analysis techniques and community detection algorithms.
• Visualization of network properties and community structures.

Q1:code in dsaa5002_project\Q1\Q1.ipynb

Q2:code in dsaa5002_project\Q2\Q2.ipynb

Q3:code in dsaa5002_project\Q3\Q3_locally.ipynb & Q3_colab.ipynb

Q4:code in dsaa5002_project\Q4\Q4_locally.ipynb & Q4_colab.ipynb

excel in dsaa5002_project\submission\Task1.xlsx & Task2.xlsx（by list）& Task2_2.xlsx (by knowledge graph) Q1: Three methods were used for similarity matching and keyword extraction after preprocessing the text by removing numbers, symbols, and tokenizing nouns.

• The first method utilized the bert-base-chinese model to calculate similarity using cosine_similarity, but it did not yield satisfactory results.

• The second method involved direct keyword matching, resulting in an extraction rate of 45.07%.

• The third method replaced bert-base-chinese with bert-wwm, which improved the extraction rate to 55.68%.

Q2: Three methods were used for sentiment analysis on the data.

• The first method involved comparing the count of positive and negative words using a sentiment analysis lexicon. However, this method sometimes resulted in an equal count of neutral words.

• The second method utilized the results from the first method to train a model. The pipeline of the model included preprocessing, tokenization, and a classification step with L2 regularization. However, the accuracy of this model was very low, only 0.47, as it couldn't learn effective features.

• The third method involved using the pre-trained sentiment analysis dictionary FinBERT 1.0, specifically trained for financial news. The positive-to-negative word ratio achieved was 11.01.

Q3： Two methods were attempted.

• The first method involved directly connecting to a database in Colab, but it resulted in an infinite loop.

• The second method was to download the community edition and create a database using the command line, followed by importing the data.

Q4： Two methods were attempted.

• 1.Querying and setting rules for Implicit_Positive_Company and Implicit_Negative_Company in Neo4j knowledge graph，which needs more than 7146.63s.

• 2.Directly searching in the list, first find the corresponding company ID, then check if :START_ID and :END_ID are present, and then perform rule-based evaluation. This process needs 4670.64s which is quicker than query in the knowdge graph.