mrhmisu / irdm-ml-models Goto Github PK

This repository contains all the source code that is done to accomplish the IRDM Course Project Part II

Directory Structure:
IRDM-Part-2/
          --/dataset
          --/result
          --/map_ndcg_evaluator
          --/bm25_model
          --/index
          --/logistic_regression
                --/sample
           --/lamda_mart
           --/neural_network
           --/file_util

 --/dataset
    Description: It is presumed that all the files are placed into this directory.
                 The sample training and validation dataset are already stored there.
                 In order to replicate the experiment all the provided .tsv file should be stored there.
 --/result
    Description: It contains all the output files generated by the models including the save models.


--/map_ndcg_evaluator
    /--metric_calculator.py
    /--main.py
    Description: It contains the implementation of MAP and NDCG. In order to calcualte MAP and NDCG, user needs to

--/bm25_model
    /--bm25.py
    /--bm25_executor.py
    /--main.py
     Description: It contains the implementation of BM25 model. In order to execute BM25 model, user needs to
                 run the main.py file by providing the file path of the input and output file.
--/index
    /--index_builder.py
    /--text_pre_processor.py
    /--index_test.py
    Description: It contains the implementation of Inverted index that is required for the BM25 implementation.
                 Besides it contain the implementation of text pre-processing.

--/logistic_regression
    --/sample
        /--sampler.py
        /--test_sampler.py
        Description: It contains the implementation of the dataset sampling. To regenerate the sampling
        user can execute the sampler.py

    /--logistic_regression_model.py
    /--lr_executor.py
    /--word2vec_builder.py
    Description: It contains the implementation of Logistic Regression (LR) Model including the Word2Vec feature implementation.
    (1)To run the LR model for testing its training and evaluation, user needs to execute the logistic_regression_model.py
    (2)To generate the output for the passage ranking, user needs to run lr_executor.py
    (3) To generate the Word2Vec featuer for quer-passage pair, user needs to run  word2vec_builder.py

--/lamda_mart
    /--hyper-parameter-tuning.py
    /--lamda_mart.py
    /--lm_executor.py
    Description: It contains the implementation of LamdaMART (LM) model including its hyper-parameter tuning.
    (1)To run the LM model for testing its training and evaluation, user needs to execute the lamda_mart.py
    (2) To tuning it hyper-parameter, user needs to execute the hyper-parameter-tuning.py
    (3)To generate the output for the passage ranking, user needs to run lm_executor.py


--/neural_network
    /--feedforward_nn.py
    /--nn_executor.py
    Description: It contains the implementation of the Neural Network (NN) model.
    (1)To run the NN model for testing its training and evaluation, user needs to execute the feedforward_nn.py
    (2)To generate the output for the passage ranking, user needs to run nn_executor.py
                 run the main.py file by providing a model generated output file.

--/file_util
    /--file_processor.py
    /--test.py
    Description: It is a utility directoty that perform IO operation.