Get rid of our sys.path.append in train_rage2e.py about dalm HOT 7 CLOSED

@Ben-Epstein Having a CLI-like interface to train the models makes a lot of sense. Playing devil's advocate here, what are the benefits of having a CLI for training over a simple Makefile?

A makefile is good for internal use, but not for a general cli, because the Makefile needs to be present to use it. The goal of the cli is for it to be global, for example:

pip install on indomain
dalm train new ...

And you can invoke dalm from anywhere. If we did makefiles, you'd have to be in the root of the DALM repo which isn't as useful for regular end users.

from dalm.

I get the following error when trying to run retriever-only training (which does not have sys.path.append). I found that Python does not add the current working directory to the path (the folder you are running the script from, which is different from the path where the script is located). As a result, the script can't import modules with absolute paths (and have to use import paths relative to the script path).

root@f4ec1ae23983:~/Shamane/DALM# python dalm/training/retriever_only/train_retriever_only.py  --dataset_path "/root/DALM/dataset/out/question_answer_pairs_train" --model_name_or_path "BAAI/bge-large-en" --output_dir "./retriever_only_c
heckpoints" --use_peft  --with_tracking --report_to all --per_device_train_batch_size 32
Traceback (most recent call last):
  File "/root/Shamane/DALM/dalm/training/retriever_only/train_retriever_only.py", line 36, in <module>
    from dalm.models.retriever_only_base_model import AutoModelForSentenceEmbedding
ModuleNotFoundError: No module named 'dalm'
root@f4ec1ae23983:~/Shamane/DALM# 

from dalm.

@Ben-Epstein Having a CLI-like interface to train the models makes a lot of sense. Playing devil's advocate here, what are the benefits of having a CLI for training over a simple Makefile?

from dalm.

I get the following error when trying to run retriever-only training (which does not have sys.path.append). I found that Python does not add the current working directory to the path (the folder you are running the script from, which is different from the path where the script is located). As a result, the script can't import modules with absolute paths (and have to use import paths relative to the script path).

root@f4ec1ae23983:~/Shamane/DALM# python dalm/training/retriever_only/train_retriever_only.py  --dataset_path "/root/DALM/dataset/out/question_answer_pairs_train" --model_name_or_path "BAAI/bge-large-en" --output_dir "./retriever_only_c
heckpoints" --use_peft  --with_tracking --report_to all --per_device_train_batch_size 32
Traceback (most recent call last):
  File "/root/Shamane/DALM/dalm/training/retriever_only/train_retriever_only.py", line 36, in <module>
    from dalm.models.retriever_only_base_model import AutoModelForSentenceEmbedding
ModuleNotFoundError: No module named 'dalm'
root@f4ec1ae23983:~/Shamane/DALM# 

@rsachira yes, you need to run it from the location of the file. We want to document that in the repo. So cd dalm/training/retriever_only and python train.... - this is how we need to document it. And a cli will help make that easier

from dalm.

@rsachira yes, you need to run it from the location of the file. We want to document that in the repo. So cd dalm/training/retriever_only and python train.... - this is how we need to document it. And a cli will help make that easier

Then, wandb and other log files will be created in the script directory?

from dalm.

We want to document that in the repo. So cd dalm/training/retriever_only and python train.... - this is how we need to document it. And a cli will help make that easier

@Ben-Epstein I tried running it from the directory. Same error. Were you able to run this? There could be some configuration problems in my machine.

from dalm.

We want to document that in the repo. So cd dalm/training/retriever_only and python train.... - this is how we need to document it. And a cli will help make that easier

@Ben-Epstein I tried running it from the directory. Same error. Were you able to run this? There could be some configuration problems in my machine.

This was solved by installing the repo (from readme docs)

from dalm.