I download myself arXiv tex files without using running s/arxiv-kickoff-download

Hi <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="

how to process arXiv tex files without downloading? about redpajama-data HOT 1 CLOSED

irene622 commented on July 17, 2024

how to process arXiv tex files without downloading?

from redpajama-data.

Comments (1)

mauriceweber commented on July 17, 2024 1

Hi @irene622 , thanks for your question!

The run_clean.py script expects your data to be organized in the same way as when it is downloaded from the arxiv s3 bucket. The most straight forward way is thus probably to simply mirror this structure. In this case you will need to have package your my_arxiv_src into a a tar file and store it in data/src. So you need something like this:

data/src
|-my_arxiv_src.tar
    |-parpername1.gz
        |- name.tex
    |-papername2.gz
        |-name.tex
        |-other_name.tex

You can then call python run_clean.py --data_dir data/src --target_dir /dir/to/desired/output. Also check out this part in the code of the arxiv_cleaner module:

RedPajama-Data/data_prep/arxiv/arxiv_cleaner.py

Lines 131 to 177 in d174968

    
           if tar_fp_list is None: 
        
               def _tar_fp_iterator(): 
        
                   for _tar_fp in self._data_dir.glob("*.tar"): 
        
                       yield _tar_fp 
        
           else: 
        
               def _tar_fp_iterator(): 
        
                   for _tar_fp in tar_fp_list: 
        
                       yield _tar_fp 
        
           failed = 0 
        
           processed = 0 
        
           for tar_fp in _tar_fp_iterator(): 
        
               print(f"[{get_timestamp()}][INFO] start processing {tar_fp}") 
        
               with tempfile.TemporaryDirectory(dir=self._work_dir) as tmpdir: 
        
                   with tarfile.open(tar_fp) as tf: 
        
                       tf.extractall(members=tf.getmembers(), path=tmpdir) 
        
                       for proj_dir_or_file in pathlib.Path(tmpdir).rglob("*.gz"): 
        
                           # get arxiv id and month from the filename 
        
                           yymm = proj_dir_or_file.parent.stem 
        
                           arxiv_id = proj_dir_or_file.stem 
        
                           # load the tex source files (we also get the timestamp 
        
                           # here) 
        
                           data = _tex_proj_loader(proj_dir_or_file) 
        
                           if data is None: 
        
                               failed += 1 
        
                               continue 
        
                           tex_files, timestamp = data 
        
                           processed += 1 
        
                           if processed > max_files > 0: 
        
                               break 
        
                           yield tex_files, yymm, arxiv_id, timestamp 
        
                       else: 
        
                           continue 
        
                       break 
        
           print(f"[{get_timestamp()}][INFO] # Failed loading : {failed}") 
        
           print(f"[{get_timestamp()}][INFO] done.")

Let me know if this helps!

from redpajama-data.

Recommend Projects

how to process arXiv tex files without downloading? about redpajama-data HOT 1 CLOSED

Comments (1)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	if tar_fp_list is None:
	def _tar_fp_iterator():
	for _tar_fp in self._data_dir.glob("*.tar"):
	yield _tar_fp
	else:
	def _tar_fp_iterator():
	for _tar_fp in tar_fp_list:
	yield _tar_fp

	failed = 0
	processed = 0

	for tar_fp in _tar_fp_iterator():
	print(f"[{get_timestamp()}][INFO] start processing {tar_fp}")

	with tempfile.TemporaryDirectory(dir=self._work_dir) as tmpdir:
	with tarfile.open(tar_fp) as tf:
	tf.extractall(members=tf.getmembers(), path=tmpdir)

	for proj_dir_or_file in pathlib.Path(tmpdir).rglob("*.gz"):

	# get arxiv id and month from the filename
	yymm = proj_dir_or_file.parent.stem
	arxiv_id = proj_dir_or_file.stem

	# load the tex source files (we also get the timestamp
	# here)
	data = _tex_proj_loader(proj_dir_or_file)

	if data is None:
	failed += 1
	continue

	tex_files, timestamp = data
	processed += 1

	if processed > max_files > 0:
	break

	yield tex_files, yymm, arxiv_id, timestamp

	else:
	continue
	break

	print(f"[{get_timestamp()}][INFO] # Failed loading : {failed}")
	print(f"[{get_timestamp()}][INFO] done.")