alexandra-chron / siatl Goto Github PK
View Code? Open in Web Editor NEWPyTorch source code of NAACL 2019 paper "An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models"
PyTorch source code of NAACL 2019 paper "An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models"
I have some data in the jsonl
format (blog on jsonl vs json).
We are trying to finetune our language model (trained using your model and config). How does the training info look like? Any inputs would be greatly appreciated.
Thanks in advance :)
I've heard this can be an issue when code written on a Linux machine is run on Windows, but I'm not sure how to solve. The error code is: load_pickle.UnpicklingError: the STRING opcode argument must be quoted. Full error is below. Thank you for your help!
PS C:\Users\colby\OneDrive\Desktop\Startup\Code\Sentiment_Discovery Model\siatl> python models/run_clf.py -i SCV2_aux_ft_gu.yaml --aux_loss --transfer
torch: 1.2.0
Cuda: None
CuDNN: None
C:\Users\colby\OneDrive\Desktop\Startup\Code\Sentiment_Discovery Model\siatl\models..\utils\config.py:34: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
cfg = yaml.load(stream)
input:SCV2_aux_ft_gu.yaml
transfer:True
aux_loss:True
config:SCV2_aux_ft_gu.yaml
c:checkpoint
device:cpu
cores:4
source:['C:\Users\colby\OneDrive\Desktop\Startup\Code\Sentiment_Discovery Model\siatl\models', 'C:\Users\colby\OneDrive\Desktop\Startup\Code\Sentiment_Discovery Model\siatl\modules', 'C:\Users\colby\OneDrive\Desktop\Startup\Code\Sentiment_Discovery Model\siatl\utils']
Traceback (most recent call last):
File "models/run_clf.py", line 33, in
dataset = load_dataset(config)
File "C:\Users\colby\OneDrive\Desktop\Startup\Code\Sentiment_Discovery Model\siatl\models..\utils\data_parsing.py", line 108, in load_dataset
dataset = load_scv2_dataset(test)
File "C:\Users\colby\OneDrive\Desktop\Startup\Code\Sentiment_Discovery Model\siatl\models..\utils\data_parsing.py", line 79, in load_scv2_dataset
data = pickle.load(f)
_pickle.UnpicklingError: the STRING opcode argument must be quoted
PS C:\Users\colby\OneDrive\Desktop\Startup\Code\Sentiment_Discovery Model\siatl>
Hi,
i try to run the code, but it is not without issues, can you help?
Should I download a dataset?
Here is the output from a run:
python models/sent_lm.py -i lm_20m_word.yaml
torch: 1.2.0
Cuda: 10.0.130
CuDNN: 7602
input:lm_20m_word.yaml
config:lm_20m_word.yaml
c:checkpoint
device:cpu
cores:4
source:['/home/ant/multitask/siatl/models', '/home/ant/multitask/siatl/modules', '/home/ant/multitask/siatl/utils']
Building training dataset...
Traceback (most recent call last):
File "models/sent_lm.py", line 44, in
seq_len=config["data"]["seq_len"])
File "/home/ant/multitask/siatl/models/../utils/datasets.py", line 250, in init
input, self.preprocess, vocab, vocab_size)
File "/home/ant/multitask/siatl/models/../utils/datasets.py", line 52, in read_corpus
for line in iterate_data(file):
File "/home/ant/multitask/siatl/models/../utils/datasets.py", line 34, in iterate_data
assert os.path.exists(data)
AssertionError
Output for git status:
pycache/
checkpoints/lm20m_70K.pt
checkpoints/lm20m_70K.pt.1
datasets/sarcasm_v2.csv
logger/pycache/
models/pycache/
modules/pycache/
utils/pycache/
Hi,
any idea for the error bellow?
WARNING:root:Setting up a new session...
Exception in user code:
------------------------------------------------------------
Traceback (most recent call last):
File "/home/ant/env/../lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/home/ant/env/../lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection
raise err
File "/home/ant/env/../lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ant/env/../lib/python3.6/site-packages/urllib3/connectionpool.py", line 603, in urlopen
chunked=chunked)
File "/home/ant/env/../lib/python3.6/site-packages/urllib3/connectionpool.py", line 355, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.6/http/client.py", line 1254, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1300, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1249, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1036, in _send_output
self.send(msg)
File "/usr/lib/python3.6/http/client.py", line 974, in send
self.connect()
File "/home/ant/env/../lib/python3.6/site-packages/urllib3/connection.py", line 183, in connect
conn = self._new_conn()
File "/home/ant/env/../lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f6ed6b2a5f8>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ant/env/../lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/ant/env/../lib/python3.6/site-packages/urllib3/connectionpool.py", line 641, in urlopen
_stacktrace=sys.exc_info()[2])
File "/home/ant/env/../lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8097): Max retries exceeded with url: /env/scv2_aux_ft_gu (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ed6b2a5f8>: Failed to establish a new connection: [Errno 111] Connection refused',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ant/env/.../lib/python3.6/site-packages/visdom/init.py", line 548, in _send
data=json.dumps(msg),
File "/home/ant/env/.../lib/python3.6/site-packages/requests/sessions.py", line 581, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/ant/env/.../lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/home/ant/env/..../lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/home/ant/env/.../lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8097): Max retries exceeded with url: /env/scv2_aux_ft_gu (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ed6b2a5f8>: Failed to establish a new connection: [Errno 111] Connection refused',))
WARNING:visdom:Without the incoming socket you cannot receive events from the server or register event handlers to your Visdom client.
Exception in user code:
------------------------------------------------------------
Traceback (most recent call last):
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection
raise err
File "/home/ant/env/../lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 603, in urlopen
chunked=chunked)
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 355, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.6/http/client.py", line 1254, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1300, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1249, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1036, in _send_output
self.send(msg)
File "/usr/lib/python3.6/http/client.py", line 974, in send
self.connect()
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connection.py", line 183, in connect
conn = self._new_conn()
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f6ed6034208>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ant/env/.../lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 641, in urlopen
_stacktrace=sys.exc_info()[2])
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8097): Max retries exceeded with url: /close (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ed6034208>: Failed to establish a new connection: [Errno 111] Connection refused',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ant/env/.../lib/python3.6/site-packages/visdom/init.py", line 548, in _send
data=json.dumps(msg),
File "/home/ant/env/.../lib/python3.6/site-packages/requests/sessions.py", line 581, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/ant/env/.../lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/home/ant/env/.../lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/home/ant/env/.../lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8097): Max retries exceeded with url: /close (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ed6034208>: Failed to establish a new connection: [Errno 111] Connection refused',))
Exception in user code:
------------------------------------------------------------
Traceback (most recent call last):
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection
raise err
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 603, in urlopen
chunked=chunked)
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 355, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.6/http/client.py", line 1254, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1300, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1249, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1036, in _send_output
self.send(msg)
File "/usr/lib/python3.6/http/client.py", line 974, in send
self.connect()
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connection.py", line 183, in connect
conn = self._new_conn()
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f6ed60344e0>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ant/env/.../lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/connectionpool.py", line 641, in urlopen
_stacktrace=sys.exc_info()[2])
File "/home/ant/env/.../lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8097): Max retries exceeded with url: /events (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ed60344e0>: Failed to establish a new connection: [Errno 111] Connection refused',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ant/env/.../lib/python3.6/site-packages/visdom/init.py", line 548, in _send
data=json.dumps(msg),
File "/home/ant/env/.../lib/python3.6/site-packages/requests/sessions.py", line 581, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/ant/env/.../lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/home/ant/env/.../lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/home/ant/env/.../lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8097): Max retries exceeded with url: /events (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ed60344e0>: Failed to establish a new connection: [Errno 111] Connection refused',))
Transferring Encoder weights ...
Classifier(
(lockdrop): LockedDropout()
(idrop): Dropout(p=0.1)
(hdrop): Dropout(p=0.1)
(drop): Dropout(p=0.1)
(embed): Embed(
(embedding): Embedding(70004, 400)
(dropout): Dropout(p=0.3)
(noise): GaussianNoise (mean=0.0, stddev=0.1)
)
(bottom_rnn): RNNModule(
(rnn): LSTM(400, 1000, num_layers=2, batch_first=True, dropout=0.5)
)
(top_rnn): RNNModule(
(rnn): LSTM(1000, 100, batch_first=True, dropout=0.3)
)
(attention): SelfAttention(
(attention): Sequential(
(0): Linear(in_features=100, out_features=1, bias=True)
(1): Tanh()
(2): Dropout(p=0.3)
)
(softmax): Softmax()
)
(vocab): Linear(in_features=1000, out_features=70004, bias=True)
(classes): Linear(in_features=100, out_features=2, bias=True)
)
THCudaCheck FAIL file=/pytorch/aten/src/THC/generic/THCStorage.cu line=58 error=2 : out of memory
Traceback (most recent call last):
File "models/run_clf.py", line 37, in
opts=opts, transfer=transfer)
File "/home/ant/multitask/siatl/models/../models/sent_clf.py", line 155, in sent_clf
train_loss = trainer.train_epoch()
File "/home/ant/multitask/siatl/models/../models/sent_clf_trainer.py", line 85, in train_epoch
loss_sum.backward()
File "/home/ant/env/.../lib/python3.6/site-packages/torch/tensor.py", line 93, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/ant/env/.../lib/python3.6/site-packages/torch/autograd/init.py", line 89, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: cuda runtime error (2) : out of memory at /pytorch/aten/src/THC/generic/THCStorage.cu:58
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.