System Info
optimum-habana 1.5.0
docker version 1.9.0
pytorch version 1.13.1
Information
Tasks
Reproduction
During the inference of bert (bert-large-uncased), finetuned on Financial PhraseBank dataset with bf16 data type, it results in an error.
The finetuning on Gaudi (HPU) is done with the help of optimum-habana library.
The transformers (4.28.1) and supporting libraries are installed as part of optimum-habana installation.
The finetuning works well for both data type (bf16 and fp32). The inference works well on fp32 data type. But when inference is done on bf16, it results in error.
The finetuning code is present here in finbert.py file.
import sys
import subprocess
subprocess.check_call([sys.executable, '-m', 'pip', 'install',
'numpy', ' pandas', ' scikit-learn', 'datasets', 'optimum.habana', '--user'])
import pandas as pd
import numpy as np
from transformers import AutoConfig, AutoTokenizer, AutoModelForSequenceClassification
from optimum.habana import GaudiConfig, GaudiTrainer, GaudiTrainingArguments
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from datasets import Dataset
def load_data():
df = pd.read_csv(
'FinancialPhraseBank-v1.0/Sentences_50Agree.txt',
sep='@',
names=['sentence', 'label'],
encoding = "ISO-8859-1")
df = df.dropna()
df['label'] = df['label'].map({"neutral": 0, "positive": 1, "negative": 2})
df.head()
df_train, df_test, = train_test_split(df, stratify=df['label'], test_size=0.1, random_state=42)
df_train, df_val = train_test_split(df_train, stratify=df_train['label'],test_size=0.1, random_state=42)
dataset_train = Dataset.from_pandas(df_train, preserve_index=False)
dataset_val = Dataset.from_pandas(df_val, preserve_index=False)
dataset_test = Dataset.from_pandas(df_test, preserve_index=False)
return dataset_train, dataset_val, dataset_test
def compute_metrics(eval_pred):
predictions, labels = eval_pred
predictions = np.argmax(predictions, axis=1)
return {'accuracy': accuracy_score(predictions, labels)}
def main():
dataset_train, dataset_val, dataset_test = load_data()
bert_model = AutoModelForSequenceClassification.from_pretrained('bert-large-uncased', num_labels=3)
bert_tokenizer = AutoTokenizer.from_pretrained('bert-large-uncased')
dataset_train = dataset_train.map(lambda e: bert_tokenizer(e['sentence'], truncation=True, padding='max_length', max_length=128), batched=True)
dataset_val = dataset_val.map(lambda e: bert_tokenizer(e['sentence'], truncation=True, padding='max_length', max_length=128), batched=True)
dataset_test = dataset_test.map(lambda e: bert_tokenizer(e['sentence'], truncation=True, padding='max_length' , max_length=128), batched=True)
dataset_train.set_format(type='torch', columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])
dataset_val.set_format(type='torch', columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])
dataset_test.set_format(type='torch', columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])
args = GaudiTrainingArguments(
output_dir='temp/',
overwrite_output_dir=True,
evaluation_strategy='epoch',
save_strategy='no',
logging_strategy='epoch',
logging_dir='logs/',
report_to='tensorboard',
learning_rate=2e-5,
per_device_train_batch_size=8,
per_device_eval_batch_size=4,
num_train_epochs=5,
weight_decay=0.01,
metric_for_best_model='accuracy',
use_habana=True, # use Habana device
use_lazy_mode=True, # use Gaudi lazy mode
use_hpu_graphs=True, # set value for hpu_graphs
gaudi_config_name='gaudi_config.json', # load config file
)
trainer = GaudiTrainer(
model=bert_model, # the instantiated 🤗 Transformers model to be trained
args=args, # training arguments, defined above
train_dataset=dataset_train, # training dataset
eval_dataset=dataset_val, # evaluation dataset
compute_metrics=compute_metrics
)
trainer.train()
if __name__ == '__main__':
main()
It also needs a gaudi_config.json file which has details for bf16 dtype training.
The gaudi_config.json file is:
{
"use_habana_mixed_precision": true,
"hmp_is_verbose": false,
"use_fused_adam": true,
"use_fused_clip_norm": true,
"hmp_bf16_ops": [
"add",
"addmm",
"bmm",
"div",
"dropout",
"gelu",
"iadd",
"linear",
"layer_norm",
"matmul",
"mm",
"rsub",
"softmax",
"truediv"
],
"hmp_fp32_ops": [
"embedding",
"nll_loss",
"log_softmax",
"cross_entropy"
]
}
Note: Keep both finbert.py and gaudi_config.json files in same folder.
Run it with command:
export MASTER_ADDR="localhost"
export MASTER_PORT="12345"
mpirun -n 8 --bind-to core --map-by socket:PE=4 --rank-by core --report-bindings --allow-run-as-root python finbert.py
Note: It can also be finetuned on 1 card for debugging purpose.
After completing the finetuning on bf16 dtype, next while running the inference code either code-1 or code-2, it results in error.
Inference code-1:
from transformers import pipeline
device=torch.device('hpu')
pipe = pipeline("text-classification", model=bert_model, tokenizer=bert_tokenizer, device=device)
print(pipe("Alabama Takes From the Poor and Gives to the Rich"))
print(pipe("Economists are predicting the highest rate of employment in 15 years"))
Inference code-2:
from transformers import TextClassificationPipeline
pipe = TextClassificationPipeline(model=bert_model, tokenizer=bert_tokenizer)
pipe.device=torch.device('hpu')
print(pipe("Alabama Takes From the Poor and Gives to the Rich"))
print(pipe("Economists are predicting the highest rate of employment in 15 years"))
Error seen after inference:
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
File /usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py:1146, in _LazyModule._get_module(self, module_name)
1145 try:
-> 1146 return importlib.import_module("." + module_name, self.__name__)
1147 except Exception as e:File /usr/lib/python3.8/importlib/__init__.py:127, in import_module(name, package)
126 level += 1
--> 127 return _bootstrap._gcd_import(name[level:], package, level)File <frozen importlib._bootstrap>:1014, in _gcd_import(name, package, level)File <frozen importlib._bootstrap>:991, in _find_and_load(name, import_)File <frozen importlib._bootstrap>:975, in _find_and_load_unlocked(name, import_)File <frozen importlib._bootstrap>:671, in _load_unlocked(spec)File <frozen importlib._bootstrap_external>:848, in exec_module(self, module)File <frozen importlib._bootstrap>:219, in _call_with_frames_removed(f, *args, **kwds)File /usr/local/lib/python3.8/dist-packages/transformers/models/gpt_bigcode/modeling_gpt_bigcode.py:56
51 # Fused kernels
52 # Use separate functions for each case because conditionals prevent kernel fusion.
53 # TODO: Could have better fused kernels depending on scaling, dropout and head mask.
54 # Is it doable without writing 32 functions?
55 @torch.jit.script
---> 56 def upcast_masked_softmax(
57 x: torch.Tensor, mask: torch.Tensor, mask_value: torch.Tensor, scale: float, softmax_dtype: torch.dtype
58 ):
59 input_dtype = x.dtypeFile /usr/local/lib/python3.8/dist-packages/torch/jit/_script.py:1343, in script(obj, optimize, _frames_up, _rcb, example_inputs)
1342 _rcb = _jit_internal.createResolutionCallbackFromClosure(obj)
-> 1343 fn = torch._C._jit_script_compile(
1344 qualified_name, ast, _rcb, get_default_args(obj)
1345 )
1346 # Forward docstringsFile /usr/local/lib/python3.8/dist-packages/torch/jit/_recursive.py:863, in try_compile_fn(fn, loc)
862 rcb = _jit_internal.createResolutionCallbackFromClosure(fn)
--> 863 return torch.jit.script(fn, _rcb=rcb)File /usr/local/lib/python3.8/dist-packages/torch/jit/_script.py:1343, in script(obj, optimize, _frames_up, _rcb, example_inputs)
1342 _rcb = _jit_internal.createResolutionCallbackFromClosure(obj)
-> 1343 fn = torch._C._jit_script_compile(
1344 qualified_name, ast, _rcb, get_default_args(obj)
1345 )
1346 # Forward docstringsRuntimeError:
Unknown type name 'DType':
File "/usr/local/lib/python3.8/dist-packages/habana_frameworks/torch/hpex/hmp/utils.py", line 1811
def softmax(input: Tensor, dim: Optional[int] = None, _stacklevel: int = 3, dtype: Optional[DType] = None) -> Tensor:
~~~~~ <--- HERE
r"""Applies a softmax function.
'softmax' is being compiled since it was called from 'upcast_masked_softmax'
File "/usr/local/lib/python3.8/dist-packages/transformers/models/gpt_bigcode/modeling_gpt_bigcode.py", line 62
x = x.to(softmax_dtype) * scale
x = torch.where(mask, x, mask_value)
x = torch.nn.functional.softmax(x, dim=-1).to(input_dtype)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
return x
The above exception was the direct cause of the following exception:RuntimeError Traceback (most recent call last)
Cell In[26], line 3
1 from transformers import pipeline
2 device=torch.device('hpu')
----> 3 pipe = pipeline("text-classification", model=trainer.model, tokenizer=bert_tokenizer, device=device)
4 #pipe = TextClassificationPipeline(model=bert_model, tokenizer=bert_tokenizer)
5 #pipe = TextClassificationPipeline(model=bert_model, tokenizer=bert_tokenizer)
6 #pipe.device=torch.device('hpu')
8 print(pipe("Alabama Takes From the Poor and Gives to the Rich"))File /usr/local/lib/python3.8/dist-packages/transformers/pipelines/__init__.py:979, in pipeline(task, model, config, tokenizer, feature_extractor, image_processor, framework, revision, use_fast, use_auth_token, device, device_map, torch_dtype, trust_remote_code, model_kwargs, pipeline_class, **kwargs)
976 if device is not None:
977 kwargs["device"] = device
--> 979 return pipeline_class(model=model, framework=framework, task=task, **kwargs)File /usr/local/lib/python3.8/dist-packages/transformers/pipelines/text_classification.py:85, in TextClassificationPipeline.__init__(self, **kwargs)
82 def __init__(self, **kwargs):
83 super().__init__(**kwargs)
---> 85 self.check_model_type(
86 TF_MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING
87 if self.framework == "tf"
88 else MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING
89 )File /usr/local/lib/python3.8/dist-packages/transformers/pipelines/base.py:942, in Pipeline.check_model_type(self, supported_models)
940 if not isinstance(supported_models, list): # Create from a model mapping
941 supported_models_names = []
--> 942 for config, model in supported_models.items():
943 # Mapping can now contain tuples of models for the same configuration.
944 if isinstance(model, tuple):
945 supported_models_names.extend([_model.__name__ for _model in model])File /usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py:644, in _LazyAutoMapping.items(self)
643 def items(self):
--> 644 mapping_items = [
645 (
646 self._load_attr_from_module(key, self._config_mapping[key]),
647 self._load_attr_from_module(key, self._model_mapping[key]),
648 )
649 for key in self._model_mapping.keys()
650 if key in self._config_mapping.keys()
651 ]
652 return mapping_items + list(self._extra_content.items())File /usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py:647, in <listcomp>(.0)
643 def items(self):
644 mapping_items = [
645 (
646 self._load_attr_from_module(key, self._config_mapping[key]),
--> 647 self._load_attr_from_module(key, self._model_mapping[key]),
648 )
649 for key in self._model_mapping.keys()
650 if key in self._config_mapping.keys()
651 ]
652 return mapping_items + list(self._extra_content.items())File /usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py:616, in _LazyAutoMapping._load_attr_from_module(self, model_type, attr)
614 if module_name not in self._modules:
615 self._modules[module_name] = importlib.import_module(f".{module_name}", "transformers.models")
--> 616 return getattribute_from_module(self._modules[module_name], attr)File /usr/local/lib/python3.8/dist-packages/transformers/models/auto/auto_factory.py:561, in getattribute_from_module(module, attr)
559 if isinstance(attr, tuple):
560 return tuple(getattribute_from_module(module, a) for a in attr)
--> 561 if hasattr(module, attr):
562 return getattr(module, attr)
563 # Some of the mappings have entries model_type -> object of another model type. In that case we try to grab the
564 # object at the top level.File /usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py:1136, in _LazyModule.__getattr__(self, name)
1134 value = self._get_module(name)
1135 elif name in self._class_to_module.keys():
-> 1136 module = self._get_module(self._class_to_module[name])
1137 value = getattr(module, name)
1138 else:File /usr/local/lib/python3.8/dist-packages/transformers/utils/import_utils.py:1148, in _LazyModule._get_module(self, module_name)
1146 return importlib.import_module("." + module_name, self.__name__)
1147 except Exception as e:
-> 1148 raise RuntimeError(
1149 f"Failed to import {self.__name__}.{module_name} because of the following error (look up to see its"
1150 f" traceback):\n{e}"
1151 ) from eRuntimeError: Failed to import transformers.models.gpt_bigcode.modeling_gpt_bigcode because of the following error (look up to see its traceback):Unknown type name 'DType':
File "/usr/local/lib/python3.8/dist-packages/habana_frameworks/torch/hpex/hmp/utils.py", line 1811
def softmax(input: Tensor, dim: Optional[int] = None, _stacklevel: int = 3, dtype: Optional[DType] = None) -> Tensor:
~~~~~ <--- HERE
r"""Applies a softmax function.
'softmax' is being compiled since it was called from 'upcast_masked_softmax'
File "/usr/local/lib/python3.8/dist-packages/transformers/models/gpt_bigcode/modeling_gpt_bigcode.py", line 62
x = x.to(softmax_dtype) * scale
x = torch.where(mask, x, mask_value)
x = torch.nn.functional.softmax(x, dim=-1).to(input_dtype)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
return x
Expected behavior
Inference is expected to work well in bf16 just like fp32 dtype.
Below shown output is expected
[{'label': 'neutral', 'score': 0.9094224572181702}]
[{'label': 'positive', 'score': 0.9752092957496643}]