Setting pad_token_id to eos_token_id:0 for open-end generation.

RuntimeError Traceback (most recent call last)
in <cell line: 2>()
1 query_engine = index.as_query_engine()
----> 2 response = query_engine.query( "What's the cost of Whisper model?")

37 frames
/usr/local/lib/python3.10/dist-packages/transformers/models/gpt_neox/modeling_gpt_neox.py in _attn(self, query, key, value, attention_mask, head_mask)
217 # Need to be on the same device, otherwise RuntimeError: ..., x and y to be on the same device
218 mask_value = torch.tensor(mask_value, dtype=attn_scores.dtype).to(attn_scores.device)
--> 219 attn_scores = torch.where(causal_mask, attn_scores, mask_value)
220
221 if attention_mask is not None:

RuntimeError: The size of tensor a (2048) must match the size of tensor b (2049) at non-singleton dimension 3

Hi.

I replicated your code as is. I just had to downgrade the langchain package to 0.0.142 and llama-index to 0.5.27 as GPTSimpleVectorIndex is deprecated in the latest version.

I am not getting the same output:
response = index.query("what is the cost of Whisper model?")
response.response

Output:
OPENAI_API_KEY" -H "Content-Type: multipart/form-data" -F model="whisper-1" -F file="@/path/to/file/openai.mp3"

FYI: I get a OPENAI_API_KEY Authentication error when I use GPTVectorStoreIndex.. Can you explain why it is connecting to OPENAI?

GPTSimpleVectorIndex was renamed to GPTVectorStoreIndex

Issue Description

Summary:
I encountered a TypeError while using the LLMPredictor class with a custom FlanLLM class. It appears to be related to the _identifying_params method.

Details:
When trying to create an instance of LLMPredictor with llm=FlanLLM(), I received the following error:

TypeError: 'method' object is not iterable

Expected Behavior
I expected to create an instance of LLMPredictor successfully using my custom FlanLLM class without encountering any errors.

Actual Behavior
I received a TypeError when attempting to create an instance of LLMPredictor. The error message indicates that there's an issue with the _identifying_params method in the FlanLLM class.

Steps to Reproduce

1. Create a custom FlanLLM class as follows:
   class FlanLLM(LLM):
   model_name = "google/flan-t5-large"
   pipeline = pipeline("text2text-generation", model=model_name, device=0, model_kwargs={"torch_dtype": torch.bfloat16})

   def _call(self, prompt, stop=None):
   return self.pipeline(prompt, max_length=9999)[0]["generated_text"]

   def _identifying_params(self):
   return {"name_of_model": self.model_name}

   def _llm_type(self):
   return "custom"

class FlanLLM(LLM):
    model_name = "google/flan-t5-large"
    pipeline = pipeline("text2text-generation", model=model_name, device=0, model_kwargs={"torch_dtype": torch.bfloat16})

    def _call(self, prompt, stop=None):
        return self.pipeline(prompt, max_length=9999)[0]["generated_text"]

    def _identifying_params(self):
        return {"name_of_model": self.model_name}

    def _llm_type(self):
        return "custom"

3. Create an instance of FlanLLM and attempt to create an instance of LLMPredictor:

llm_instance = FlanLLM()
llm_predictor = LLMPredictor(llm=llm_instance)

Code Snippet

class FlanLLM(LLM):
    model_name = "google/flan-t5-large"
    pipeline = pipeline("text2text-generation", model=model_name, device=0, model_kwargs={"torch_dtype": torch.bfloat16})

    def _call(self, prompt, stop=None):
        return self.pipeline(prompt, max_length=9999)[0]["generated_text"]

    def _identifying_params(self):
        return {"name_of_model": self.model_name}

    def _llm_type(self):
        return "custom"

llm_instance = FlanLLM()
llm_predictor = LLMPredictor(llm=llm_instance)

Additional Information

• I have ensured that the FlanLLM class correctly inherits from the LLM class.
• The error occurs at the line where LLMPredictor is instantiated.
• I am using the appropriate versions of the required libraries and packages.

Screenshots or Log Output

[If applicable, include screenshots or log output that may help diagnose the issue.]

Possible Solutions

I'm not sure what is causing this TypeError. It appears to be related to the _identifying_params method, but I'm unsure how to resolve it. Any guidance or suggestions would be greatly appreciated.

Steps Taken to Resolve

I have reviewed my code, checked for typos, and ensured that the method names and parameters match the expected format. However, I have not been able to resolve this issue on my own.

Note: Please let me know if you need any additional information or if there are specific steps I should take to troubleshoot this issue further.