huangwl18 / language-planner Goto Github PK

Hello Wenlong,

I think there might be an error in calculating mean log probability when using GPT-3. The main issue is that GPT-3 does not only return generated texts in response, it returns more than these (including token_logprobs of logprobs). Therefore, in order to calculate the mean log probability, we cannot simply use

# calculate mean log prob across tokens
mean_log_probs = [np.mean(response['choices'][i]['logprobs']['token_logprobs']) for i in range(sampling_params['n'])]

Instead, we should stop counting when a stop token is met.

For example, here is a response with a stop sequence of "\n". The generated text is "Walk to kitchen", however GPT-3 returns more than that,

response: {
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "logprobs": {
        "text_offset": [
          317,
          322,
          325,
          333,
          333,
          333,
          333,
          333
        ],
        "token_logprobs": [
          -0.2976162,
          -0.00012346054,
          -0.5069456,
          -0.0011470452,
          -0.0060894582,
          -0.00028055036,
          -6.838237e-05,
          -0.054386232
        ],
        "tokens": [
          " Walk",
          " to",
          " kitchen",
          "\n",
          "Step",
          " 2",
          ":",
          " Walk"
        ],
        "top_logprobs": [
          {
            " Get": -3.9821253,
            " Go": -3.5860093,
            " Make": -3.1428235,
            " Wake": -2.513738,
            " Walk": -0.2976162
          },
          {
            " To": -12.335158,
            " in": -11.411637,
            " into": -9.384543,
            " to": -0.00012346054,
            " upstairs": -12.2138815
          },
          {
            " bedroom": -5.3587174,
            " dining": -1.0860167,
            " kitchen": -0.5069456,
            " living": -4.34434,
            " the": -3.2986841
          },
          {
            "\n": -0.0011470452,
            " ": -7.6692185,
            " table": -9.372099,
            ".": -8.122213,
            "ette": -9.167303
          },
          {
            "\n": -5.1904135,
            " Step": -7.8304586,
            "Step": -0.0060894582,
            "Task": -9.905375,
            "step": -10.6300955
          },
          {
            " 1": -10.295448,
            " 2": -0.00028055036,
            " 3": -11.589857,
            " 4": -12.77457,
            "2": -8.387781
          },
          {
            "\n": -11.062581,
            " :": -11.94543,
            ",": -12.268325,
            ".": -10.367215,
            ":": -6.838237e-05
          },
          {
            " Find": -3.783928,
            " Open": -4.0909195,
            " Turn": -5.903181,
            " Walk": -0.054386232,
            "Walk": -5.14835
          }
        ]
      },
      "text": " Walk to kitchen"
    }
  ],
  "model": "text-davinci-001",
  "object": "text_completion",
  "usage": {
    "completion_tokens": 3,
    "prompt_tokens": 94,
    "total_tokens": 97
  }
}

The current way of calculating mean log prob gives -0.10833211608375, where it should be mean(-0.2976162, -0.00012346054, -0.5069456) = -0.26822842018

Please let me know what you think. Great work!

Cheers,
Kaixian

Dear @huangwl18,

Thank you for your excellent work. I have a small question regarding the dataset you used in your research.

I noticed that the dataset you used contains around 5k examples, while the original dataset only contains around 2k. Additionally, the unique task you used is 204, whereas the original dataset contains more than five hundred tasks. I was curious if you selected specific tasks from the original dataset and expanded it? If yes, I would be grateful if you could share the criteria you used for task selection and how you expanded the dataset.

Looking forward to your reply. Thanks

On running the cell of initializing the environment, while its installing the requirements, its getting struck on unballe to find the appropriate version of torch, could you please guide on this please?

Hello,

Thanks for sharing your code! Could you share the code for calculating Executability and Correctness as well?

Thank you!

KeyError Traceback (most recent call last)
in <cell line: 13>()
14 best_overall_score = -np.inf
15 # query Planning LM for single-step action candidates
---> 16 samples, log_probs = generator(curr_prompt + f'\nStep {step}:', sampling_params)
17 for sample, log_prob in zip(samples, log_probs):
18 most_similar_idx, matching_score = find_most_similar(sample, action_list_embedding)

in _generate(prompt, sampling_params)
15 input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
16 prompt_len = input_ids.shape[-1]
---> 17 output_dict = model.generate(input_ids, max_length=prompt_len + sampling_params['max_tokens'], **sampling_params) # Update parameter name
18 # discard the prompt (only take the generated text)
19 generated_samples = tokenizer.batch_decode(output_dict.sequences[:, prompt_len:])

KeyError: 'max_tokens'

Hello, @huangwl18
First, I would like to thank you for sharing your awesome code!

I ran your demo without any change and this is what I got from the last cell (i.e., the autoregressive plan generation part) as below.
But it is a bit hard for me to find the output plan represents a possible solution (e.g., it tries to just open multiple objects but does not make some food).
Is this output expected (i.e., did you get that same output)? If not, what could be a reason for this?

---------- GIVEN EXAMPLE ----------
Task: Make toast
Step 1: Walk to dining room
Step 2: Walk to freezer
Step 3: Find freezer
Step 4: Open freezer
Step 5: Find food bread
Step 6: Grab food bread
Step 7: Close freezer
Step 8: Find toaster
Step 9: Plug in toaster
Step 10: Put food bread on toaster
Step 11: Switch on toaster
---------- EXAMPLE END ----------

Task: Make breakfast
Step 1: Walk to kitchen
Step 2: Turn to fridge
Step 3: Open fridge
Step 4: Open microwave
Step 5: Open oven
Step 6: Open stove
Step 7: Open cupboard

[Terminating early because best overall score is lower than CUTOFF_THRESHOLD (0.7268587350845337 < 0.8)]

Thank you!

Hi, thanks for the excellent work. I was wondering how you could initialize the scene and execute the corresponding generated plan in the Virutal Home simulator? Specifically, I want to test the excitability of the plans.