i am trying to run newly announced model: Phi-4-mini-instruct

It has example, I tried it on Google Colab T4 Notebook, but here is what I got

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

 
torch.random.manual_seed(0)

model_path = "microsoft/Phi-4-mini-instruct"

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    device_map="auto",
    torch_dtype="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_path)
 
messages = [
    {"role": "system", "content": "You are a helpful AI assistant."},
    {"role": "user", "content": "Can you provide ways to eat combinations of bananas and dragonfruits?"},
    {"role": "assistant", "content": "Sure! Here are some ways to eat bananas and dragonfruits together: 1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey. 2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey."},
    {"role": "user", "content": "What about solving an 2x + 3 = 7 equation?"},
]
 
pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
)
 
generation_args = {
    "max_new_tokens": 500,
    "return_full_text": False,
    "temperature": 0.0,
    "do_sample": False,
}
 
output = pipe(messages, **generation_args)
print(output[0]['generated_text'])

Device set to use cuda:0 /usr/local/lib/python3.11/dist-packages/transformers/generation/configuration_utils.py:628: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.0` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.   warnings.warn(
--------------------------------------------------------------------------- 
RuntimeError                              Traceback (most recent call last) <ipython-input-3-c4a0ae4782d8> in <cell line: 0>()
     19 }
     20 
---> 21 output = pipe(messages, **generation_args)
     22 print(output[0]['generated_text'])

7 frames /usr/local/lib/python3.11/dist-packages/transformers/generation/utils.py in _sample(self, input_ids, logits_processor, stopping_criteria, generation_config, synced_gpus, streamer, **model_kwargs)    3307     3308             # update generated ids, model inputs, and length for next step
-> 3309             input_ids = torch.cat([input_ids, next_tokens[:, None]], dim=-1)    3310             if streamer is not None:    3311   streamer.put(next_tokens.cpu())

RuntimeError: Tensors must have same number of dimensions: got 2 and 3

I just copied and pasted as the example, what mistake do i make?

i am trying to run newly announced model: Phi-4-mini-instruct

https://huggingface.co/microsoft/Phi-4-mini-instruct

It has example, I tried it on Google Colab T4 Notebook, but here is what I got

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

 
torch.random.manual_seed(0)

model_path = "microsoft/Phi-4-mini-instruct"

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    device_map="auto",
    torch_dtype="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_path)
 
messages = [
    {"role": "system", "content": "You are a helpful AI assistant."},
    {"role": "user", "content": "Can you provide ways to eat combinations of bananas and dragonfruits?"},
    {"role": "assistant", "content": "Sure! Here are some ways to eat bananas and dragonfruits together: 1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey. 2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey."},
    {"role": "user", "content": "What about solving an 2x + 3 = 7 equation?"},
]
 
pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
)
 
generation_args = {
    "max_new_tokens": 500,
    "return_full_text": False,
    "temperature": 0.0,
    "do_sample": False,
}
 
output = pipe(messages, **generation_args)
print(output[0]['generated_text'])

Device set to use cuda:0 /usr/local/lib/python3.11/dist-packages/transformers/generation/configuration_utils.py:628: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.0` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.   warnings.warn(
--------------------------------------------------------------------------- 
RuntimeError                              Traceback (most recent call last) <ipython-input-3-c4a0ae4782d8> in <cell line: 0>()
     19 }
     20 
---> 21 output = pipe(messages, **generation_args)
     22 print(output[0]['generated_text'])

7 frames /usr/local/lib/python3.11/dist-packages/transformers/generation/utils.py in _sample(self, input_ids, logits_processor, stopping_criteria, generation_config, synced_gpus, streamer, **model_kwargs)    3307     3308             # update generated ids, model inputs, and length for next step
-> 3309             input_ids = torch.cat([input_ids, next_tokens[:, None]], dim=-1)    3310             if streamer is not None:    3311   streamer.put(next_tokens.cpu())

RuntimeError: Tensors must have same number of dimensions: got 2 and 3

I just copied and pasted as the example, what mistake do i make?

• python
• runtime-error
• tensor
• large-language-model

Share Improve this question Follow edited Mar 4 at 15:33 desertnaut 60.5k3232 gold badges155155 silver badges182182 bronze badges asked Mar 4 at 14:44 최영진최영진 111 bronze badge 1

• error shows problem in line pipe(messages, **generation_args) so first you could use print() to check what you have in variables and what dimentions they have. maybe problem is your data which may have wrong dimensions. – furas Commented Mar 5 at 2:47

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

Same issue reported on huggingface: https://huggingface.co/microsoft/Phi-4-mini-instruct/discussions/14

The suggestion there was to downgrade to Python 3.8.

When I used Python 3.10 the error was raised, but after changing it to 3.8, it works!