Evaluate GPT-OSS-20B on code benchmark datasets

[neilwen987]

Hi, apologies if the question is naive. I'm trying to evaluate GPT-OSS-20B on code benchmark datsets(e.g. mbpp, humaneval).

export HF_ALLOW_CODE_EVAL=1
accelerate launch --num_processes 1 -m lm_eval \
  --model hf \
  --model_args pretrained=openai/gpt-oss-20b,dtype=bfloat16 \
  --tasks mbpp\
  --batch_size 32 \
  --confirm_run_unsafe_code \

and I got following results:

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
mbpp	1	extract_code	3	pass_at_1	↑	0	±	0
mbpp_instruct	1	extract_code	3	pass_at_1	↑	0	±	0
humaneval	1	create_test	0	pass@1		0.2622	±	0.0345

which are relativly low. I'm wondering can we use thinking mode when evaluating these tasks? And if so, how should I do it?

[neilwen987]

Oops, I have solved it. Btw, I still want to use think mode to conduct code evaluation. Is there any way to do it?

export HF_ALLOW_CODE_EVAL=1
accelerate launch --num_processes 1 -m lm_eval \
  --model hf \
  --model_args pretrained=openai/gpt-oss-20b,parallelize=True,dtype=bfloat16 \
  --tasks mbpp_instruct \
  --confirm_run_unsafe_code \
  --num_fewshot 3 \
  --apply_chat_template \
  --gen_kwargs temperature=0,top_p=0,max_new_tokens=512 \
  --batch_size 1 \
  --limit 10 \
  --log_samples \