lm_eval compatibility with NeMo v2.0 models (config file missing error)

[Harsha0056]

I have been experimenting with lm_eval using NeMo models. Based on my understanding, there are two different NeMo formats:

• Version 1.0

  • Models are saved in the format:

    Meta-Llama-3-8B-Instruct.nemo
    

  • Running lm_eval with this format works fine and I don’t face any issues.

• Version 2.0

  • The saved model format looks different (see screenshots below).
  • When I try to run lm_eval with this format, I get errors saying that the config file is missing. i did look inside the folders i could find model.yaml but not model_config.yaml .
  • Note : This models works fine with regular nemo inference code.

Screenshots:

*** CLI Code :**

nemo 2.0 model format :

error with lm_eval wrt nemo 2.0 model

---

Question

• Is lm_eval currently supported for NeMo version 2.0 models?
• If yes, how can I properly configure or resolve the missing config error [manually change the name]?
• If not, is there a roadmap or timeline for adding NeMo v2.0 support to lm_eval?