Question around DeepSpeed-MoE

[hyhuang00]

Thank you for contributing to this beautiful work! I have some questions regarding section 5 of the DeepSpeed-MoE paper:

1. The first paragraph says that "the inference latency of an MoE model depends on the time it takes to load the model parameters from the main memory", and I'm wondering why it is the case here. Isn't it the case that the model parameters can be loaded to GPUs for once? If that is the case, then the cost for loading the model should get amortized and not becoming a major bottleneck here.
2. For experiments being done in this section, what is the setting here? Specifically, would you be able to provide some information regarding what kind of task the model has been executing, what is the batch size and sequence length, and how are the latencies measured? I would like to ask this question because I find Figure 11 very confusing. For Fig. 11(a), the DeepSpeed-MoE model is achieving a latency of 10ms, or 0.01 sec, and each GPU has a throughput around 800 tokens per second. With 128 GPUs, that gives around 102400 tokens per second in total, which is also about 1024 tokens per 10ms. This is different from the 2k sequence length in the training side. I feel like I am missing something here.

Any help will be greatly appreciated!

Edit: Would you be able to help me on this issue? @tjruwase

[tjruwase]

@hyhuang00, apologies for the delay. I don't work on this, but someone should respond soon.

[RezaYazdaniAminabadi]

Hi @hyhuang00,

Thanks for your interest in our paper, and apology for the late response.

1. Yes, you are right there is only one-time cost to bring the data to GPU. But the data-movement we are referring to here is from the GPU memory to the GPU cores, and since the batch size is relatively small for inference, we only read them once to do the computation and that becomes the main bottleneck.
2. The latency and throughput evaluation we did in the paper is regarding the generation tasks. For the batch size, we use per-GPU batch size of 8 queries. The throughput measurement is done based on the average latency to generate a token per query. So, for Fig.11(a), the throughput calculation is like this: (1/10 * 1000 * 8 * 128) / 128 (note that we are showing the per-GPU throughput, that's why there is a division by 128).

I hope this helps and answers your questions.
Please let me know if there is any other question.
Thanks,
Reza

[hyhuang00]

Thank you for your thorough explanation! I have a followup question on the throughput evaluation. For each query, how many tokens will we ask the model to generate? Suppose we are using incremental decoding or similar techniques during the inference, then the first forward will takes more time than the later forward runs, right? Is there any beam search involved in this process?

Best.
Haiyang

[hyhuang00]

Hi @RezaYazdaniAminabadi,

Thank you again for your help! I am currently modifying the generate_samples_gpt.py and try to utilize the DeepSpeed inference system for my own model. I have run into a bunch of issues when using the DS-MoE inference module that I cannot get rid of on my own -- none of these issues appear when DS-inference is disabled. Some of these issues, along with my environment, have already been reported here. The most important one particularly the following:

1. The FP16 kernel for attention qkv calculation seems to be broken. As shown in the above GitHub issues, the fused qkv matrix may generate zeros while it should not. Such zeros will leads to NaNs in later computation and make the result unreliable
2. When running on small scale inputs (strings that are less than 10 tokens), the cublas_gemm_ex kernel and cublas_strided_batched_gemm may generate kernel execution error. Moreover, when I run my script with multiple devices, some of them can execute these kernels normally while the others cannot. The configuration and the input being sent to these devices are completely the same, which makes me super confused.
3. Following Q2, even though I have specified a batch_size for my script, it seems like the batches being sent to different devices are the same. Is there a reason behind this setting or there are other parameters that controls the batchsize?
4. When running on some strings that have more than 10 tokens, the gating function will hang when executing the all2all communication collective. The bandwidth between my devices are sufficiently large and I'm only using 200MB per device regarding the memory, and I have no idea why this is happening.

I would appreciate if you are able to share some more pointers towards how to run the script with DS-inference features enabled. Thanks again for your attention!