Keras benchmarks slow on a 256 threads machine as compared to 16 threads laptop

[planetprivate]

Why would a 256 threads machine slower in benchmarks as compared to a laptop with 16 threads when running.

python model_benchmark/image_classification_benchmark.py

Any suggestions glad to test.

• Both run ubuntu 24.04 LTS (code from keras master).
• Both machines had no other processes or users to interfere
• Server has 20TB disk
• Laptop has 1 x 1TB NVMe
• There are no gfx cards in either machine
• Server does use almost all of the cpu
• Yes, I tested it around 5 times with similar results.

nproc= 256
RAM 1TB
AMD EPYC 7713 64-Core Processor (supermicro server)
Wall Time: 9496.9490 seconds.
Validation Accuracy: 0.9819
examples_per_second: 2.2220

nproc=16
16GB RAM
AMD Ryzen 7 PRO 5850U (thinkpad P14s)
Wall Time: 4028.4497 seconds.
Validation Accuracy: 0.9841
examples_per_second: 5.3173

This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.

SERVER

python model_benchmark/image_classification_benchmark.py 
2025-08-29 20:35:56.554513: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-08-29 20:35:56.557552: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2025-08-29 20:35:56.566919: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1756492556.582181  883852 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1756492556.586874  883852 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-08-29 20:35:56.603314: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
AttributeError: 'MessageFactory' object has no attribute 'GetPrototype'
AttributeError: 'MessageFactory' object has no attribute 'GetPrototype'
AttributeError: 'MessageFactory' object has no attribute 'GetPrototype'
AttributeError: 'MessageFactory' object has no attribute 'GetPrototype'
AttributeError: 'MessageFactory' object has no attribute 'GetPrototype'
2025-08-29 20:35:59.810072: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:152] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
I0829 20:35:59.810239 131813355827328 image_classification_benchmark.py:102] Benchmarking configs...
=========================
MODEL: EfficientNetV2B0
TASK: image classification/dogs-vs-cats 
BATCH_SIZE: 4
EPOCHS: 1
=========================
I0829 20:35:59.923239 131813355827328 dataset_info.py:707] Load dataset info from /home/ubuntu/tensorflow_datasets/cats_vs_dogs/4.0.1
I0829 20:35:59.951896 131813355827328 reader.py:262] Creating a tf.data.Dataset reading 15 files located in folders: /home/ubuntu/tensorflow_datasets/cats_vs_dogs/4.0.1.
I0829 20:36:00.037844 131813355827328 reader.py:262] Creating a tf.data.Dataset reading 2 files located in folders: /home/ubuntu/tensorflow_datasets/cats_vs_dogs/4.0.1.
I0829 20:36:00.061944 131813355827328 logging_logger.py:49] Constructing tf.data.Dataset cats_vs_dogs for split ['train[:90%]', 'train[90%:]'], from /home/ubuntu/tensorflow_datasets/cats_vs_dogs/4.0.1
INFO:2025-08-29 20:36:00,207:jax._src.xla_bridge:822: Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory
I0829 20:36:00.207858 131813355827328 xla_bridge.py:822] Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory
I0829 20:36:17.063706 131813355827328 image_classification_benchmark.py:137] Starting Training...
2025-08-29 20:36:17.129975: I tensorflow/core/kernels/data/tf_record_dataset_op.cc:376] The default buffer size is 262144, which is overridden by the user specified `buffer_size` of 8388608
 image_classification_benchmark.py:156] Training Finished! 
Wall Time: 9496.9490 seconds.
Validation Accuracy: 0.9819
examples_per_second: 2.2220

LAPTOP

Constructing tf.data.Dataset cats_vs_dogs for split ['train[:90%]', 'train[90%:]'], from /home/debian/tensorflow_datasets/cats_vs_
dogs/4.0.1
INFO:2025-08-29 19:12:35,145:jax._src.xla_bridge:822: Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or direc
tory
I0829 19:12:35.145455 140459198907712 xla_bridge.py:822] Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or di
rectory
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/efficientnet_v2/efficientnetv2-b0_notop.h5
24274472/24274472 ━━━━━━━━━━━━━━━━━━━━ 1s 0us/step  
I0829 19:12:53.689717 140459198907712 image_classification_benchmark.py:137] Starting Training...
2025-08-29 19:12:53.728752: I tensorflow/core/kernels/data/tf_record_dataset_op.cc:396] The default buffer size is 262144, which is overridden by the user specified `buffer_size` of 8388608
5234/5234 ━━━━━━━━━━━━━━━━━━━━ 4027s 766ms/step - loss: 0.1114 - sparse_categorical_accuracy: 0.9566 - val_loss: 0.0381 - val_sparse_categorical_accuracy: 0.9841
I0829 20:20:02.139775 140459198907712 image_classification_benchmark.py:156] Training Finished! 
I0829 20:20:02.139874 140459198907712 image_classification_benchmark.py:157] Wall Time: 4028.4497 seconds.
I0829 20:20:02.139915 140459198907712 image_classification_benchmark.py:158] Validation Accuracy: 0.9841
I0829 20:20:02.139949 140459198907712 image_classification_benchmark.py:159] examples_per_second: 5.3173