Abstract Hand gestures are an effective method of communication, especially when we are communicating with people who cannot understand our spoken language. Furthermore, it is a key aspect to human–computer interaction. Understanding hand gestures is very important to ensure that listeners understand what speakers are attempting to communicate. Even though several researchers have proposed deep learning-based models for hand gesture recognition, the hyper-parameter tuning of these models is a relatively unexplored area. In this work, Convolutional Neural Networks (CNN) are used to classify hand gesture images. To tune the hyper-parameters of the CNN, a recently developed metaheuristic algorithm, namely, the Harris Hawks Optimization (HHO) algorithm, is used. Our in-depth comparative analysis proves that the proposed HHO-CNN hybrid model outperforms the existing models by attaining an Accuracy of 100%.