This work presents a framework to develop optimal coils of arbitrary geometry for deep transcranial magnetic stimulation (dTMS). It has been based on a continuous current density inverse boundary element method (IBEM) to generate dTMS coils with depth control and minimum power dissipation. Novel coil geometries that readily adapted to human head have been studied to provide focal stimulation in areas on both prefrontal cortex and right temporal lobe. The designed dTMS coils performance has been numerically validated in a realistic human head model and prototypes have been built for experimental evaluation. The numerical simulations indicate that the proposed dTMS coils focally stimulate the prescribed brain regions with the desired target depth. The calculated metrics demonstrate that the presented designs outperform existing dTMS coils (the stimulation depth can be increased more than 15% compared to conventional dTMS coils with similar focality). Stream function IBEM can be used to develop novel dTMS coils with improved properties for a wide range of geometries. The proposed design method opens up a possibility for exploring new coil solutions based on complex shapes to focally deliver a desired stimulation dose to relatively deep brain targets, which might allow novel and more effective brain stimulation protocols.