Computational model-based analyses of behavioral and neural responses to natural sounds offer insights into the acoustic-to-semantic transformations involved in sound recognition. However, the inherent relation between low-level/intermediate features and semantic dimensions in natural stimuli complicates interpretation. Here, we present a method to identify optimal sets of natural sounds, minimizing the dependence between modeled representations at acoustic/intermediate and semantic level. We applied this approach in a behavioral experiment where participants made pairwise similarity judgments of sounds and semantic labels describing them. Our findings demonstrate that sound similarity judgements were most accurately modeled by the intermediate layers of a sound-to-event DNN (Yamnet), with minimal contribution of a semantic model (word2vec), whereas semantic similarity judgements exhibited the opposite pattern. These results highlight the effectiveness of our approach in dissociating acoustic and semantic processing of natural sounds, providing a framework for investigating further the neural computations underlying the processing of such stimuli.