They didn't evolve to live on clothes in closets. They were around before that! They evolved to live on animal fur and in birds' nests. They have adopted a new habitat, on our woollen clothes in closets, which, to them, resembles the original one. They no more evolved to live with us than house sparrows, swallows ( US: 'barn swallows'), or house martins did.
They are poor flyers. If disturbed, they fly only a very short distance and seem to prefer crawling. That fits them well for life in their original habitat and now in their adopted one with us. You don't find many spiders living in the moths' original habitat ; in fact, the moths may be in more danger in our houses, since there are various species of spider that are happy living in the larger habitat of a house, finding there a variety of insects, and a moth that flies feebly or crawls may blunder into the path of the hunting spiders or into the webs of some spiders that we find in dark corners at home.These spiders did not have to evolve to have a specific diet of clothes moths; clothes moths are just part of their general diet.