As you discovered, you don't need to put anything in the box, the birds do ot all themselves,
As for whats happened, assuming the birds checking it out aren't the same ones that cleaned it, somethindg may have happened to one of the original pair, (cat, hawk etc), in which case the remining bird won't come back, hpefully there are no eggs there, but I would think, if the others aren't making a home there, they may believe it to be occupied.
If Tbird+s around, he may be able to come up with something else.