You say builders don't need to be smart on the job, but I see more and more who are expected to conform to certain dress codes. At work we've recently had a lot of work done on the building and more than once I have overheard the site manager giving someone grief because they were working without their shirt on or were wearing seriously torn jeans.
I think what to wear for interview, though, can depend on the job and possibly what's expected at the interview. A job I went for once (and got) was as a technician in a music venue. It's a job where jeans and t-shirts is the norm. At the interview we had to demonstrate an ability to climb, lift and carry and someone 'in the know' advised me not dress too smartly as 'you'll be laughed out of the room'. I wore a clean, wholesome pair of jeans and rugby shirt in similar condition. Apparently one woman turned up in a short skirt and high heels! Another bloke did the whole suit thing. Neither of them were taken on, even though the man, at least, had far more experience than I had.
So I don't think formal outifts are always necessary, but you should always make the effort to be clean and presentable at interview. I once read somewhere that an ideal interview outfit is a smarter version of something you would expect to wear on the job.