I had a lot of anxiety about leaving my son after he was born, and had to really force myself out of it.
Mine was more of a control issue - as in would his Dad or other people look after him properly, what if something happened or they couldn't cope, what if they didn't feed him enough/properly. It was a real issue, and my (now ex) partner really got annoyed as he felt I didn't trust him or think him capable of looking after him. Once I did actually start taking the time out I began to relax a bit more, and it actually helped my relationship with my son. Although he's a pest now, so it didn't last long, lol!