Hmmm - I was told something similar when I went into a job. In my case, it was a call centre job (funny, but in my current job I'm the one who has to make tricky phone calls because "you have such a brilliant telephone manner"). Anyways, I stuck at for a year and was thoroughly miserable for most of that time. In the end, what with that and the flaws in the company system that were beginning to show up, I sat at my desk one day, typed out my resignation, emailed it to the boss and then went down to his office with a paper copy. I left four weeks later, having not found any other work, but the day I left, the temp agency called and offered me three months' work.
What I guess I'm trying to say is if you think that job's not you, you will never be happy, you will never give your best because when people tell you that's not good enough then it has a habit of demotivating you. You spend all your time wondering and worrying if they're going to pull you up on something.
If you walk ... what? You may be out of work for a while longer. You may be a little poorer. But you'll be happier. And there will be the ideal job for you sooner or later - maybe where you least expected it to appear. I found mine. It's part-time and I would never have imagined myself working here. Sure, I don't have as much money as I did, but then, what price sanity?