From past experience I'd tread very carefully.
I met a lovely guy, he didn't sweep me off my feet, I didn't have that phwoar factor for him but I did lovel him.
I have quite an eventful sexual past which he knew about but, during the time we were together, I just had no urge for it. I think it was a combination of factors, stress, illness, medication, lack of confidence...
It made him feel a similar way to what it sounds like you do though I never made comments about other women though maybe it's more a way of him covering up his inadequacies.
I found it very uncomfortable and difficult when he brought it up as I didn't know what it was and how to fix it and, because I didn't feel like doing it at all. I guess like talking about food and eating when the last thing you want to do is eat.
I was pushed into going to Relate by him and felt very uncomfortable about the whole thing. I only went because I thought it would threaten our relationship if I didn't.
I said I'd initially go on my own because of that and Relate themselved questioned me being there as it wasn't my choice. I went twice then we ended up splitting up.
My lack of sex drive with him was no reflection on how I loved him. To me, our relationship was about so much more than just sex. I never saw him as ugly or unattractive, I loved him.