Is it really hard for people to believe that big people want to lose weight?

I have a body media, and for those of you who do not know what that is it is like a body bugg, it tracks my steps/calories burned/exercise minutes. Lots of people ask me "what is that" and I will just say "a heart rate monitor" because technically that is what it is. Then they will be like "oh, you have a heart condition?" and act like they feel sorry for me. Then I correct them and say "no, it is to track how many calories I burn", their whole reaction changes. The facial expressions they give make it seem like they want to say "yeah right, like you exercise", but all they say is "oh". I guess I am upset about it because that has happened twice this week, usually I only get asked once every couple weeks. It's like it is hard for people that are not overweight, or who have never had a weight issue to believe that bigger people actually do watch what they eat, exercise and try to lose weight. Most of society looks at us as lazy pigs.
But I guess in the end, I really should not care, because I am not doing it for THEM, I am doing it for me!