They don't tell you about this in seminary or in leadership training: throughout ministry you will bump into people who essentially want you to be short and tall at the same time. Here's how it works. Some will assume you are not working if your car is not at the church. Others will assume you are not working (calling, tending the flock) if your car is in the church parking lot. If you try to lead, some will think that you are trying to be a dictator and others will think you aren't forceful enough. Some will think you should be taking every evening to call on prospective new people while other will assume you need to spend more time with your family. Some will think you preach too long, others will say that you are too shallow because you don't preach long enough. You get the point.
The fact is that people gauge and judge your performance in ministry on a variety of things: their previous experience, their exposure to what you do, their projection of work style from their own occupations, and sometimes what they have heard from other people who have the same limitations. Trying to prove yourself to all these different people is tantamount to playing spiritual "whack-a-mole", trying to address each concern that raises its head. It is part of the reason so many people leave ministry.
So, what to do? There are several things that are helpful and one that is necessary.
--Make sure your job description is readily available and rehearsed in front of your leadership.
--Have a group (pastor-parish committee, personnel committee, elders) that you talk to about your ministry and act as your advocates.
--Ask yourself the question: Should I work harder or smarter? (go with the latter)
Be clear about your calling and what you believe God is wanting you to spend your time on. This trumps everything. You will never be able to please everyone, but you must strive to please God and make peace with that being enough. To expect anything else leads to disillusionment, one of the biggest enemies of ministry.