Everyone has dreams and desires.  You can choose to keep them a fantasy or bring them into reality anytime you wish.  It's always your decision.  Whatever you decide know that it is right for you.  Nobody can tell you what is right or wrong for you, only you can decide that. 
One bit of caution, however, is that when making your decision, choose to do so by listening to your heart and not by listening to guilt, shame, anger, fear, or lust.  Those things will not guide you to the right answer and to a happy outcome.  But your heart will.

For instance, you may have a sexual fantasy about another who is married.  You are physically attracted to this person but you aren't really in love with them.  Because they are in love with you, you have the ability to entice themfor your own gratification even though you don't want a relationship with them.  In this case, you believe that making your fantasy into a reality would be a selfish thing to do and that it may hurt other people, so you choose not to.  Your heart was simply not in it.  In listening to your heart you made the right decision.

In a different scenario, your dream is to be in a deeply loving marriage and partnership.  You are currently married but in love with another.  You believe that if you stay in your current marriage you will be a liar and a fake by denying and suppressing this dream and this love for another.  Love no longer exists in your current marriage and you've exhausted all efforts in reviving it.  You believe in the importance of always acting out of integrity, even if it seems other people may get hurt.  And you believe that the truth truly sets everyone free and only good can come of that.  Here your heart is engaged in the decision to bring your dream into reality.  The decision is not selfish as it is based on your true values that lie in your heart, not based on having the one you are in love with.  

Dreams and desires are what keep us going and growing.  Heart centered ones are the true reflections of what is important to us and are in alignment with who we are.  They represent our highest views of our true, authentic self.  We are here in this lifetime to discover who we really are and experience the goodness that brings.  When we are not being true to ourselves and out of alignment with our heart's desires, we may continuously experience painful lessons to teach us about integrity and authenticity.   Bad karma is the result.  You have heard the old adage "What goes around, comes around" which is another meaning for karma. One example of this type of karma would be if the person in the first scenario was led by their sexual desire instead of their heart, and suffered painful and humiliating consequences.

Every heart centered decision to bring a dream or desire into reality is guaranteed to bring positive results.  It may not appear that way initially and may take some time to reap the rewards of that decision, but it is bound to be a joyful success.  Recognize when you are not listening to the truth in your heart and then resolve to engage your heart in the decision.  Then choose to bring that true heart's desire into reality and enjoy all the goodness it brings!