Even though windows and doors don't consume energy, they can be a significant source of energy loss. A third of all heat from your home escapes from gaps and cracks.
There are a few ways you can check to see if you are losing precious heat in the winter or letting in heat in the summer.
Check for drafts using a thermal detector
Point the detector anywhere you suspect heat could be escaping or getting in. If you see a 5°C or more change, in either direction, you know you have a leaky spot. Check out Dave's DIY to see how a thermal heat detector works in action.
Other ways to check for drafts
DIY smoke test: On a cool and windy day, light a stick of incense and hold it up close to the areas you suspect air might be leaking. If the smoke moves back and forth, or if it's blown out, there's a leak.
Noise check: Open the window or door a fraction and listen to the noise level coming from outside, then close the window or door. If you can still hear clearly hear the outside noise, there's a draft. A good seal would cut out most of the noise.
Blower door test: If you want a professional assessment, energy auditors can come in and use a powerful fan to evaluate the overall air tightness of your home.