smoke/fumes from your fire going up the flue (chimney) is dependant upon differential air pressure between the hot air of the fumes and the cold air outside and is greatly affected by wind conditions but also by other factors such as sufficient ventilation in your room, effective flue height etc, flue performance is also affected by having the internal doors to your room open, flue pull will not be as great with the doors open.
you can see the smoke coming into your room but exactly the same thing can happen with a gas burning appliance and you wouldn't be able to see the fumes coming back into your room then and could be in real danger.
it may be worth getting your flue assessed by a corgi registered gas engineer as the job involves assessing flues and flue performance not just gas appliances.
the revolving stainless steel flue terminal someone mentioned are designed to improve flue pull but they can be a nuisance with the sun glinting off them into the neighbours windows/eyes and they are meant to be oiled regulary