If you have a previous history of PE, particularly if it was unprovoked (and you are no longer anti coagulated) I would say it would be sensible to exclude this. Although if your only symptom is breathlessness on exertion it does seem quite unlikely. In my experience actually very few people present with leg swelling or significant chest pain with PE.
Most hospitals have ambulatory care units with specific PE pathways to try and diagnose and treat people quickly avoiding the need to go through a+e. It’s a shame your GP didn’t refer to this. Did you actually speak to a medic or just a receptionist?