If the UN rules were properly applied they'd have no option, mushroom.
I think I've discussed the "Le Touquet" agreement before but I do not believe that agreement would be torn up in the event of a "Brexit". France (and indeed the rest of Europe) depends more on cross-channel trade than the UK does. Furthermore the port of Calais has just announced a plan to spend 700 million euros expanding their operations there:
http://www.lloydsloadinglist.com/freight-directory/news/Calais-posts-annual-freight-record/61597.htm#.VoqiwFKWe-d
Trade between the UK and the Continent will not suddenly cease if the UK leaves the EU. In fact the chances are it will grow faster than if we remained. If the UK border was shifted back from Calais/Dunkirk/Bolougne/Coquelles to Dover/Folkestone the UK authorities would simply ensure that all illegals found trying to disembark from ships and trains were returned forthwith (as per the rules). It would be far more difficult for France to deal with this than it would for the UK and would hinder the efficiency on both sides of the channel. However there are other European gateways to and from the UK apart from the French ports and it would be commercial suicide for Calais if a sizeable chunk of their UK traffic was moved elsewhere.
In short the threat of abandonment of the "Le Touquet" agreement is a scaremongering tactic which actually holds little water.