I am a strong supporter of Brexit, but I would like Britain to follow it by building an alternative, without the flaws the EU has. Immigration has never been my problem with the EU - TTIP and workers rights are; along with the idiotic idea of the Eurozone. I would support open borders and a free trade area within Commonwealth countries *with similar economies*. Realistically this would be Britain, Canada, Australia and New Zealand (the so-called Anglosphere, minus America, which has a far bigger economy and would thus dominate any union there... bad). Ireland could also join if/when it left the EU (or, hopefully, the EU collapses entirely)
I would however be quite happy to accept the Carribbean and Pacific Island countries. I would not support them joining any potential future currency union between CANZUK because that would replicate the flaw of the eurozone - unbalanced economies joining it and thus ending up with a new bunch of Greece, Portugal, Cyprus etc. But their demographics would not unbalance a trade zone, and they don't have economies based around doing what we do for less money, so there is no reason to keep them out.
As for the African and Asian members of the Commonwealth - I don't support them joining a free-trade zone with us simply because it'd be disastrous for our economies, by making outsourcing of jobs and relocation of factories that much easier, and I support tariffs and protectionism there. Visa-free travel would be nice, but total freedom of movement is not good. Those members of the Commonwealth have massively higher populations with far lower wages, and that just doesn't work out well for us.
Immigration from countries which have a significantly lower GDP per Capita and lower wages creates a downward pressure on wages for natives; which is why large-scale migration from poorer countries is bad. Until we live in a Global superstate (which I hope we never live to see) unilateral action toward one is a terrible idea. The social side of immigration doesn't especially bother me, because if it's well planned then it doesn't really cause significant issues. Bradford in the UK is a prime example of bad planning, but as of yet Germany has not had any such modern day "ghetto" crop up.
A governments primary duty is to its own people, not toward those of other nations. Inequality could be fixed (or at least lessened) at a national level. At international level that's just a pipedream.