The same people affected by extreme weather are also likely to worry that they will face a future disaster, according to a new poll