Yeast does not like dramatic changes in temperature but unlike brewing too hot, low temperatures do not create problems as far as taste or clarity.
When the temperature drops below the optimum temperature for the yeast you are using it simply goes dormant. When the brew heats up the yeast will activate again.
Of course in this situation you do want to warm the brew up, wrap it in a towel/blanket, use a heater pad or heater belt to maintain a constant temperature where possible. It is more detrimental brewing too hot than too cold.