You would have to manage your SLA policy under Admin > Workflow> SLA policy so that the SLA works only during the Business hours and make sure you set your business hours under Admin > Business hours.