Under each number's setting, you will find an option that says Accessible by. Once you select a group name here, only the agents belonging to that specific group can use the number to make outbound calls.