How do I restrict outbound calls from a number to a specific agent?

Under each number's setting, you'll find an option that says Accessible by. You can enter a group's name there, and from then on, only that group will be able to use that number to make outbound calls.