There is a configuration under Admin -> Phone -> Edit-> Settings called "Accessible by" using which you can make the phone channel available only for a specific group or groups.