The Categories that you display would be the only categories that would be displayed to the customer on that specific portal. Even if the folders are made available for 'All Users', only if the Category is shown on the portal, the users would be able to view articles.