"Following" a person means that you save any messages that are written by them or their follows. This is a separate thing from "connecting": connecting lets you ask the pub if they have seen any new messages from the feeds you're saving. It does not require you to be saving the pub's feed, unless one of your follows is following that pub.

The pubs you follow are in your Friends + Follow lists just like people are. They really are just regular accounts that are reachable through the internet. If you have a publicly reachable IP address you can announce it for friends to connect with you directly.**

Cons of unfollowing all pubs: If you are behind a NAT or firewall and do not connect to anyone locally, you want to make sure that the pub is still following you, or one of your friends. If the pubs unfollow you and anyone who follows you, then your messages will never leave your computer.

Pros of unfollowing pubs: The public pubs get a lot of new people, so not following them means avoiding a lot of boring "Hi i'm new" messages and the occasional troll. Patchwork is better at hiding the extent of the problem than Patchbay, but these messages are still taking up space on your hard-drive.

** it looks like the gossip module might not automatically interpret these yet.