I don't think the address metaphor is that off-the-mark -- real addresses are more segmented then IP addresses, true, but I don't know if that's really that much of a conceptual barrier.
Regarding ports, I would tweak your metaphor slightly -- a port is more like an apartment number, rather then the apartment itself. That is, the port just serves as an additional piece of information that's a part of your overall address, and can be used to route the mail to the appropriate apartment resident once your mail arrives.
(I guess to extend the metaphor, the people who live in the apartments are programs, and whoever sorts the mail at the apartment is like the operating system.)
What might also help is to explain that at the end of the day, a port is literally just a 16 bit int, nothing more, and that it's up to the receiving computer to do something reasonable with that number. If you emphasize that a port is just a number, it might make it a little easier for your students to independently derive what exactly you can and can't do with ports. After all, there are only so many things you can do with a single piece of metadata.
An alternative/complementary approach might be to focus less on metaphors and instead focus on motivating why we'd need ports in the first place. I'd personally start from the top down, and explain things like this:
- Start by explaining how IP addresses are used to route packets.
Explain that once a computer finally receives a packet, we're still not done: the operating system need to figure out which currently-running program to give it to, if at all.
This might also be a good moment to ask your students to discuss and try coming up with a solution themselves -- hopefully, most of them will independently realize they need to include metadata of some kind. You can turn this into a mini-design lesson: an ideal solution should be simple, minimizes packet size, is future-proof, flexible, doesn't break the transport layer abstraction...
Tell them about ports, and explain that was the solution people ended up coming up with.
The hope is that focusing more on motivation will allow your students to acquire a reasonable conceptual model of how network protocols tend to work, even if they forget the exact details.
I think this approach also does a better job of answering the implicit "why?" question people have when learning new things. (Why ports? Why IP addresses? Why TCP? Why abstract things like this? Why is everything so complicated? etc.) Basically, confront it head-on by presenting things in a problem/motivation -> solution approach.
Caveat: this approach might just be a reflection of how I personally like learning things, so YMMV.