@J12t Yes, that's certainly one use-case for the outbox... to "catch up" on things you missed, on a per-user basis.
But as far as I am concerned they're separate APIs that pull from a common data source.
For example, reading the ActivityPub spec, one could draw the conclusion that the outbox is an ordered list of activities, likely read as-is from the database.. but that's not strictly defined, so it's possible to dynamically create outbox items on-demand based on user history stored elsewhere.
That would mean less duplication of content (and the syncing that is required), fewer bugs, etc... but the cost is the collection then has to be dynamically constructed.