Add state-specifying values at crawl-time

There are various values that specify some sort of state that can be
specified with a feed but not during retrieval. This commit adds support
for those values during crawling, but they are emulated by sending a
feed.

The eventual goal is to have all state be provided during retrieval and
only knowledge of documents' existance and events being provided through
feeds.
9 files changed