Add metadata to list item contents for snippets

For list items that are not treated as files, we provide their metadata
as contents so that the GSA can form snippets for found words. Although
we mark the metadata in the content to not be indexed (because it is
provided via headers as well), it is still used when forming the
snippets.

Since SharePoint commonly has HTML in the metadata and we can't safely
treat the HTML as HTML nor can we be guaranteed to properly detect HTML,
we always treat metadata values as text and we minimally try to detect
HTML. When we detect HTML we just strip its tags and resolve a few HTML
entities. This is simply so the HTML does not show up in the snippets.

We go ahead and print all the metadata keys in addition to their values
simply because it makes the cached version marginally less random/ugly
and it is more helpful when debugging, since we are printing the
metadata values anyway.
3 files changed