`Search.get_items()`: `query` vs `self.parameters` divergence

[dale-wahl]

Almost the same, but different. When looking at implementing validate_query into BasicProcessor classes more broadly, I asked myself if we need to add query (which only exists in Search) to be used for our many processors.

	query (= dataset.get_parameters())	self.parameters
Question it answers	"What did the user + validate_query actually decide?"	"What's the effective value of every option right now?"
Source	raw stored dict	stored dict ∪ get_options(), defaults back-filled (processor.py:188)
Honors validate_query deletions?	yes	no — re-adds the default
Excluded conditional options?	absent	present, filled with default

I noted the deletion issues, which is more of a footgun than a bug necessarily. We do this for daterange a lot where we unpack to min_date/max_date and then del query["daterange"]. self.parameters['daterange'] returns the default value (established in get_options) while query.get('daterange') returns None. So a potential bug if you are not aware of the difference.

The second divergence is with conditional options (e.g. when we use requires to gate options). I have actually run into this before but did not dive into the why until now. It's the same sort of problem where self.parameters['gated_option'] returns the default while query.get('gated_option') returns None (i.e. "user never saw the option"). Again, fine if you know about it, but BasicProcessors do not even have query so have to do the same requirement checks as get_options in order to determine if the self.parameters value is user defined.

The final is a purposeful or load-bearing which needs to be accounted for in any change. delete_parameter does not delete the self.parameters in memory (on purpose) which is fed to query. This allows the query to have sensitive parameters (e.g. API key) even though the parameter was already deleted (and get_parameters() would return a new dict without the deleted parameter). Again on purpose.

What does this all mean? BasicProcessor would benefit from query in that it tells us what the user actually selected (or didn't) particularly for validate_query methods to be implemented cleanly. I think that means making self.parameters more "faithful" to only add applicable defaults instead of all defaults. It may be useful to also separate "what is persisted" and "what the running worker sees" instead of relying on a stale cache of self.parameters for sensitive data.