Skip to content

Smart Start can generate impossible filter sets

A reasonable guiding principle for the LLM-driven Smart Start is that it should not be able to generate a combination of filters that no human could generate by manually clicking around the interface. If the LLM can't be prevented from generating such a combination, some part of the system should recognize it happened and fix things before putting the user interface -- and thus the user -- into an impossible state. (Like East Carolina, or Texas.)

For example, @kfogel entered a search string which generated the following combination of filters:

  • Key Words:
    • India
    • Waste
  • Primary Subject Area:
    • Clean water supply
    • Irrigation and water management
    • Water access, sanitation, and hygiene
    • Water conservation
    • Water pollution
    • Water resources
    • Water utilities

However, if a human (including but not limited to @kfogel) went to Explore and manually selected the "India" and "Waste" keywords, the Primary Subject Area options available to them would be limited to these:

Primary Subject Areas

That's because the other subject areas, which the LLM decided were relevant to the query, don't intersect with Keywords=India,Waste in the actual database — but the LLM doesn't know that.

In fact, for a real user to "back out" of the LLM response and get back to any usable responses, they would have to deselect all of the suggested Primary Subject Areas: seven clicks and queries before getting any results at all.

This issue is an expression of the problem.

Edited by Justin Reese