Adjust script for use with analysis support
This MR revises the script at scripts/llm/draft-analysis.py
:
- Formatting the code.
- Revising the comments to include missing dependencies.
- Summarizing panel comments into three bullet points with one prompt (further prompt engineering to come later).
- Making the LLM API endpoint configurable.
- Correcting an error that prevented it from finding the LLMProposal template at the correct path.
- Using a template section that just includes panel comments.
- Removing the specific competition name from the file name and configuration, so it can be reused for other contests.
- Attempting to address an error if the
LLM LFC Analysis
field was not already part of the proposal data.
The result appears to give us an MVP of a summary for the new Analysis Support direction.
Steps to Test
pipenv install torqueclient jinja2
sudo apt-get install pandoc
cd scripts/llm
cp config.py.tmpl config.py
$EDITOR config.py
pipenv run ./draft-analysis.py --dry-run
Where config.py
should contain something like:
TORQUE_PASSWORD = "<REDACTED>"
TORQUE_USER = "<REDACTED>@opentechstrategies.com"
TORQUE_URL = "https://torque.leverforchange.org/GlobalView"
LLM_API_KEY = "<REDACTED>"
LLM_ENDPOINT = "http://localhost:8889"
COMPETITION = "<REDACTED>"
PROPOSALS = [218]
For this to work, the LLM API has to be set up and running.
Here's what I get when I do this:
About to generate draft analyses for:
* #<REDACTED>
*** Prompt and LLM Draft Analysis for #<REDACTED>: ***
{'Overview': {'llmgenerated': 'llm',
'generating': False,
'prompt': {'sent': '\n'
" Summarize the panel's comments into "
'three concise bullet points.\n'
'\n'
' Create this analysis using the '
'following comments:\n'
' ',
'display': '\n'
" Summarize the panel's comments "
'into three concise bullet points.\n'
' ',
'extrainstructions': ''},
'value': 'Here are three concise bullet points summarizing the '
"panel's comments:\n"
'\n'
'• The project demonstrates strong <REDACTED...> '
'\n'
'• The solution has the potential to effect significant '
'<REDACTED...>'
'\n'
'• The team has a nuanced understanding of '
'<REDACTED...>'
'id': 'run-507fe9e7-1ddf-4ddf-ba57-048b348b40a4-0'}}
Edited by Chris Zubak-Skees