Skip to content

Adjust script for use with analysis support

Chris Zubak-Skees requested to merge revise-analysis-script into main

This MR revises the script at scripts/llm/draft-analysis.py:

  1. Formatting the code.
  2. Revising the comments to include missing dependencies.
  3. Summarizing panel comments into three bullet points with one prompt (further prompt engineering to come later).
  4. Making the LLM API endpoint configurable.
  5. Correcting an error that prevented it from finding the LLMProposal template at the correct path.
  6. Using a template section that just includes panel comments.
  7. Removing the specific competition name from the file name and configuration, so it can be reused for other contests.
  8. Attempting to address an error if the LLM LFC Analysis field was not already part of the proposal data.

The result appears to give us an MVP of a summary for the new Analysis Support direction.

Steps to Test

pipenv install torqueclient jinja2
sudo apt-get install pandoc
cd scripts/llm
cp config.py.tmpl config.py
$EDITOR config.py
pipenv run ./draft-analysis.py --dry-run

Where config.py should contain something like:

TORQUE_PASSWORD = "<REDACTED>"
TORQUE_USER = "<REDACTED>@opentechstrategies.com"
TORQUE_URL = "https://torque.leverforchange.org/GlobalView"
LLM_API_KEY = "<REDACTED>"
LLM_ENDPOINT = "http://localhost:8889"

COMPETITION = "<REDACTED>"
PROPOSALS = [218]

For this to work, the LLM API has to be set up and running.

Here's what I get when I do this:

About to generate draft analyses for:

  * #<REDACTED>

*** Prompt and LLM Draft Analysis for #<REDACTED>: ***

{'Overview': {'llmgenerated': 'llm',
              'generating': False,
              'prompt': {'sent': '\n'
                                 "        Summarize the panel's comments into "
                                 'three concise bullet points.\n'
                                 '\n'
                                 '        Create this analysis using the '
                                 'following comments:\n'
                                 '        ',
                         'display': '\n'
                                    "        Summarize the panel's comments "
                                    'into three concise bullet points.\n'
                                    '        ',
                         'extrainstructions': ''},
              'value': 'Here are three concise bullet points summarizing the '
                       "panel's comments:\n"
                       '\n'
                       '• The project demonstrates strong <REDACTED...> '
                       '\n'
                       '• The solution has the potential to effect significant '
                       '<REDACTED...>'
                       '\n'
                       '• The team has a nuanced understanding of '
                       '<REDACTED...>'
              'id': 'run-507fe9e7-1ddf-4ddf-ba57-048b348b40a4-0'}}
Edited by Chris Zubak-Skees

Merge request reports

Loading