tree: 830dd2465d85966bcb45f49d0465cfd140225d56 [path history] [tgz]
  1. bin/
  2. lib/
  3. test/
  4. tool/
  5. .gitignore
  6. analysis_options.yaml
  7. mono_pkg.yaml
  8. pubspec.yaml
  9. README.md
pkgs/sdk_triage_bot/README.md

What's this?

A LLM based triage automation system for the dart-lang/sdk repo. It processes new issues filed against the repo and triages them in the same manner that a human would. This includes:

  • re-summarizing the issue for clarity
  • assigning the issues to an area- label (first line triage)

Bot trigger and entry-point

This bot is generally triggered by a GitHub workflow listening for new issues on the dart-lang/sdk repo.

See https://github.com/dart-lang/sdk/blob/main/.github/workflows/issue-triage.yml.

Overview

The general workflow of the tool is:

  • download the issue information (existing labels, title, first comment)
  • ask Gemini to summarize the issue (see prompts)
  • ask Gemini to classify the issue (see prompts)
  • create a comment on the issue (@dart-github-bot) with the summary; apply any labels produced as part of the classification

Tuning

We create a tuned Gemini model in order to improve the performance of classification. This involves:

  • downloading recent, already triaged issues (~800 issues)
  • writing them in a format suitable for tuning (either .csv or .jsonl)
  • tuning via the Gemini APIs; this gives us a new model name to use when calling Gemini (e.g., model: 'tunedModels/sdk-triage-tuned-prompt-1l96e2n')

See tool/create_tuning_data.dart.