feat(checks): ENG-1390 judges directory for llm and prompt reliant checks

[davidberenstein1957]

• Added a new module for LLM-based judge checks, including BaseLLMCheck, Conformity, Groundedness, and LLMJudge.
• Refactored existing checks to import judges from the new module for better organization and maintainability.
• Implemented Jinja2 templating for dynamic rule evaluation in Conformity and context validation in Groundedness.
• Created prompt templates for both Conformity and Groundedness checks to standardize evaluation criteria.
• Enhanced the overall structure of the checks to support LLM evaluations more effectively.

Description

Related Issue

Type of Change

• 📚 Examples / docs / tutorials / dependencies update
• 🔧 Bug fix (non-breaking change which fixes an issue)
• 🥂 Improvement (non-breaking change which improves an existing feature)
• 🚀 New feature (non-breaking change which adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to change)
• 🔐 Security fix

Checklist

• I've read the CODE_OF_CONDUCT.md document.
• I've read the CONTRIBUTING.md guide.
• I've written tests for all new methods and classes that I created.
• I've written the docstring in Google format for all the methods and classes that I used.
• I've updated the pdm.lock running pdm update-lock (only applicable when pyproject.toml has been
  modified)

[linear]

ENG-1390 Judges directory for LLM and prompt-reliant checks

[gemini-code-assist]

Summary of Changes

Hello @davidberenstein1957, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly refactors the giskard.checks module by creating a dedicated judges directory for all LLM-based checks. This change improves the organization and maintainability of the codebase by centralizing related functionalities and their corresponding prompt templates, making it easier to manage and extend LLM evaluation capabilities.

Highlights

• New judges Module Introduced: A dedicated judges module has been created to centralize all LLM-based checks, including BaseLLMCheck, Conformity, Groundedness, and LLMJudge, improving code organization.
• Refactoring of LLM-Related Checks: Existing LLM-related checks and their associated prompt templates have been moved from the builtin and checks directories to the new judges module and judges prompt directory, respectively.
• Updated Module Imports: Module imports across the giskard.checks package have been updated to correctly reference the new judges module, ensuring proper functionality and maintainability.
• Jinja2 Template Path Adjustments: Jinja2 template paths within Conformity and Groundedness checks were adjusted to reflect their new location in the judges prompt directory.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• libs/giskard-checks/src/giskard/checks/init.py
  • Removed direct imports of BaseLLMCheck, Conformity, Groundedness, LLMCheckResult, and LLMJudge from builtin.
  • Added judges to the module imports.
  • Imported BaseLLMCheck, Conformity, Groundedness, LLMCheckResult, and LLMJudge from the new judges module.
  • Added judges to the __all__ export list.
• libs/giskard-checks/src/giskard/checks/builtin/init.py
  • Removed direct imports of BaseLLMCheck, LLMCheckResult, Conformity, Groundedness, and LLMJudge.
  • Added imports for BaseLLMCheck, Conformity, Groundedness, LLMCheckResult, and LLMJudge from ..judges for backward compatibility.
• libs/giskard-checks/src/giskard/checks/builtin/base.py
  • Renamed to libs/giskard-checks/src/giskard/checks/judges/base.py.
• libs/giskard-checks/src/giskard/checks/builtin/conformity.py
  • Renamed to libs/giskard-checks/src/giskard/checks/judges/conformity.py.
  • Updated get_prompt method to reference giskard.checks::judges/conformity.j2.
• libs/giskard-checks/src/giskard/checks/builtin/groundedness.py
  • Renamed to libs/giskard-checks/src/giskard/checks/judges/groundedness.py.
  • Updated get_prompt method to reference giskard.checks::judges/groundedness.j2.
• libs/giskard-checks/src/giskard/checks/builtin/judge.py
  • Renamed to libs/giskard-checks/src/giskard/checks/judges/judge.py.
• libs/giskard-checks/src/giskard/checks/judges/init.py
  • Added new file to define and export BaseLLMCheck, LLMCheckResult, Conformity, Groundedness, and LLMJudge.
• libs/giskard-checks/src/giskard/checks/prompts/checks/conformity.j2
  • Renamed to libs/giskard-checks/src/giskard/checks/prompts/judges/conformity.j2.
• libs/giskard-checks/src/giskard/checks/prompts/checks/groundedness.j2
  • Renamed to libs/giskard-checks/src/giskard/checks/prompts/judges/groundedness.j2.

Activity

• No specific activity (comments, reviews, progress) has been recorded for this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request effectively refactors the LLM-based checks into a new judges module, which is a solid improvement for code organization and future maintainability. The changes correctly handle backward compatibility by re-exporting the moved classes. My review includes one suggestion to add a DeprecationWarning to guide users towards the new API and simplify future code cleanup.