feat(generators): introduce personas and extended context for UserSimulator

[davidberenstein1957]

• Added PersonaSimulator class to simulate user interactions with predefined or custom personas.
• Implemented output structure for client descriptions and conversation management.
• Created a Jinja2 template for persona instructions and context handling.
• Added tests to validate functionality and ensure consistent client descriptions across conversation turns.

Description

closes #ENG-1388

Related Issue

Type of Change

• 📚 Examples / docs / tutorials / dependencies update
• 🔧 Bug fix (non-breaking change which fixes an issue)
• 🥂 Improvement (non-breaking change which improves an existing feature)
• 🚀 New feature (non-breaking change which adds functionality)
• 💥 Breaking change (fix or feature that would cause existing functionality to change)
• 🔐 Security fix

Checklist

• I've read the CODE_OF_CONDUCT.md document.
• I've read the CONTRIBUTING.md guide.
• I've written tests for all new methods and classes that I created.
• I've written the docstring in Google format for all the methods and classes that I used.
• I've updated the pdm.lock running pdm update-lock (only applicable when pyproject.toml has been
  modified)

[linear]

ENG-1388 Provide default responder classes (UserResponder, PersonaResponder) for random interactions

[gemini-code-assist]

Summary of Changes

Hello @davidberenstein1957, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new PersonaSimulator class designed to enhance user simulation capabilities within the system. It allows for the creation of realistic user interactions based on either predefined or custom personas, incorporating contextual information to guide their behavior. The primary impact is to provide a robust tool for testing and evaluating systems against diverse user profiles, ensuring consistent client descriptions throughout multi-turn conversations.

Highlights

• New Feature: Introduced the PersonaSimulator class to enable user simulation with predefined or custom personas.
• Output Structure: Implemented PersonaSimulatorOutput to ensure consistent client descriptions and structured conversation management.
• Templating: Created a Jinja2 template (persona_simulator.j2) for dynamic generation of persona instructions and context handling.
• Testing: Added comprehensive unit tests to validate the PersonaSimulator's functionality, including the persistence of client descriptions across conversation turns.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• libs/giskard-checks/src/giskard/checks/init.py
  • Imported PersonaSimulator from the new generators module.
  • Added PersonaSimulator to the __all__ export list.
• libs/giskard-checks/src/giskard/checks/generators/init.py
  • Created a new __init__.py file for the generators subpackage.
  • Exported UserSimulator and the newly added PersonaSimulator.
• libs/giskard-checks/src/giskard/checks/generators/persona.py
  • Added the PersonaSimulator class, inheriting from InputGenerator and WithGeneratorMixin.
  • Defined PersonaSimulatorOutput for structured output, including client_description, goal_reached, and message.
  • Implemented the __call__ method for asynchronous generation of persona messages, managing conversation steps and client description persistence.
• libs/giskard-checks/src/giskard/checks/prompts/generators/persona_simulator.j2
  • Created a Jinja2 template to define predefined personas and generate instructions for the LLM based on the persona and context.
  • Included logic for generating a detailed client_description on the first turn and reusing it in subsequent turns.
• libs/giskard-checks/tests/generators/test_persona.py
  • Added a MockPersonaGenerator for testing purposes.
  • Implemented tests for accepting predefined and custom personas, handling context, generating and maintaining client descriptions across turns, and respecting max_steps.

Activity

• No human activity has been recorded on this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a PersonaSimulator for simulating user interactions, complete with a Jinja2 template for prompt generation and a comprehensive test suite. The implementation is well-structured and robust. I've suggested a couple of minor improvements to add validation to the pydantic model fields for persona and max_steps to make the class even more robust against invalid inputs.

[davidberenstein1957]

@kevinmessiaen @henchaves, I feel this also introduces some discussion on how to pass along and re-use information from Traces in preceding interactions or checks when using the n_runs setting. If we used the current persona setup, the persona defined during the first interaction would not be reused in subsequent interactions, because we would need to re-initialize the class. I'd expect to have the options to do something like trace.last.interactionspec to re-use the same class instance.

We could pass some class information along by default as metadata, or we can configure something more robust. Happy to hear your thoughts.

[kevinmessiaen]

@kevinmessiaen @henchaves, I feel this also introduces some discussion on how to pass along and re-use information from Traces in preceding interactions or checks when using the n_runs setting. If we used the current persona setup, the persona defined during the first interaction would not be reused in subsequent interactions, because we would need to re-initialize the class. I'd expect to have the options to do something like trace.last.interactionspec to re-use the same class instance.

We could pass some class information along by default as metadata, or we can configure something more robust. Happy to hear your thoughts.

@davidberenstein1957 I not sure to understand the issue with n_runs. To me if we have multiple runs, they should all be self-contained and not sharing information.

Regarding reusing the persona, I feel like the simpliest fix would be to allow describe a reusable persona and create simulators from persona.

helpfull_persona = Persona("helpful_user", ...) # Persona only hold information on how to generate message, not the what to generate

simulator_1 = helpfull_persona.simulator("Describe your problem installing giskard v3", max_turn=3)
simulator_2 = helpfull_persona.simulator("Say that you tried the solution but you get resolution error, you are using python 3.10", max_turn=3)

# Change persona dynamically (assume inside a scenario
simulator_3 = Persona("thankfull_user", ...) .simulator("State that it's working now with python 3.13", max_turn=1)

[davidberenstein1957]

@kevinmessiaen

[kevinmessiaen]

Looking good to me, just some small comments

[kevinmessiaen]

It's not documented what are the predefined persona. It would be nice to provide the list in the docstring

[kevinmessiaen]

While it's concise to have a single str field I'm fearing of potential issues:

• Typos won't be noticed: persona=helpfull_user or persona=helpful-user will work without error, but the instruction in the prompt won't be what we expect them to be
• No error if a persona has been removed/deleted (even if we should not make breaking changes, if this happens it will be easy to miss this)

Shall we have an enum of persona?

class Persona(StrEnum):
    HelpfulUser = "helful_user"
    ...

# In the UserSimulator class
persona: Persona | str = Field(...)

# Then we can instaciate persona safely
UserSimulator(persona=Persona.HelpfulUser)
UserSimulator(persona="helful_user") # Will act as Persona.HelpfulUser but without type safety
UserSimulator(persona="You are an angry customer") # Custom prompt

This would also help fixing the documentation issue, as it'll be possible to explore personas with autocompletion/autodoc