refactor: introduce JsonPathStr type for JSONPath validation by davidberenstein1957 · Pull Request #2264 · Giskard-AI/giskard-oss

davidberenstein1957 · 2026-02-18T17:06:11Z

Updated comparison, groundedness, semantic similarity, and string matching checks to use the new JsonPathStr type for key fields.
Added validation for JSONPath syntax in the extraction module.
Introduced unit tests for JSONPath validation and enforcement tests to ensure all relevant fields use JsonPathStr.

Description

Related Issue

Type of Change

📚 Examples / docs / tutorials / dependencies update
🔧 Bug fix (non-breaking change which fixes an issue)
🥂 Improvement (non-breaking change which improves an existing feature)
🚀 New feature (non-breaking change which adds functionality)
💥 Breaking change (fix or feature that would cause existing functionality to change)
🔐 Security fix

Checklist

I've read the CODE_OF_CONDUCT.md document.
I've read the CONTRIBUTING.md guide.
I've written tests for all new methods and classes that I created.
I've written the docstring in Google format for all the methods and classes that I used.
I've updated the pdm.lock running pdm update-lock (only applicable when pyproject.toml has been
modified)

- Updated comparison, groundedness, semantic similarity, and string matching checks to use the new JsonPathStr type for key fields. - Added validation for JSONPath syntax in the extraction module. - Introduced unit tests for JSONPath validation and enforcement tests to ensure all relevant fields use JsonPathStr.

linear · 2026-02-18T17:06:15Z

OSS-2 Validate JSONPath expressions at initialization

gemini-code-assist · 2026-02-18T17:06:35Z

Summary of Changes

Hello @davidberenstein1957, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the robustness and maintainability of JSONPath usage across the codebase by introducing a dedicated JsonPathStr type. This type provides automatic validation for JSONPath expressions, ensuring that all such fields conform to expected syntax and structure. By integrating this type into existing checks and adding an enforcement test, the change prevents common errors related to malformed JSONPaths and promotes consistent data extraction logic.

Highlights

Introduced new JsonPathStr type: A new Pydantic Annotated type, JsonPathStr, has been introduced in the extraction module. This type enforces that string fields intended to be JSONPath expressions adhere to valid JSONPath syntax and start with the 'trace.' prefix.
Updated existing checks to use JsonPathStr: Key fields in ComparisonCheck, Groundedness, SemanticSimilarity, and StringMatching checks that represent JSONPath expressions (e.g., key, answer_key, context_key, reference_text_key, actual_answer_key, text_key, keyword_key) have been updated to use the new JsonPathStr type, ensuring automatic validation.
Added unit tests for JSONPath validation: Comprehensive unit tests have been added for the _validate_jsonpath_syntax function and the JsonPathStr Pydantic type, covering various valid and invalid JSONPath scenarios.
Implemented enforcement test for JsonPathStr usage: A new enforcement test (test_jsonpath_enforcement.py) has been added to automatically verify that all fields named 'key' or ending in '_key' within Check subclasses are correctly annotated with JsonPathStr (or its optional variants), preventing future regressions.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

libs/giskard-checks/src/giskard/checks/builtin/comparison.py
- Imported JsonPathStr from ..core.extraction.
- Updated the key field type from str to JsonPathStr.
- Updated the expected_value_key field type from str | NotProvided to JsonPathStr | NotProvided.
libs/giskard-checks/src/giskard/checks/builtin/groundedness.py
- Imported JsonPathStr from ..core.extraction.
- Updated the answer_key field type from str to JsonPathStr.
- Updated the context_key field type from str to JsonPathStr.
libs/giskard-checks/src/giskard/checks/builtin/semantic_similarity.py
- Imported JsonPathStr from ..core.extraction.
- Updated the reference_text_key field type from str to JsonPathStr.
- Updated the actual_answer_key field type from str to JsonPathStr.
libs/giskard-checks/src/giskard/checks/builtin/string_matching.py
- Imported JsonPathStr from ..core.extraction.
- Updated the text_key field type from str to JsonPathStr.
- Updated the keyword_key field type from str | None to JsonPathStr | None.
libs/giskard-checks/src/giskard/checks/core/extraction.py
- Imported Annotated, AfterValidator from typing and pydantic respectively.
- Imported JsonPathLexerError, JsonPathParserError from jsonpath_ng.exceptions.
- Defined _JsonPathStrMarker class for internal metadata tracking.
- Implemented _validate_jsonpath_syntax function to validate JSONPath expressions, ensuring they start with 'trace.' and have correct syntax.
- Defined JsonPathStr as an Annotated type using str with _validate_jsonpath_syntax as an AfterValidator and _JsonPathStrMarker for metadata.
libs/giskard-checks/tests/core/test_extraction.py
- Added a new test file.
- Implemented TestValidateJsonpathSyntax with tests for valid and invalid JSONPath expressions, including prefix validation and syntax errors.
- Implemented TestJsonPathStrAnnotatedType to verify JsonPathStr behavior with Pydantic models, including valid paths, invalid syntax, missing prefixes, and optional usage.
libs/giskard-checks/tests/core/test_jsonpath_enforcement.py
- Added a new test file.
- Implemented an enforcement test to ensure all fields named 'key' or ending in '_key' within Check subclasses are correctly annotated with JsonPathStr.

Activity

No human activity recorded yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

The pull request introduces a new JsonPathStr type for JSONPath validation, enhancing type safety and ensuring correct JSONPath syntax in various check implementations. This is a good improvement for code quality and robustness. The changes correctly apply the new type to relevant fields in comparison, groundedness, semantic similarity, and string matching checks, and include appropriate unit and enforcement tests.

…at-initialization

libs/giskard-checks/src/giskard/checks/core/extraction.py

henchaves · 2026-02-19T11:15:19Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a JsonPathStr type to validate JSONPath expressions at model creation time, which is a great improvement for robustness and user experience. The implementation is solid, using Pydantic's Annotated type and AfterValidator to enforce that paths are syntactically correct and start with the required 'trace.' prefix. The changes are consistently applied across all relevant check classes. The addition of comprehensive unit tests for the validation logic and a clever enforcement test to ensure future compliance demonstrates strong engineering practice. Overall, this is an excellent refactoring that enhances code quality and maintainability.

mattbit

I noted a quick fix on naming convention :)

mattbit · 2026-02-20T10:15:21Z

libs/giskard-checks/src/giskard/checks/builtin/string_matching.py

        description="The text string to search within. If None, extracted from trace using text_key.",
    )
-    text_key: str = Field(
+    text_key: JsonPathStr = Field(


Minor thing but for consistency, JSONPathStr would be the recommended PEP8 style:

When using acronyms in CapWords, capitalize all the letters of the acronym. Thus HTTPServerError is better than HttpServerError.

mattbit · 2026-02-20T10:17:43Z

libs/giskard-checks/src/giskard/checks/core/extraction.py

+    """Marker placed in JsonPathStr metadata. Used by the enforcement test."""
+
+
+_REQUIRED_JSONPATH_PREFIX = "trace."


I'm wondering: what's the use of forcing each key to start with trace? If this is the only thing allowed, shouldn't we extract the trace attributes instead?
E.g. trace.last -> last. Not needed in this PR, but maybe worth a quick discussion on the desired interface here.

to make clear that is coming from the trace

davidberenstein1957 requested a review from henchaves February 18, 2026 17:06

github-actions bot added the Scope: Checks label Feb 18, 2026

gemini-code-assist bot reviewed Feb 18, 2026

View reviewed changes

davidberenstein1957 and others added 2 commits February 19, 2026 07:29

Merge branch 'main' into feature/OSS-2-validate-jsonpath-expressions-…

c3aa85c

…at-initialization

Merge branch 'main' into feature/OSS-2-validate-jsonpath-expressions-…

0ccfa7f

…at-initialization

henchaves requested changes Feb 19, 2026

View reviewed changes

libs/giskard-checks/src/giskard/checks/core/extraction.py Outdated Show resolved Hide resolved

henchaves added 3 commits February 19, 2026 12:09

extract trace suffix to a constant

1ca5db4

add edge cases to test_extraction

20a89d8

add explanation to meta test

6b0d684

gemini-code-assist bot reviewed Feb 19, 2026

View reviewed changes

henchaves approved these changes Feb 19, 2026

View reviewed changes

henchaves assigned davidberenstein1957 Feb 19, 2026

henchaves requested a review from mattbit February 19, 2026 11:32

mattbit requested changes Feb 20, 2026

View reviewed changes

rename JsonPathStr to JSONPathStr

6b12803

henchaves requested a review from mattbit February 20, 2026 15:44

style: fix formatting issues

d73c012

mattbit approved these changes Feb 26, 2026

View reviewed changes

mattbit merged commit 807ecdd into main Feb 26, 2026
17 checks passed

mattbit deleted the feature/OSS-2-validate-jsonpath-expressions-at-initialization branch February 26, 2026 10:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refactor: introduce JsonPathStr type for JSONPath validation#2264

refactor: introduce JsonPathStr type for JSONPath validation#2264
mattbit merged 8 commits intomainfrom
feature/OSS-2-validate-jsonpath-expressions-at-initialization

davidberenstein1957 commented Feb 18, 2026

Uh oh!

linear bot commented Feb 18, 2026

Uh oh!

gemini-code-assist bot commented Feb 18, 2026

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

henchaves commented Feb 19, 2026

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

mattbit left a comment

Uh oh!

mattbit Feb 20, 2026

Uh oh!

mattbit Feb 20, 2026

Uh oh!

henchaves Feb 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

		"""Marker placed in JsonPathStr metadata. Used by the enforcement test."""


		_REQUIRED_JSONPATH_PREFIX = "trace."

Uh oh!

Conversation

davidberenstein1957 commented Feb 18, 2026

Description

Related Issue

Type of Change

Checklist

Uh oh!

linear bot commented Feb 18, 2026

Uh oh!

gemini-code-assist bot commented Feb 18, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

henchaves commented Feb 19, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

mattbit left a comment

Choose a reason for hiding this comment

Uh oh!

mattbit Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

mattbit Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

henchaves Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants