feature/debug

[rabah-khalek]

ToDo

• spec
• check the outdated branch of this feature.
• converge on the technical implementation
• to be filled

comment (deprecated)

I found couple of issues with the previous implementation:
https://github.com/Giskard-AI/giskard/tree/GSK-106_Reduce_the_number_of_test_outputs

1. Our test should be agnostic to the type of dataset we use. Once we have CV, we might not have pandas.DataFrame anymore. Therefore, the slicing operation shouldn't be only valid for the latter.
2. We should return a dataset and not a df that can be ready to be rendered in the UI (with new name, uuid, etc.).
3. The debug_filters along which we define the slices_to_debug should not be duplicated and defined per test, rather collected somewhere for readability and better handling.

Here's my first preliminary proposal:

• a function that returns a fresh_copy of a dataset:
  https://github.com/Giskard-AI/giskard/blob/3d9ae1e0b20185979b5fab8ec6d977147e2a23a1/python-client/giskard/ml_worker/testing/tests/debug_utils.py#L13-L17
• creating a filter catalogue where each filter has the name of the test it belongs to (valid for any type of dataset we support):
  https://github.com/Giskard-AI/giskard/blob/3d9ae1e0b20185979b5fab8ec6d977147e2a23a1/python-client/giskard/ml_worker/testing/tests/debug_utils.py#L20-L26
• adding slices_to_debug to the TestResult by using the debug_filters catalogue:
  https://github.com/Giskard-AI/giskard/blob/3d9ae1e0b20185979b5fab8ec6d977147e2a23a1/python-client/giskard/ml_worker/testing/tests/performance.py#L45-L49
  so basically:
  • debug_filters.get(test_name) will get the filter for the corresponding test.
  • (gsk_dataset, prediction) are the parameters needed by this filter.
  • returns a list of slices (Datasets) assigned to slices_to_debug.

This will result in:

@andreybavt and @jmsquare let me know what you think about this.

minutes of 12/04/2023

After the 12/04/2023 discussion with @jmsquare and @andreybavt :

• add debug arg to the tests
• The filter has to return a list of either booleans or ints
• The name slice for Dataset might not be ideal -- to spec (deprecated)
• The catalogue idea to review by @andreybavt (abandonned)
• For next week we should have at least one representative slice per type of tests:
  • performance
  • statistical
  • metamorphic
  • drift

https://linear.app/giskard/project/test-debug-feature-0024fb6da8c5?filter=eyJhbmQiOlt7InN0YXRlIjp7Im5hbWUiOnsibmluIjpbIkRvbmUiXX19fV19

[rabah-khalek]

In this commit 19de6c6:

details (deprecated)

• I added the possibility of data.slice(..., get_mask=True) which returns a boolean mask as a numpy.array.
• The decision on the type of mask output was based on the fact that it was the least memory occupying object:
  
• I dumped the idea of the catalogue in favour of gathering all the debugging slicing functions here (where we are already collecting them) -- the tag "hidden" is to be communicated to @kevinmessiaen in order to now show these function in the UI (unless they're relevant).
• Here's a small demo of what is expected:
  
  we can see that by caching the mask we're gaining 40x in memory (as opposed to caching the sliced dataset)

Update 1

Update: regarding the second point, I think that's a more fair comparison to do (that favours int masks):
--> assuming we expect the accuracy of inspected models to be > 50%.

Example:
>>> a = np.ones(100)
>>> b = np.zeros(100)
>>> c = (a == b) # bool mask of 100 entries
>>> d = a.astype(int)[:50] # int mask of 50 entries (assuming 50% accuracy)

>>> sys.getsizeof(c)
212
>>> sys.getsizeof(d)
112

Update 2

Python bool is 1 byte, int is 4 bytes. So if the model is at least 75% correct it’s cheaper to store int line numbers and not bool map.

[andreybavt]

where is it used?

[rabah-khalek]

@Googleton could you answer this one?

[rabah-khalek]

According to @Googleton :

it’s what we use for the frontend to know which debugging session to open

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
14 Code Smells

55.2% Coverage
1.9% Duplication