[fstachura]

This series is a continuation of labels series published by Stephen Finucane
back in 2018.

Changes since v2:

• Implemented a command for relabeling existing patches
• Added a labels filter with autocomplete
• Added a labels filter to the API
• Updated selectize to the latest version

I have started implementing series -> patch labels inheritance. It's not
included in this series as I'm not sure how it should work. As I understand
it, the name of the series may be changed as new patches are parsed.
I'm also not sure where in the UI users could see/modify labels for series.

This version uses a many-to-many relation for storing labels assigned to a
patch, like in v2. I understand that there is a performance concern as this
could cause excessive JOINs in queries. From what I can see, Django only uses
JOINs when patches are filtered by labels. When patches are just listed without
any filters, a separate query to the patch-labels table is executed instead
of a JOIN, that query seems to be rather cheap.

I have a POC alternative version that uses JSON columns to store an array with
labels inside the patches table.

So far it's not obvious to me that the JSON version is better performance-wise
(in my tests it sometimes was, sometimes not, especially on MySQL, that could be
an issue with my setup or the tests). Downsides: the code is more complex,
and I think there may be some issues around update atomicity.

Some of the commits from v2 were modified beyond just rebase and formatting.
I have retained the original author, unless the changes were really significant.
I'm submitting these patches now, I have added myself to Signed-off-by.
I'm not sure if this is 100% OK, so please let me know.

Sponsored-By: The Linux Foundation
Closes: #22

[fstachura]

I rebased the series on current main and fixed some selectize-related bugs (unnecessary escaping, some options were missing from patch forms selectize, it also was impossible to type after selecting an item from the dropdown).

I'm planning to make the following changes, likely by Friday:

• Make labels case-insensitive - I noticed that people sometimes capitalize patch prefixes
• Remove labels from patch name - I dropped a commit related to that from v2 as it didn't work, but in the end I think that the redundancy is annoying
• Store labels as json - In the description I mention that storing labels as json didn't make an obvious difference. I believe that it does with query optimization patches that were merged recently. I will post benchmark results when I'm done.

[fstachura]

I gave up on the JSON version, I think it makes no sense if many database engines are supported. GIN indexes in Postgres work well and are supported by Django. Postgres also supports array datatype, which is better suited for this task. Multi-valued indexes in MySQL have some limitations and are not officially supported by Django (although the database sill uses them in ordinary queries). The latest, mostly complete version of my JSON attempt is on labels_json_str branch of my fork.

MySQL multi-valued index issues

The main issue I have with the MySQL multi-valued index is that rows with empty arrays are not added to the index at all. This becomes a problem with multi-column indexes (the idea was to have another ~covering index, ex. project_id, archived, state_id, labels...) as the database may use such index even when the query does not filter by labels. In that case, all rows with an empty list of labels will be skipped from the results. Rows with SQL NULLs (but not JSON nulls, those cause an error) are supported by the index. So technically, it is possible to work around this problem with a custom field that saves empty arrays as SQL NULL. I find this workaround kind of hacky.

In the end, I have managed to optimize the filtering query further, to the point where performance is comparable with my JSON attempts. Indexes introduced with recent query optimization patches really help (thanks for adding the primary key to the new patches index). If more optimizations are needed, it's probably be best to just cache commonly queried Patch fields in the PatchLabels table, with an index on that.

Benchmark results

The benchmark was done in the default view (not archived, action required) on a project with 300k patches and 5 labels. The instance stores 2 million patches. The first label is assigned to roughly 50% of the patches, the second to ~25% and so on. The test involved filtering by various combinations of the labels (first, first and second..., all labels, first label and not the second label, first, second but not third...) and traversing up to 200 pages of the results.

MySQL average: 701 ms (m2m) vs 652 ms (json)
Postgres average: 427ms (m2m) vs 398ms (json)

Both methods perform similarly when filtering by a single label (349 ms for JSON vs 449 ms on Postgres, 648 ms for JSON vs 679 ms on MySQL), for m2m performance gets worse the more labels are included in the filter. With JSON on Postgres it's the opposite, which is not surprising as the GIN index is optimized for overlap filtering.

I have also added some features I find optional:

• Modifying patch labels over the API
• Label descriptions rendered as tooltips - I don't see the point of the description field otherwise
• Cover->patch label inheritance - I remember that the original idea was to allow patches to inherit labels from series. I guess this implements a half of it. I'm not sure how to properly detect the first patch from a series without a cover letter. Such patch probably should not have In-Reply-To, but is this guaranteed to always be true?
• Fixes for one test that I believe is flaky - dependabot also hit this in build(deps-dev): update django-dbbackup requirement from ~=5.3.0 to ~=5.4.0 #681