VCIO-next: Design new Improver #1395
Description
Context
Improvers are scripts that take a bunch of objects as an input and can selectively improve the data about packages and vulnerabilities. The current design is that Importers create Advisories and then a "Default Improver" does an actual proper Import of the data through an intermediate "Inference" object.
Other improvers have the same overall model:
- An Improver subclass implements an "interesting_advisories" method
- And then implements a "get_inferences" method that takes AvisoryData (a plain object not a model instance) and yields Inferences
The management command will control all this.
Problem
With the current design, we cannot do several things:
- We cannot improve if there are no advisories. For instance we cannot loop through some Package or Vulnerabilities
- We cannot do arbitrary operations on the data: we must go through an Inference
- We cannot control the frequency/periodicity at which we run improvers
Because of all this, there is also confusion in the community about how to add new improvers.
Solution
We should simplify improvers to the max.
An Improver should be a class that has a label, description and some run frequency and should have just an improve() method that can do ANYTHING it likes.
The default improver should be retired and its code merged in the Importer loop instead.
See also these related issues:
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
[-]Design new Improver[/-][+]RFC: Design new Improver[/+]pombredanne commentedon Jan 11, 2024
@TG1999 @Hritik14 @ziadhany @keshav-space @DennisClark @johnmhoran FYI
pombredanne commentedon Jan 11, 2024
Should we wait for the model refactoring?
I think not. In all cases we will have to amend improvers to work with the new models. We have many improvement work blocked by this current issue #1395 ... we may end up doing a bit more work if we do not wait, but this is marginal IMHO compared to waiting
armijnhemel commentedon Jan 15, 2024
Another solution is to create "pseudo-advisories", for example, one per purl.
pombredanne commentedon Aug 5, 2024
@keshav-space is this what you are working on with the pipelines?
keshav-space commentedon Aug 5, 2024
@pombredanne yes!
Merge pull request #1533 from aboutcode-org/917-ghost-remover
3 remaining items
[-]RFC: Design new Improver[/-][+]VCIO-next: Design new Improver[/+]pombredanne commentedon Jan 9, 2025
The design has been completed: we are now using the aboutcode.pipelines library extracted from scancode.io for both importers and improvers. Migrations is in progress and tracked in:
keshav-space commentedon Mar 28, 2025
Completed in: #1559 and #1621
The simple way to test this is to follow the below steps after installing VulnerableCode (https://github.com/aboutcode-org/vulnerablecode?tab=readme-ov-file#run-with-docker):
docker compose exec vulnerablecode ./manage.py improve flag_ghost_packages