Skip to content

Parallel Processing #202

Open
Open

Description

@itdependsnetworks

Environment

  • DiffSync version: latest

Proposed Functionality

The ability to process parts of the sync that do not effect each other.

This likely requires a facility that developers can describe what can be parallel as well as potentially finding specific scenarios where it can automagically happen.

Use Case

Use cases that would benefit from this include:

  • Getting Side A and Side B at the same time
  • Getting all of table 1 and all of table 2 from Side A at the same time
  • Update the model where there is no relationships at the same time
  • A parent of multiple children, each of those children can run at the same time

Activity

  1. chadell commented on Feb 10, 2023

    @chadell
    Contributor

    isn't this something you could infer based on the children relationships? I mean, all the object of one type, e.g site, should be able to be processed in parallel. Obviously, you can find shared children, but this conflict can be postponed for a final merge.
    Don't know if assuming, by default, that the objects of parent models can run in parallel makes sense. Then, if the library offers a remediation final step using the already supported Redis memory, you could run multiprocessing.
    does it make sense to you? or maybe missing some point?

  2. itdependsnetworks commented on Feb 10, 2023

    @itdependsnetworks
    ContributorAuthor

    Some can be inferred, some may not be able to be inferred, but yes, that is the point

  3. jdrew82 commented on Dec 3, 2025

    @jdrew82
    Contributor

    I'm not quite sure I follow the intention behind this request. It sounds more akin to the parallel processing FR for SSoT that was referenced. For DiffSync I could potentially see parallel processing of the DiffElements in diff_to/from(). Possibly in the sync_from/to() also but that's trickier due to model interdependencies.

  4. itdependsnetworks commented on Dec 3, 2025

    @itdependsnetworks
    ContributorAuthor

    If I am syncing all of my locations and all of my devices, I should be able to run those both at the same time, as an example.

  5. jdrew82 commented on Dec 3, 2025

    @jdrew82
    Contributor

    If I am syncing all of my locations and all of my devices, I should be able to run those both at the same time, as an example.

    But that's nothing to do with DiffSync itself. DiffSync is simply comparing the data once you've loaded it in. How you load it in is up to you and that can easily be done in parallel. In fact, I recently updated SSoT to do just this.

  6. itdependsnetworks commented on Dec 3, 2025

    @itdependsnetworks
    ContributorAuthor

    Okay, good point. Re-reading my issue, I think the best example is the last one.

    Multiprocessing the devices given a location.

  7. jdrew82 commented on Dec 3, 2025

    @jdrew82
    Contributor

    Okay, good point. Re-reading my issue, I think the best example is the last one.

    Multiprocessing the devices given a location.

    So the CRUD operations? I agree entirely that it'd be great to figure out a why to delegate this work to a queue or pool of workers to be worked as they can but then you run into dependency issues. Definitely something I'd love to get figured out though. Doing that I think we'd need to have some way of having multiple relationships outside a 1:1 parent/child relationship and potentially somehow documenting interdependencies.

  8. itdependsnetworks commented on Dec 4, 2025

    @itdependsnetworks
    ContributorAuthor

    Yea, that is basically it. I don't remember the originating reason this came up tbh. Overall, I think it is a good improvement, but likely easier areas to address with larger impact, before this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions