You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The ability to process parts of the sync that do not effect each other.
This likely requires a facility that developers can describe what can be parallel as well as potentially finding specific scenarios where it can automagically happen.
Use Case
Use cases that would benefit from this include:
Getting Side A and Side B at the same time
Getting all of table 1 and all of table 2 from Side A at the same time
Update the model where there is no relationships at the same time
A parent of multiple children, each of those children can run at the same time
isn't this something you could infer based on the children relationships? I mean, all the object of one type, e.g site, should be able to be processed in parallel. Obviously, you can find shared children, but this conflict can be postponed for a final merge.
Don't know if assuming, by default, that the objects of parent models can run in parallel makes sense. Then, if the library offers a remediation final step using the already supported Redis memory, you could run multiprocessing.
does it make sense to you? or maybe missing some point?
I'm not quite sure I follow the intention behind this request. It sounds more akin to the parallel processing FR for SSoT that was referenced. For DiffSync I could potentially see parallel processing of the DiffElements in diff_to/from(). Possibly in the sync_from/to() also but that's trickier due to model interdependencies.
If I am syncing all of my locations and all of my devices, I should be able to run those both at the same time, as an example.
But that's nothing to do with DiffSync itself. DiffSync is simply comparing the data once you've loaded it in. How you load it in is up to you and that can easily be done in parallel. In fact, I recently updated SSoT to do just this.
Okay, good point. Re-reading my issue, I think the best example is the last one.
Multiprocessing the devices given a location.
So the CRUD operations? I agree entirely that it'd be great to figure out a why to delegate this work to a queue or pool of workers to be worked as they can but then you run into dependency issues. Definitely something I'd love to get figured out though. Doing that I think we'd need to have some way of having multiple relationships outside a 1:1 parent/child relationship and potentially somehow documenting interdependencies.
Yea, that is basically it. I don't remember the originating reason this came up tbh. Overall, I think it is a good improvement, but likely easier areas to address with larger impact, before this one.
Activity
chadell commentedon Feb 10, 2023
isn't this something you could infer based on the children relationships? I mean, all the object of one type, e.g
site, should be able to be processed in parallel. Obviously, you can find shared children, but this conflict can be postponed for a final merge.Don't know if assuming, by default, that the objects of parent models can run in parallel makes sense. Then, if the library offers a remediation final step using the already supported Redis memory, you could run multiprocessing.
does it make sense to you? or maybe missing some point?
itdependsnetworks commentedon Feb 10, 2023
Some can be inferred, some may not be able to be inferred, but yes, that is the point
jdrew82 commentedon Dec 3, 2025
I'm not quite sure I follow the intention behind this request. It sounds more akin to the parallel processing FR for SSoT that was referenced. For DiffSync I could potentially see parallel processing of the DiffElements in diff_to/from(). Possibly in the sync_from/to() also but that's trickier due to model interdependencies.
itdependsnetworks commentedon Dec 3, 2025
If I am syncing all of my locations and all of my devices, I should be able to run those both at the same time, as an example.
jdrew82 commentedon Dec 3, 2025
But that's nothing to do with DiffSync itself. DiffSync is simply comparing the data once you've loaded it in. How you load it in is up to you and that can easily be done in parallel. In fact, I recently updated SSoT to do just this.
itdependsnetworks commentedon Dec 3, 2025
Okay, good point. Re-reading my issue, I think the best example is the last one.
Multiprocessing the devices given a location.
jdrew82 commentedon Dec 3, 2025
So the CRUD operations? I agree entirely that it'd be great to figure out a why to delegate this work to a queue or pool of workers to be worked as they can but then you run into dependency issues. Definitely something I'd love to get figured out though. Doing that I think we'd need to have some way of having multiple relationships outside a 1:1 parent/child relationship and potentially somehow documenting interdependencies.
itdependsnetworks commentedon Dec 4, 2025
Yea, that is basically it. I don't remember the originating reason this came up tbh. Overall, I think it is a good improvement, but likely easier areas to address with larger impact, before this one.