Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE REQUEST] - Apply different filters on Source vs Target environments #935

Closed
nktnet1 opened this issue Nov 27, 2024 · 10 comments · Fixed by #936 · May be fixed by #937
Closed

[FEATURE REQUEST] - Apply different filters on Source vs Target environments #935

nktnet1 opened this issue Nov 27, 2024 · 10 comments · Fixed by #936 · May be fixed by #937
Assignees
@hknokh
Labels
completed The issue was successfully resolved/Feature is completed feature-request New feature request or a change in the existing functionality

Comments

@nktnet1
Copy link
Contributor

nktnet1 commented Nov 27, 2024 •
edited
Loading

I want to UPSERT a subset of my Event objects from Production to Sandbox.

The query I'm using is:

SELECT
    id,
    Start_Date1__c
FROM
    Event__c
WHERE
    CALENDAR_YEAR(Start_Date1__c) >= 2024

However, the type of Start_Date1__c is Roll-Up Summary (MIN Block).
For context, a parent Event can have many children Blocks, and the Block with the minimum start date becomes the Start_Date1__c for the Event.
I am not migrating the Block object to Sandbox.

This causes an issue with UPSERT, as the WHERE filter condition will only find Events in Production, but not Sandbox (i.e. the External ID of Sandbox Events are not tracked), thus Events unique by their externalId will be inserted multiple times (instead of UPSERT, where an UPDATE is expected).

I've looked into the targetRecordsFilter option, and I believe what I need might be the inverse of that, e.g. sourceRecordsFilter.
Alternatively, perhaps the ability to specify the sourceQuery and targetQuery separately.

Is there currently a workaround for this? Otherwise, I'll convert this question to a feature request.

Thanks ~
@nktnet1 nktnet1 added the help-wanted User need a help or something not working, not a bug label Nov 27, 2024
@hknokh
Copy link
Collaborator

hknokh commented Nov 27, 2024

Hello, @nktnet1

Thank you for reaching out.
I will do my best to assist you as quickly as possible and will keep you updated on my progress.

Cheers

@hknokh
Copy link
Collaborator

hknokh commented Nov 27, 2024

Hello,

Thank you for reaching out to me.
Based on the information you've provided, it appears that this is not a bug related to the Plugin.
For issues of this nature, please consult the SFDMU Help Center documentation at https://help.sfdmu.com/ or check the developer forums.
Additionally, you can review our support policy at https://help.sfdmu.com/full-documentation/additional-information/support_policy to understand the types of issues that fall within the scope of my assistance.
Thank you for your understanding and cooperation.

Regards.

@nktnet1 nktnet1 changed the title [QUESTION] - How to apply filter to only source records during an UPSERT? [FEATURE-REQUEST] - How to apply filter to only source records during an UPSERT? Nov 27, 2024
@nktnet1
Copy link
Contributor Author

nktnet1 commented Nov 27, 2024 •
edited
Loading

Thanks @hknokh - I think it could be considered a "Bug/Issue" in a sense (or at least not clear/easy to debug), since the UPSERT operation will only ever perform INSERT due to the filtering condition functioning differently on the Source vs Target environment.

Following the linked support policy, I've re-titled the issue to be a Feature Request, with the goal being the ability to apply filters differently on the Target vs Source environment. Is this okay?

@nktnet1 nktnet1 changed the title [FEATURE-REQUEST] - How to apply filter to only source records during an UPSERT? [FEATURE REQUEST] - Apply different filters on Source vs Target environments Nov 27, 2024
@hknokh
Copy link
Collaborator

hknokh commented Nov 27, 2024

OK, got it. Thanks for the suggestion.

Basically for this moment I do not plan to add any new features to the SFDMU.
However, I’m adding this ticket to my roadmap for review at a later time, but please note there's no guarantee of implementation. I’ll keep you updated on any progress if available.

Regards.

@hknokh hknokh closed this as completed Nov 27, 2024
@hknokh hknokh added feature-request New feature request or a change in the existing functionality roadmap This case is on the roadmap to be reviewed/implemented in future and removed help-wanted User need a help or something not working, not a bug not-supported labels Nov 27, 2024
@nktnet1 nktnet1 mentioned this issue Nov 28, 2024
Merged
@nktnet1
Copy link
Contributor Author

nktnet1 commented Nov 28, 2024 •
edited
Loading

I've made a pull request - #936, if you don't mind taking a look.
No rush though, will just be using my own fork.

@hknokh2
Copy link
Contributor

hknokh2 commented Nov 28, 2024

Thank you for the PR.
I will take a look on it as soon as possible.

@hknokh hknokh added completed The issue was successfully resolved/Feature is completed and removed roadmap This case is on the roadmap to be reviewed/implemented in future labels Nov 28, 2024
@hknokh
Copy link
Collaborator

hknokh commented Nov 28, 2024 •
edited
Loading

Hey
I reviewed and merged your PR.
It looks ok after my visual code review without actual testing how it works.
Please confirm that you've also tested it in a live scenario, when the original query string does already contain WHERE clause having multiple conditions with parentheses, and in this case the actual source query is constructed as expected by adding defined extra source filter expression in a correct way.
Thanks.

@hknokh hknokh reopened this Nov 28, 2024
@nktnet1
Copy link
Contributor Author

nktnet1 commented Nov 28, 2024 •
edited
Loading

Hi @hknokh,

Yes, I've tested it with complex nested WHERE clauses in both the original query and the additional sourceRecordsFilter simultaneously.

I'm actively using it in my organisation's production and sandbox environments.

I left some sample unit tests in this repository here:

@nktnet1 nktnet1 mentioned this issue Nov 28, 2024
Open
@hknokh
Copy link
Collaborator

hknokh commented Nov 28, 2024 •
edited
Loading

Thank you for the update. I will document this new feature soon.

@hknokh hknokh closed this as completed Nov 28, 2024
@nktnet1
Copy link
Contributor Author

nktnet1 commented Nov 28, 2024

No worries - thanks @hknokh.

I made some style fixes + jsdocs (developer documentations) in #937. No functional changes, if you don't mind taking a look. No rush & low priority.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hknokh hknokh

Labels
completed The issue was successfully resolved/Feature is completed feature-request New feature request or a change in the existing functionality
Projects
None yet
Milestone
No milestone
3 participants
@hknokh @nktnet1 @hknokh2