Task/rdmp 122 update a database extraction #1779

JFriel · 2024-03-14T09:28:48Z

Adds ability to re-extract a project to a database
Extract to database now respects the "IsExtractionPrimaryKey" flag on column items

How To Test:

Create a simple catalogue and extract it to a database
Update the catalogue with new data and attempt to re-extract
The new data should be extracted into the target database

…sk/RDMP-122-update-a-database-extraction

…onToDatabaseMSSql.md

Rdmp.Core.Tests/DataLoad/Engine/Integration/DataTableUploadDestinationTests.cs

Rdmp.Core/DataExport/DataExtraction/Pipeline/Sources/ExecuteDatasetExtractionSource.cs

Rdmp.Core/DataLoad/Engine/Pipeline/Destinations/DataTableUploadDestination.cs

@@ -103,6 +108,9 @@
    private bool _firstTime = true;
    private HashSet<string> _primaryKey = new(StringComparer.CurrentCultureIgnoreCase);
    private DiscoveredTable _discoveredTable;
+    private readonly string _extractionTimeStamp = "extraction_timestamp";
+
+    private IExternalCohortTable _externalCohortTable;


...DataExport/DataExtraction/ExecuteFullExtractionToDatabaseMSSqlDestinationReExtractionTest.cs

+                new DatabaseColumnRequest("dtCreated", new DatabaseTypeRequest(typeof(DateTime)))
+                    { AllowNulls = false, Default = MandatoryScalarFunctions.GetTodaysDate }
+            });
+        var idColumn = definitionTable.DiscoverColumn("id");


...DataExport/DataExtraction/ExecuteFullExtractionToDatabaseMSSqlDestinationReExtractionTest.cs

+        Assert.That(dt.Rows, Has.Count.EqualTo(1));
+
+        //add new entry here
+        var tblInfo = CatalogueRepository.GetAllObjects<TableInfo>().First();


...DataExport/DataExtraction/ExecuteFullExtractionToDatabaseMSSqlDestinationReExtractionTest.cs

+        CohortIdentificationConfiguration cic2 = new CohortIdentificationConfiguration(CatalogueRepository, "Cohort1");
+        cic2.CreateRootContainerIfNotExists();
+        var agg12 = new AggregateConfiguration(CatalogueRepository, catalogue, "agg1");
+        var conf2 = new AggregateConfiguration(CatalogueRepository, catalogue, "UnitTestShortcutAggregate");


...DataExport/DataExtraction/ExecuteFullExtractionToDatabaseMSSqlDestinationReExtractionTest.cs

+        var newCohortCmd2 = new ExecuteCommandCreateNewCohortByExecutingACohortIdentificationConfiguration(
+            new ThrowImmediatelyActivator(RepositoryLocator),
+            cic2,
+            newExternal,
+            "MyCohort2",
+            project,
+            cohortPipeline
+            );


Rdmp.Core/DataLoad/Engine/Pipeline/Destinations/DataTableUploadDestination.cs

* bump to net8 * add sqlcli * Bump Autoupdater.NET.Official from 1.8.4 to 1.8.5 Bumps [Autoupdater.NET.Official](https://github.com/ravibpatel/AutoUpdater.NET) from 1.8.4 to 1.8.5. - [Release notes](https://github.com/ravibpatel/AutoUpdater.NET/releases) - [Commits](ravibpatel/AutoUpdater.NET@v1.8.4...v1.8.5) --- updated-dependencies: - dependency-name: Autoupdater.NET.Official dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Delete .nuget/packages.config (#1758) * Delete .nuget/packages.config Remove hopefully obsolete package list confusing dependency tracking * Remove .nuget folder --------- Co-authored-by: James A Sutherland <> * Bump svenstaro/upload-release-action from 2.7.0 to 2.9.0 (#1759) * Bump shogo82148/actions-setup-perl from 1.28.0 to 1.29.0 (#1751) Bumps [shogo82148/actions-setup-perl](https://github.com/shogo82148/actions-setup-perl) from 1.28.0 to 1.29.0. - [Release notes](https://github.com/shogo82148/actions-setup-perl/releases) - [Commits](shogo82148/actions-setup-perl@v1.28.0...v1.29.0) --- updated-dependencies: - dependency-name: shogo82148/actions-setup-perl dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: James Friel <[email protected]> * Bump CsvHelper from 30.0.3 to 30.1.0 (#1737) Bumps [CsvHelper](https://github.com/JoshClose/CsvHelper) from 30.0.3 to 30.1.0. - [Commits](JoshClose/CsvHelper@30.0.3...30.1.0) --- updated-dependencies: - dependency-name: CsvHelper dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: James Friel <[email protected]> * Bump SSH.NET from 2023.0.1 to 2024.0.0 Bumps [SSH.NET](https://github.com/sshnet/SSH.NET) from 2023.0.1 to 2024.0.0. - [Release notes](https://github.com/sshnet/SSH.NET/releases) - [Commits](sshnet/SSH.NET@2023.0.1...2024.0.0) --- updated-dependencies: - dependency-name: SSH.NET dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> * Bump SixLabors.ImageSharp.Drawing from 2.1.0 to 2.1.1 Bumps [SixLabors.ImageSharp.Drawing](https://github.com/SixLabors/ImageSharp.Drawing) from 2.1.0 to 2.1.1. - [Release notes](https://github.com/SixLabors/ImageSharp.Drawing/releases) - [Commits](SixLabors/ImageSharp.Drawing@v2.1.0...v2.1.1) --- updated-dependencies: - dependency-name: SixLabors.ImageSharp.Drawing dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Bump MongoDB.Driver from 2.23.1 to 2.24.0 Bumps [MongoDB.Driver](https://github.com/mongodb/mongo-csharp-driver) from 2.23.1 to 2.24.0. - [Release notes](https://github.com/mongodb/mongo-csharp-driver/releases) - [Commits](mongodb/mongo-csharp-driver@v2.23.1...v2.24.0) --- updated-dependencies: - dependency-name: MongoDB.Driver dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Bump NUnit from 4.0.1 to 4.1.0 Bumps [NUnit](https://github.com/nunit/nunit) from 4.0.1 to 4.1.0. - [Release notes](https://github.com/nunit/nunit/releases) - [Changelog](https://github.com/nunit/nunit/blob/master/CHANGES.md) - [Commits](nunit/nunit@v4.0.1...4.1.0) --- updated-dependencies: - dependency-name: NUnit dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Bump FluentFTP from 49.0.1 to 49.0.2 Bumps [FluentFTP](https://github.com/robinrodricks/FluentFTP) from 49.0.1 to 49.0.2. - [Release notes](https://github.com/robinrodricks/FluentFTP/releases) - [Changelog](https://github.com/robinrodricks/FluentFTP/blob/master/RELEASES.md) - [Commits](https://github.com/robinrodricks/FluentFTP/commits) --- updated-dependencies: - dependency-name: FluentFTP dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Task/rdmp-140 Update Dependabot yml file (#1756) * centrally managed dependacies * add directory.project.props file * fix typo * fix build * fix up test * Whitespace cleanup * Avoid searching for and reading all csproj files twice * remove console log * update dependabot actions * remove dependabot lower bounds --------- Co-authored-by: James A Sutherland <> * fix merge * run net8 * Bump YamlDotNet from 15.1.1 to 15.1.2 (#1767) Bumps [YamlDotNet](https://github.com/aaubry/YamlDotNet) from 15.1.1 to 15.1.2. - [Release notes](https://github.com/aaubry/YamlDotNet/releases) - [Commits](aaubry/YamlDotNet@v15.1.1...v15.1.2) --- updated-dependencies: - dependency-name: YamlDotNet dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * update sqlclient * add version * remove obsolete properties from tests * workaround depricated formatter * add text fix * add package listing * Bump coverlet.collector from 6.0.0 to 6.0.1 (#1757) Bumps [coverlet.collector](https://github.com/coverlet-coverage/coverlet) from 6.0.0 to 6.0.1. - [Release notes](https://github.com/coverlet-coverage/coverlet/releases) - [Commits](coverlet-coverage/coverlet@v6.0.0...v6.0.1) --- updated-dependencies: - dependency-name: coverlet.collector dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * RDMP-143 Reorder Excel Attacher Processes (#1752) * update excel attacher processing ordering * Bump CsvHelper from 30.0.1 to 31.0.0 (#1761) * Bump SixLabors.ImageSharp from 3.1.2 to 3.1.3 (#1770) Bumps [SixLabors.ImageSharp](https://github.com/SixLabors/ImageSharp) from 3.1.2 to 3.1.3. - [Release notes](https://github.com/SixLabors/ImageSharp/releases) - [Commits](SixLabors/ImageSharp@v3.1.2...v3.1.3) --- updated-dependencies: - dependency-name: SixLabors.ImageSharp dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump SixLabors.ImageSharp.Drawing from 2.1.1 to 2.1.2 Bumps [SixLabors.ImageSharp.Drawing](https://github.com/SixLabors/ImageSharp.Drawing) from 2.1.1 to 2.1.2. - [Release notes](https://github.com/SixLabors/ImageSharp.Drawing/releases) - [Commits](SixLabors/ImageSharp.Drawing@v2.1.1...v2.1.2) --- updated-dependencies: - dependency-name: SixLabors.ImageSharp.Drawing dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Resetting Dependabot for central versioning * Reinstate Dependabot to reset state * Move ManagePackageVersionsCentrally flag to Directory.Build.props * Fix casing on Directory.Build.props file * Task/tidy up net 8 migration (#1775) * fix packages.md * Bump coverlet.collector from 6.0.1 to 6.0.2 Bumps [coverlet.collector](https://github.com/coverlet-coverage/coverlet) from 6.0.1 to 6.0.2. - [Release notes](https://github.com/coverlet-coverage/coverlet/releases) - [Commits](coverlet-coverage/coverlet@v6.0.1...v6.0.2) --- updated-dependencies: - dependency-name: coverlet.collector dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Bump HIC.BadMedicine from 1.1.2 to 1.2.0 (#1776) * Bump HIC.BadMedicine from 1.1.2 to 1.2.0 Bumps [HIC.BadMedicine](https://github.com/HicServices/BadMedicine) from 1.1.2 to 1.2.0. - [Release notes](https://github.com/HicServices/BadMedicine/releases) - [Changelog](https://github.com/HicServices/BadMedicine/blob/develop/CHANGELOG.md) - [Commits](HicServices/SynthEHR@v1.1.2...v1.2.0) --- updated-dependencies: - dependency-name: HIC.BadMedicine dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * BadMedicine API update --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: James A Sutherland <[email protected]> * bump fansisql (#1778) * Bump Terminal.Gui from 1.15.1 to 1.16.0 (#1781) Bumps [Terminal.Gui](https://github.com/gui-cs/Terminal.Gui) from 1.15.1 to 1.16.0. - [Release notes](https://github.com/gui-cs/Terminal.Gui/releases) - [Commits](gui-cs/Terminal.Gui@v1.15.1...v1.16.0) --- updated-dependencies: - dependency-name: Terminal.Gui dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump NUnit.Analyzers from 4.0.1 to 4.1.0 (#1780) Bumps [NUnit.Analyzers](https://github.com/nunit/nunit.analyzers) from 4.0.1 to 4.1.0. - [Release notes](https://github.com/nunit/nunit.analyzers/releases) - [Changelog](https://github.com/nunit/nunit.analyzers/blob/master/CHANGES.txt) - [Commits](nunit/nunit.analyzers@4.0.1...4.1.0) --- updated-dependencies: - dependency-name: NUnit.Analyzers dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump HIC.BadMedicine from 1.2.0 to 1.2.1 Bumps [HIC.BadMedicine](https://github.com/HicServices/BadMedicine) from 1.2.0 to 1.2.1. - [Release notes](https://github.com/HicServices/BadMedicine/releases) - [Changelog](https://github.com/HicServices/BadMedicine/blob/develop/CHANGELOG.md) - [Commits](HicServices/SynthEHR@v1.2.0...v1.2.1) --- updated-dependencies: - dependency-name: HIC.BadMedicine dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * external database commands saved * Bigfix: Bring Notification Boxes To Front (#1785) * bring notification to front * add top most * Bump NPOI from 2.6.2 to 2.7.0 (#1787) Bumps [NPOI](https://github.com/nissl-lab/npoi) from 2.6.2 to 2.7.0. - [Release notes](https://github.com/nissl-lab/npoi/releases) - [Commits](https://github.com/nissl-lab/npoi/commits) --- updated-dependencies: - dependency-name: NPOI dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix value to string conversion (#1788) * fix value to string conversion * Update ExcelDataFlowSource.cs * Task/RDMP-65 remember last used extraction pipeline (#1768) * extractions remember pipeline * update changelog * reorder testing * fix test * revert test * add order to test * updates from codeql * tidy up code * Update ExecuteExtractionUI.cs * Task/allow extended timeouts (#1773) * add 50min timeout * interim * add optional timeout * add changelog * tidy up from codeql * update * Task/rdmp 145 improve bak experience (#1772) * improve bak file usage * add documentation stub * add docs * tidy up code * bump sixLabors * update to ui * restore files * improve documentation * add using statement * specify server when importing existing cohorts * Task/RDMP-129 Remove wix toolset (#1789) * start of msi upgrade * remove wix * update build steps * tidy up * restore file * update sln * Feature/rdmp 117 multiple loads per catalogue (#1754) * basic db objects * fix typo * working db linkage * working build * add sql update * update sql * update tests * update sql * update create sql * update sql * update sql * update class * fix extra space * attempt to fix tests * remove linkage * fix typo * add test case * attempt image fix * fix up image * add missing file * add changelog * interim * interim * tidy up code * fix build * icons * rename icon file (#1755) * working icon * update create sql * update patch * add patch * working patches * add basic tests * tidy up code * add go * attempt to fix patch * tidy up code * remove true equal true check * tidy up from review * update migration * remove dupe file (#1774) * fix build * interim sql * update patch * fix sql patch * update patch id * update patch id * fix bad merge * tidy up code * tidy up code --------- Co-authored-by: James A Sutherland <[email protected]> * allow for multiple identifiers in an project catalogue (#1791) * fix logic issue * update changelog * only convert core categories * add release date * update changelog * Task/rdmp-156 prep 8.1.5 release (#1793) * add release date * update changelog * attempt to fix nuge tpackaging * Update build.yml * Task/rdmp 156 prep 8.1.5 release (#1795) * disable tag check * add sgen * revert bad fix * nudge deps * fix duplicate deps * drop dep * fix build --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: James A Sutherland <[email protected]> Co-authored-by: Brian <[email protected]> Co-authored-by: James A Sutherland <> Co-authored-by: James A Sutherland <[email protected]> Co-authored-by: rdteviotdale <[email protected]>

…raction

…sk/RDMP-122-update-a-database-extraction

CHANGELOG.md

…nto task/RDMP-122-update-a-database-extraction

Task/fix main merge

…sk/RDMP-122-update-a-database-extraction

JFriel and others added 20 commits February 1, 2024 15:49

add extraction additions

ca44e2c

interim

43e298f

Merge branch 'develop' of https://github.com/HicServices/RDMP into ta…

f153e74

…sk/RDMP-122-update-a-database-extraction

add test

2cfa886

Merge branch 'develop' of https://github.com/HicServices/RDMP into ta…

a7498e5

…sk/RDMP-122-update-a-database-extraction

interim

55323ca

Merge branch 'develop' of https://github.com/HicServices/RDMP into ta…

4aa99a7

…sk/RDMP-122-update-a-database-extraction

working dedupe

a21d400

improved checking

2878d6e

Merge branch 'develop' of https://github.com/HicServices/RDMP into ta…

af27e92

…sk/RDMP-122-update-a-database-extraction

add timestamp option

8f7c865

fix extra looping

32e97f9

add check

53e6a2d

start on tests

5cc6867

Merge branch 'develop' of https://github.com/HicServices/RDMP into ta…

56c745e

…sk/RDMP-122-update-a-database-extraction

Merge branch 'develop' of https://github.com/HicServices/RDMP into ta…

3e696ba

…sk/RDMP-122-update-a-database-extraction

tidy up code

f04860d

update link

ece5183

tidy up

17ffe22

Rename executeFullExtractionToDatabaseMSSql.md to ExecuteFullExtracti…

a9d32f3

…onToDatabaseMSSql.md

github-advanced-security bot found potential problems Mar 14, 2024

View reviewed changes

JFriel added 2 commits March 14, 2024 10:30

fix typo

85dd61c

add docs

6d651ce

github-advanced-security bot found potential problems Mar 14, 2024

View reviewed changes

Rdmp.Core/DataExport/DataExtraction/Pipeline/Sources/ExecuteDatasetExtractionSource.cs Fixed Show fixed Hide fixed

JFriel added 4 commits March 15, 2024 08:46

update

48e90ed

update documentation

7c7f6bb

attempt fix docs

cbf544c

update docs

e16ade8

JFriel marked this pull request as ready for review March 15, 2024 11:25

Merge branch 'develop' into task/RDMP-122-update-a-database-extraction

dc93206

JFriel added 2 commits April 5, 2024 13:03

working clash

70054a0

add test

f975889

github-advanced-security bot found potential problems Apr 5, 2024

View reviewed changes

JFriel added 2 commits April 5, 2024 13:54

fix test

7b33248

improved clash checker

bfd1331

JFriel changed the base branch from develop to release/8.2.0 April 5, 2024 13:09

JFriel added 3 commits April 5, 2024 14:13

tidy up

c93d98d

update test

051d5c8

fix up test

90af235

JFriel requested a review from rdteviotdale April 8, 2024 10:24

JFriel marked this pull request as ready for review April 8, 2024 10:24

JFriel and others added 6 commits April 8, 2024 11:28

Merge branch 'release/8.2.0' into task/RDMP-122-update-a-database-ext…

c68d3a7

…raction

Merge branch 'develop' of https://github.com/HicServices/RDMP into ta…

7c31a54

…sk/RDMP-122-update-a-database-extraction

update from codeql

6cc142e

tidy up code

13e24c1

fix bad merge

e1be2af

JFriel commented May 23, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

JFriel requested review from bpeacock001 and JBaird00183 May 23, 2024 08:13

JFriel and others added 7 commits May 24, 2024 11:53

Merge branch 'release/8.2.0' of https://github.com/HicServices/RDMP i…

5305b79

…nto task/RDMP-122-update-a-database-extraction

fix typo

cac56a5

skip over for now

b1947fd

Merge branch 'main' of https://github.com/HicServices/RDMP into develop

0b4c4db

Merge pull request #1841 from HicServices/task/fix-main-merge

659bb25

Task/fix main merge

Merge branch 'develop' of https://github.com/HicServices/RDMP into ta…

92d26ff

…sk/RDMP-122-update-a-database-extraction

revert change

72eb886

bpeacock001 approved these changes May 28, 2024

View reviewed changes

bpeacock001 merged commit 138ffc1 into release/8.2.0 May 28, 2024
5 checks passed

bpeacock001 deleted the task/RDMP-122-update-a-database-extraction branch May 28, 2024 07:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Task/rdmp 122 update a database extraction #1779

Task/rdmp 122 update a database extraction #1779

JFriel commented Mar 14, 2024 •

edited

Loading

Task/rdmp 122 update a database extraction #1779

Task/rdmp 122 update a database extraction #1779

Conversation

JFriel commented Mar 14, 2024 • edited Loading

JFriel commented Mar 14, 2024 •

edited

Loading