Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EW-1011 Add migration to TspSyncStrategy #5280

Merged
merged 76 commits into from
Nov 15, 2024
Merged
Show file tree
Hide file tree
Changes from 55 commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
d74dc3a
Add school part to sync.
mkreuzkam-cap Oct 8, 2024
fe7dd8c
Merge branch 'main' into EW-1006
mkreuzkam-cap Oct 8, 2024
f3265dc
use client
mkreuzkam-cap Oct 8, 2024
96ef010
fix dependencies and schema json
alweber-cap Oct 8, 2024
a492886
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Oct 9, 2024
37c6f3a
use client
mkreuzkam-cap Oct 8, 2024
dbff609
fix dependencies and schema json
alweber-cap Oct 8, 2024
bd30949
work on school sync
mkreuzkam-cap Oct 9, 2024
64eb815
Merge branch 'main' into EW-1006
mkreuzkam-cap Oct 9, 2024
805f9ff
Merge branch 'EW-1006' into EW-1011
SimoneRadtke-Cap Oct 9, 2024
49e4087
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Oct 10, 2024
6760484
EW-1011 Working on migration
SimoneRadtke-Cap Oct 11, 2024
44394ef
Merge branch 'main' into EW-1011
mkreuzkam-cap Oct 14, 2024
555f1e0
Add school data to TSP sync.
mkreuzkam-cap Oct 15, 2024
6fb8c8b
Merge branch 'main' into EW-1005
mkreuzkam-cap Oct 16, 2024
4078822
work on tests and loggables.
mkreuzkam-cap Oct 16, 2024
4568562
Merge branch 'main' into EW-1005
mkreuzkam-cap Oct 16, 2024
8a932c5
work on school data sync.
mkreuzkam-cap Oct 16, 2024
90c7fd1
Merge branch 'main' into EW-1005
mkreuzkam-cap Oct 16, 2024
f348d2a
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Oct 17, 2024
4d05f6b
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Oct 17, 2024
eb954a5
Add tests.
mkreuzkam-cap Oct 17, 2024
5868d14
Merge branch 'main' into EW-1005
mkreuzkam-cap Oct 17, 2024
b94dc5c
Merge branch 'main' into EW-1005
mkreuzkam-cap Oct 17, 2024
2d4cd6c
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Oct 18, 2024
fbe85be
Code review.
mkreuzkam-cap Oct 21, 2024
743cea2
Merge branch 'main' into EW-1005
mkreuzkam-cap Oct 21, 2024
6c657e6
remove unnecessary email and align format.
mkreuzkam-cap Oct 22, 2024
c31f962
EW-1011 Working on migration
SimoneRadtke-Cap Oct 22, 2024
2f1500d
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Oct 22, 2024
e678b4c
Merge branch 'EW-1005' into EW-1011
SimoneRadtke-Cap Oct 22, 2024
4bd1aaa
EW-1011 Working on migration and add p-limit
SimoneRadtke-Cap Oct 22, 2024
9ccadcb
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Oct 22, 2024
fb7c25a
EW-1011 Adjust sync service after merge
SimoneRadtke-Cap Oct 22, 2024
69efcf1
EW-1011 Generalize migrate method
SimoneRadtke-Cap Oct 22, 2024
0f9845d
EW-1011 Add feature flag for migration
SimoneRadtke-Cap Oct 23, 2024
8735040
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Oct 23, 2024
f60740c
EW-1011 Adjust implementation of migration limit
SimoneRadtke-Cap Oct 23, 2024
69f65bf
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Oct 23, 2024
ddf6d1d
EW-1011 Add feature flag for tsp migration
SimoneRadtke-Cap Oct 24, 2024
c9c78f6
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Oct 24, 2024
510d6da
EW-1011 Add loggables
SimoneRadtke-Cap Oct 25, 2024
a7de450
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Oct 25, 2024
99dcf7d
EW-1011 Add tests
SimoneRadtke-Cap Oct 25, 2024
6b69ead
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Oct 25, 2024
f61bb5b
EW-1011 Add account and user module to sync module
SimoneRadtke-Cap Oct 28, 2024
abe7ebb
EW-1011 Add test for syc strategy
SimoneRadtke-Cap Oct 28, 2024
5ddba24
EW-1011 Add a try catch
SimoneRadtke-Cap Oct 28, 2024
e53fe60
EW-1011 Add loggable test
SimoneRadtke-Cap Oct 28, 2024
735c486
EW-1011 Try to fix test for sync strategy
SimoneRadtke-Cap Oct 28, 2024
62a167f
EW-1011 Fix old code in sync service
SimoneRadtke-Cap Oct 28, 2024
dda2743
EW-1011 Try to fix test
SimoneRadtke-Cap Oct 28, 2024
4d0dc2e
Tests run.
mkreuzkam-cap Oct 28, 2024
55452cb
EW-1011 Add test
SimoneRadtke-Cap Oct 28, 2024
a977bca
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Oct 28, 2024
797fe91
EW-1011 Clean up and fix test coverage
SimoneRadtke-Cap Oct 29, 2024
d32e640
EW-1011 Working on review comments
SimoneRadtke-Cap Oct 29, 2024
d9281e2
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Oct 29, 2024
e8339c2
EW-1011 Fix code coverage
SimoneRadtke-Cap Oct 30, 2024
827cfcf
EW-1011 Removed unnecessary code from test
SimoneRadtke-Cap Oct 30, 2024
39f712a
EW-1011 Optimize test
SimoneRadtke-Cap Oct 30, 2024
5f0cfae
EW-1011 Cleanup
SimoneRadtke-Cap Oct 30, 2024
3eb271b
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Oct 30, 2024
ff5acd4
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Nov 4, 2024
17e1dc7
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Nov 4, 2024
603435e
EW-1011 Fix typo
SimoneRadtke-Cap Nov 4, 2024
bfcba5f
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Nov 5, 2024
72fc6a8
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Nov 6, 2024
816a45b
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Nov 7, 2024
1bb116c
EW-1011 Fix typo in test
SimoneRadtke-Cap Nov 7, 2024
cb60050
EW-1011 Adjust configmaps for tsp
SimoneRadtke-Cap Nov 7, 2024
ce04bea
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Nov 8, 2024
a5d0485
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Nov 11, 2024
09a5c1c
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Nov 12, 2024
186a2e2
Merge branch 'main' into EW-1011
SimoneRadtke-Cap Nov 13, 2024
2836210
Merge branch 'main' into EW-1011
mkreuzkam-cap Nov 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion apps/server/src/infra/sync/sync.module.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,11 @@ import { Configuration } from '@hpi-schul-cloud/commons/lib';
import { ConsoleWriterModule } from '@infra/console';
import { RabbitMQWrapperModule } from '@infra/rabbitmq';
import { TspClientModule } from '@infra/tsp-client/tsp-client.module';
import { AccountModule } from '@modules/account';
import { LegacySchoolModule } from '@modules/legacy-school';
import { SchoolModule } from '@modules/school';
import { SystemModule } from '@modules/system';
import { UserModule } from '@modules/user';
import { Module } from '@nestjs/common';
import { LoggerModule } from '@src/core/logger';
import { ProvisioningModule } from '@src/modules/provisioning';
Expand All @@ -20,7 +22,16 @@ import { SyncUc } from './uc/sync.uc';
LoggerModule,
ConsoleWriterModule,
...((Configuration.get('FEATURE_TSP_SYNC_ENABLED') as boolean)
? [TspClientModule, SystemModule, SchoolModule, LegacySchoolModule, RabbitMQWrapperModule, ProvisioningModule]
? [
TspClientModule,
SystemModule,
SchoolModule,
LegacySchoolModule,
RabbitMQWrapperModule,
ProvisioningModule,
UserModule,
AccountModule,
]
: []),
],
providers: [
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
import { TspStudentsFetchedLoggable } from './tsp-students-fetched.loggable';

describe(TspStudentsFetchedLoggable.name, () => {
let loggable: TspStudentsFetchedLoggable;

beforeAll(() => {
loggable = new TspStudentsFetchedLoggable(10);
});

describe('when loggable is initialized', () => {
it('should be defined', () => {
expect(loggable).toBeDefined();
});
});

describe('getLogMessage', () => {
it('should return a log message', () => {
expect(loggable.getLogMessage()).toEqual({
message: `Fetched 10 students for migration from TSP`,
data: {
tspStudentCount: 10,
},
});
});
});
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
import { Loggable, LogMessage } from '@src/core/logger';

export class TspStudentsFetchedLoggable implements Loggable {
constructor(private readonly tspStudentCount: number) {}

getLogMessage(): LogMessage {
psachmann marked this conversation as resolved.
Show resolved Hide resolved
const message: LogMessage = {
message: `Fetched ${this.tspStudentCount} students for migration from TSP`,
data: {
tspStudentCount: this.tspStudentCount,
},
};

return message;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
import { TspStudentsMigratedLoggable } from './tsp-students-migrated.loggable';

describe(TspStudentsMigratedLoggable.name, () => {
let loggable: TspStudentsMigratedLoggable;

beforeAll(() => {
loggable = new TspStudentsMigratedLoggable(10);
});

describe('when loggable is initialized', () => {
it('should be defined', () => {
expect(loggable).toBeDefined();
});
});

describe('getLogMessage', () => {
it('should return a log message', () => {
expect(loggable.getLogMessage()).toEqual({
message: `Migrated students: 10 students migrated`,
data: {
migratedStudents: 10,
},
});
});
});
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
import { Loggable, LogMessage } from '@src/core/logger';

export class TspStudentsMigratedLoggable implements Loggable {
constructor(private readonly migratedStudents: number) {}

getLogMessage(): LogMessage {
psachmann marked this conversation as resolved.
Show resolved Hide resolved
const message: LogMessage = {
message: `Migrated students: ${this.migratedStudents} students migrated`,
data: {
migratedStudents: this.migratedStudents,
},
};

return message;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
import { TspTeachersFetchedLoggable } from './tsp-teachers-fetched.loggable';

describe(TspTeachersFetchedLoggable.name, () => {
let loggable: TspTeachersFetchedLoggable;

beforeAll(() => {
loggable = new TspTeachersFetchedLoggable(10);
});

describe('when loggable is initialized', () => {
it('should be defined', () => {
expect(loggable).toBeDefined();
});
});

describe('getLogMessage', () => {
it('should return a log message', () => {
expect(loggable.getLogMessage()).toEqual({
message: `Fetched 10 teachers for migration from TSP`,
data: {
tspTeacherCount: 10,
},
});
});
});
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
import { Loggable, LogMessage } from '@src/core/logger';

export class TspTeachersFetchedLoggable implements Loggable {
constructor(private readonly tspTeacherCount: number) {}

getLogMessage(): LogMessage {
psachmann marked this conversation as resolved.
Show resolved Hide resolved
const message: LogMessage = {
message: `Fetched ${this.tspTeacherCount} teachers for migration from TSP`,
data: {
tspTeacherCount: this.tspTeacherCount,
},
};

return message;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
import { TspTeachersMigratedLoggable } from './tsp-teachers-migrated.loggable';

describe(TspTeachersMigratedLoggable.name, () => {
let loggable: TspTeachersMigratedLoggable;

beforeAll(() => {
loggable = new TspTeachersMigratedLoggable(10);
});

describe('when loggable is initialized', () => {
it('should be defined', () => {
expect(loggable).toBeDefined();
});
});

describe('getLogMessage', () => {
it('should return a log message', () => {
expect(loggable.getLogMessage()).toEqual({
message: `Migrated teachers: 10 teachers migrated`,
data: {
migratedTeachers: 10,
},
});
});
});
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
import { Loggable, LogMessage } from '@src/core/logger';

export class TspTeachersMigratedLoggable implements Loggable {
constructor(private readonly migratedTeachers: number) {}

getLogMessage(): LogMessage {
psachmann marked this conversation as resolved.
Show resolved Hide resolved
const message: LogMessage = {
message: `Migrated teachers: ${this.migratedTeachers} teachers migrated`,
data: {
migratedTeachers: this.migratedTeachers,
},
};

return message;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
import { TspUsersMigratedLoggable } from './tsp-users-migrated.loggable';

describe(TspUsersMigratedLoggable.name, () => {
let loggable: TspUsersMigratedLoggable;

beforeAll(() => {
loggable = new TspUsersMigratedLoggable(10);
});

describe('when loggable is initialized', () => {
it('should be defined', () => {
expect(loggable).toBeDefined();
});
});

describe('getLogMessage', () => {
it('should return a log message', () => {
expect(loggable.getLogMessage()).toEqual({
message: `Migrated users: 10 users migrated`,
data: {
migratedUsers: 10,
},
});
});
});
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
import { Loggable, LogMessage } from '@src/core/logger';

export class TspUsersMigratedLoggable implements Loggable {
psachmann marked this conversation as resolved.
Show resolved Hide resolved
constructor(private readonly migratedUsers: number) {}

getLogMessage(): LogMessage {
psachmann marked this conversation as resolved.
Show resolved Hide resolved
const message: LogMessage = {
message: `Migrated users: ${this.migratedUsers} users migrated`,
data: {
migratedUsers: this.migratedUsers,
},
};

return message;
}
}
2 changes: 2 additions & 0 deletions apps/server/src/infra/sync/tsp/tsp-sync.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,6 @@ export interface TspSyncConfig {
TSP_SYNC_SCHOOL_DAYS_TO_FETCH: number;
TSP_SYNC_DATA_LIMIT: number;
TSP_SYNC_DATA_DAYS_TO_FETCH: number;
TSP_SYNC_MIGRATION_LIMIT: number;
FEATURE_TSP_MIGRATION_ENABLED: boolean;
}
Loading
Loading