Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EW-1019: TSP Testing #5375

Open
wants to merge 47 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
0ce5351
Add empty line.
mkreuzkam-cap Dec 4, 2024
7bbabe7
log token
mkreuzkam-cap Dec 4, 2024
fcd8d6d
Remove pLimit temporarily for testing.
mkreuzkam-cap Dec 9, 2024
49bfdda
Manual batching of data.
mkreuzkam-cap Dec 9, 2024
628d8c3
Merge branch 'main' into EW-1019
mkreuzkam-cap Dec 9, 2024
397dff6
Fix array size.
mkreuzkam-cap Dec 9, 2024
e4548fc
Better logging.
mkreuzkam-cap Dec 9, 2024
634049c
Merge branch 'main' into EW-1019
mkreuzkam-cap Dec 9, 2024
0f7acca
Even more logs!
mkreuzkam-cap Dec 9, 2024
1c342a0
Logging of errors.
mkreuzkam-cap Dec 10, 2024
a9669a5
Use bulk operations for sync. (Very WIP!!!)
mkreuzkam-cap Dec 11, 2024
3633cc0
EW-1019 Fix Linter warnings
SimoneRadtke-Cap Dec 11, 2024
ac863a7
EW-1019 Fix linter warnings
SimoneRadtke-Cap Dec 11, 2024
551f0e6
Use bulk operations for migration.
mkreuzkam-cap Dec 11, 2024
3c805b4
Merge branch 'main' into EW-1019
mkreuzkam-cap Dec 11, 2024
79c6a7c
EW-1019 Add docker compose file to gitignore
SimoneRadtke-Cap Dec 11, 2024
18217e6
fix linter warning
mkreuzkam-cap Dec 11, 2024
643c843
Bulk migration for students.
mkreuzkam-cap Dec 13, 2024
0a38d25
Merge branch 'main' into EW-1019
mkreuzkam-cap Dec 16, 2024
a04d22d
Use configService directly and don't read values in constructor.
mkreuzkam-cap Dec 16, 2024
c16a7bf
Add batching.
mkreuzkam-cap Dec 16, 2024
92e1b49
Merge branch 'main' into EW-1019
mkreuzkam-cap Dec 17, 2024
7dc774f
remove console log, add comments
mkreuzkam-cap Dec 17, 2024
2587c90
Clean up migration and move into service class.
mkreuzkam-cap Dec 17, 2024
470329c
Clean up loggables.
mkreuzkam-cap Dec 17, 2024
4a163ae
EW.1019 Add saveAll to account-idm-service
SimoneRadtke-Cap Dec 17, 2024
29534a2
adjust config var usage.
mkreuzkam-cap Dec 17, 2024
8d19ebf
adjust tests (wip)
mkreuzkam-cap Dec 17, 2024
a34c3a8
Fix findMany test in accoundIdm service.
mkreuzkam-cap Dec 18, 2024
1b0627b
Fix tsp sync strategy test.
mkreuzkam-cap Dec 18, 2024
216dfb0
Merge branch 'main' into EW-1019
mkreuzkam-cap Dec 18, 2024
c734169
Add tests for user service and repo.
mkreuzkam-cap Dec 18, 2024
91d5632
Merge branch 'main' into EW-1019
mkreuzkam-cap Dec 18, 2024
54a3d91
EW-1019 Adjust tsp sync service test
SimoneRadtke-Cap Dec 18, 2024
a6e97fa
Fix tests which broke for some reason.
mkreuzkam-cap Dec 18, 2024
b9edb39
Fix another test which broke for no reason.
mkreuzkam-cap Dec 18, 2024
9f3e9f1
Add tests for accounts.
mkreuzkam-cap Dec 19, 2024
e64b935
Fix an oopsie.
mkreuzkam-cap Dec 19, 2024
c58364b
EW-1019 Add test
SimoneRadtke-Cap Dec 19, 2024
905685d
Finish tsp sync migration service test.
mkreuzkam-cap Dec 19, 2024
b49aedb
Merge branch 'main' into EW-1019
mkreuzkam-cap Dec 19, 2024
d534101
Reduce amount of awaits.
mkreuzkam-cap Dec 19, 2024
27c73ea
remove async/await.
mkreuzkam-cap Dec 19, 2024
45c7d41
Remove line.
mkreuzkam-cap Dec 20, 2024
b002a07
Update apps/server/src/modules/account/domain/services/account-db.ser…
mkreuzkam-cap Dec 20, 2024
c6bd5ec
Add return type.
mkreuzkam-cap Dec 20, 2024
a7b728d
fix linter.
mkreuzkam-cap Dec 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Clean up migration and move into service class.
  • Loading branch information
mkreuzkam-cap committed Dec 17, 2024
commit 2587c90e1744f327cd17d34e70a49eefa283dace
10 changes: 9 additions & 1 deletion apps/server/src/infra/sync/sync.module.ts
Original file line number Diff line number Diff line change
@@ -18,6 +18,7 @@ import { TspSyncService } from './tsp/tsp-sync.service';
import { TspSyncStrategy } from './tsp/tsp-sync.strategy';
import { SyncUc } from './uc/sync.uc';
import { TspFetchService } from './tsp/tsp-fetch.service';
import { TspSyncMigrationService } from './tsp/tsp-sync-migration.service';

@Module({
imports: [
@@ -41,7 +42,14 @@ import { TspFetchService } from './tsp/tsp-fetch.service';
SyncUc,
SyncService,
...((Configuration.get('FEATURE_TSP_SYNC_ENABLED') as boolean)
? [TspSyncStrategy, TspSyncService, TspOauthDataMapper, TspFetchService, TspLegacyMigrationService]
? [
TspSyncStrategy,
TspSyncService,
TspOauthDataMapper,
TspFetchService,
TspLegacyMigrationService,
TspSyncMigrationService,
]
: []),
],
exports: [SyncConsole],
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
import { Loggable, LogMessage } from '@src/core/logger';

export class TspMigrationBatchSummaryLoggable implements Loggable {
constructor(
private readonly batchSize: number,
private readonly usersUpdated: number,
private readonly accountsUpdated: number,
private readonly totalDone: number,
private readonly totalMigrations: number
) {}

public getLogMessage(): LogMessage {
const message: LogMessage = {
message: `Migrated ${this.usersUpdated} users and ${this.accountsUpdated} accounts in batch of size ${this.batchSize} (total done: ${this.totalDone}, total migrations: ${this.totalMigrations})`,
data: {
batchSize: this.batchSize,
usersUpdated: this.usersUpdated,
accountsUpdated: this.accountsUpdated,
totalDone: this.totalDone,
totalMigrations: this.totalMigrations,
},
};

return message;
}
}
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
import { Loggable, LogMessage } from '@src/core/logger';

export class TspTeachersFetchedLoggable implements Loggable {
constructor(private readonly tspTeacherCount: number) {}
constructor(private readonly tspUserMigrationCount: number) {}

public getLogMessage(): LogMessage {
const message: LogMessage = {
message: `Fetched ${this.tspTeacherCount} teachers for migration from TSP`,
message: `Fetched ${this.tspUserMigrationCount} users for migration from TSP`,
data: {
tspTeacherCount: this.tspTeacherCount,
tspUserMigrationCount: this.tspUserMigrationCount,
},
};

Original file line number Diff line number Diff line change
@@ -1,13 +1,19 @@
import { Loggable, LogMessage } from '@src/core/logger';

export class TspUsersMigratedLoggable implements Loggable {
constructor(private readonly migratedUsers: number) {}
constructor(
private readonly totalMigrations: number,
private readonly migratedUsers: number,
private readonly migratedAccounts: number
) {}

public getLogMessage(): LogMessage {
const message: LogMessage = {
message: `Migrated users: ${this.migratedUsers} users migrated`,
message: `Migrated ${this.migratedUsers} users and ${this.migratedAccounts} accounts. Total amount of migrations requested: ${this.totalMigrations}`,
data: {
totalMigrations: this.totalMigrations,
migratedUsers: this.migratedUsers,
migratedAccounts: this.migratedAccounts,
},
};

140 changes: 140 additions & 0 deletions apps/server/src/infra/sync/tsp/tsp-sync-migration.service.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
import { Account, AccountService } from '@modules/account';
import { System } from '@modules/system';
import { UserService } from '@modules/user';
import { Injectable } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { UserDO } from '@shared/domain/domainobject';
import { UserSourceOptions } from '@shared/domain/domainobject/user-source-options.do';
import { Logger } from '@src/core/logger';
import { TspTeachersFetchedLoggable } from './loggable/tsp-teachers-fetched.loggable';
import { TspSyncConfig } from './tsp-sync.config';
import { TspMigrationBatchSummaryLoggable } from './loggable/tsp-migration-batch-summary.loggable';

@Injectable()
export class TspSyncMigrationService {
constructor(
private readonly logger: Logger,
private readonly userService: UserService,
private readonly accountService: AccountService,
private readonly configService: ConfigService<TspSyncConfig, true>
) {
this.logger.setContext(TspSyncMigrationService.name);
}

public async migrateTspUsers(
system: System,
oldToNewMappings: Map<string, string>
): Promise<{
totalAmount: number;
totalUsers: number;
totalAccounts: number;
}> {
const totalIdCount = oldToNewMappings.size;
this.logger.info(new TspTeachersFetchedLoggable(totalIdCount));

const batches = this.getOldIdBatches(oldToNewMappings);

let totalAmount = 0;
let totalUsers = 0;
let totalAccounts = 0;
for await (const oldIdsBatch of batches) {
const { users, accounts, accountsForUserId } = await this.loadUsersAndAccounts(oldIdsBatch);
const updated = this.updateUsersAndAccounts(system.id, oldToNewMappings, users, accountsForUserId);
await this.saveUsersAndAccounts(users, accounts);

totalAmount += oldIdsBatch.length;
totalUsers += updated.usersUpdated;
totalAccounts += updated.accountsUpdated;

this.logger.info(
new TspMigrationBatchSummaryLoggable(
oldIdsBatch.length,
updated.usersUpdated,
updated.accountsUpdated,
totalAmount,
totalIdCount
)
);
}

return {
totalAmount,
totalUsers,
totalAccounts,
};
}

private updateUsersAndAccounts(
systemId: string,
oldToNewMappings: Map<string, string>,
users: UserDO[],
accountsForUserId: Map<string, Account>
): { usersUpdated: number; accountsUpdated: number } {
let usersUpdated = 0;
let accountsUpdated = 0;
users.forEach((user) => {
const oldId = user.sourceOptions?.tspUid;

if (!oldId) {
return;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this case be logged?

}

const newUid = oldToNewMappings.get(oldId);

if (!newUid) {
return;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this case be logged?

}

const newEmailAndUsername = `${newUid}@schul-cloud.org`;

user.email = newEmailAndUsername;
user.externalId = newUid;
user.previousExternalId = oldId;
user.sourceOptions = new UserSourceOptions({ tspUid: newUid });
usersUpdated += 1;

const account = accountsForUserId.get(user.id ?? '');
if (account) {
account.username = newEmailAndUsername;
account.systemId = systemId;
accountsUpdated += 1;
}
});

return { usersUpdated, accountsUpdated };
}

private getOldIdBatches(oldToNewMappings: Map<string, string>): string[][] {
const oldIds = Array.from(oldToNewMappings.keys());
const batchSize = this.configService.get<number>('TSP_SYNC_MIGRATION_LIMIT', 100);

const batchCount = Math.ceil(oldIds.length / batchSize);
const batches: string[][] = [];
for (let i = 0; i < batchCount; i += 1) {
const start = i * batchSize;
const end = Math.min((i + 1) * batchSize, oldIds.length);
batches.push(oldIds.slice(start, end));
}

return batches;
}

private async loadUsersAndAccounts(
tspUids: string[]
): Promise<{ users: UserDO[]; accounts: Account[]; accountsForUserId: Map<string, Account> }> {
const users = await this.userService.findByTspUids(tspUids);

const userIds = users.map((user) => user.id ?? '');
const accounts = await this.accountService.findMultipleByUserId(userIds);

const accountsForUserId = new Map<string, Account>();
accounts.forEach((account) => accountsForUserId.set(account.userId ?? '', account));

return { users, accounts, accountsForUserId };
}

private async saveUsersAndAccounts(users: UserDO[], accounts: Account[]): Promise<void> {
await this.userService.saveAll(users);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could these 2 be in a Promise.all?

await this.accountService.saveAll(accounts);
}
}
Loading
Loading