feat/incremental document sync #674

JGiter · 2024-11-06T13:47:38Z

Documents will be synchronized starting from the block of latest synchronization
Only documents updated since the latest synchronization will be synchronized
Fixed using of DID synchronization job

JGiter · 2024-11-06T13:49:51Z

src/modules/did/did.processor.ts

  }

  @OnQueueWaiting()
-  async OnQueueWaiting(job: Job) {
-    this.logger.debug(`Waiting ${job.name} document ${job.data}`);
+  async OnQueueWaiting(jobId: number) {


Argument to this hook differs from others https://docs.nestjs.com/techniques/queues#event-listeners-1

jrhender

Good start to the PR but I think the issue with the jobs disappearing should be addressed

jrhender · 2024-11-08T00:16:01Z

src/modules/did/did.service.ts

+    ).filter((doc) => {
+      const identity = doc.id.split(':')[3];
+      return changedIdentities.includes(identity);
+    });
+    didsToSynchronize.forEach(async (did) => {
+      this.logger.debug(`Synchronizing DID ${did.id}`);
      await this.pinDocument(did.id);
    });
+
+    await this.latestDidSyncRepository.save({ block: topBlock });


Hey @JGiter, thanks for the PR. Really exciting that this code is being improved.

I think this approach is flawed however. The issue that I see is that jobs are persisted in separate storage (Redis) and then processed asynchronously, but the latestDidSync topBlock is persisted synchronously. So we could have a situation like this:

A bunch of DID update jobs are queued and lastestDidSync.block is updated

The Redis storage fails or is wiped (it's in-memory, so definitely could happen) -> then the system won't update the DIDs in the lost jobs because lastestDidSync.block already ahead of these events.

I'm thinking an alternative approach to get around this problem would be to:

Mark updated DIDs as "stale":

Query all of the "changedIdentities" since the last check (as you've done in this PR)

Mark all of these DIDs as "invalid" (in the Postgres Entities). I think something like this could be used to bulk update entities.

Update latestDidSync.block

Query all of DID entities with invalid status and add a job to the queue (if it doesn't exist already)

When done processing the job, update the DID to be "valid"

In this above, even if the job queue is reset/wiped, the DID entities will still have invalid status, so job for them can be re-added.

Also, we can change the GET Did Doc endpoint to synchronously query the RPC if the cached DID is invalid.

Thanks. This is really good improvement

src/modules/did/did.service.ts

jrhender · 2024-11-08T00:19:49Z

src/modules/did/did.service.ts

@@ -486,7 +511,7 @@ export class DIDService implements OnModuleInit, OnModuleDestroy {

  private async pinDocument(did: string): Promise<void> {
    try {
-      await this.didQueue.add(UPDATE_DID_DOC_JOB_NAME, did);
+      await this.didQueue.add(UPDATE_DID_DOC_JOB_NAME, { did });


@JGiter does this change relate to this comment you made in the PR description?

Fixed using of DID synchronization job

Does the data need to be in an object?

I can not say why, but when passing job as a string, I observed multiple errors error parsing JSON though there are no seemingly JSON parsing in DidProcessor. So I decided to make job as an object, same as in PinProcessor.

JGiter · 2024-11-08T09:00:39Z

src/modules/did/did.service.ts

@@ -486,7 +508,7 @@ export class DIDService implements OnModuleInit, OnModuleDestroy {

  private async pinDocument(did: string): Promise<void> {
    try {
-      await this.didQueue.add(UPDATE_DID_DOC_JOB_NAME, did);
+      await this.didQueue.add(UPDATE_DID_DOC_JOB_NAME, { did }, { jobId: did });


Specifying job id to avoid adding job for the same document

JGiter · 2024-11-08T09:51:55Z

src/modules/did/didSyncStatus.entity.ts

+  @PrimaryGeneratedColumn()
+  id: number;
+
+  @OneToOne(() => DIDDocumentEntity, (document) => document.id)


Addinig relation only on DidSyncStatusEntity side. The DidDocumentEntity will not be changed

jrhender

Some further comments but I think it is continuing to move in a good direction 👍

jrhender · 2024-11-10T16:10:48Z

src/modules/did/didSyncStatus.entity.ts

+  @OneToOne(() => DIDDocumentEntity, (document) => document.id)
+  @JoinColumn()
+  document: string;


It seems wrong to me that this property is string type. I think that typically in a one-to-one join column, the join property has the same type as the joined entity.
See the Profile example at the start of the TypeORM documentation: https://orkhan.gitbook.io/typeorm/docs/one-to-one-relations

Suggested change

@OneToOne(() => DIDDocumentEntity, (document) => document.id)

@JoinColumn()

document: string;

@OneToOne(() => DIDDocumentEntity, (document) => document.id)

@JoinColumn()

document: DIDDocumentEntity;

Yes, I overlooked that

jrhender · 2024-11-10T16:32:32Z

src/modules/did/did.service.ts

+      const updated = await this.didRepository.save(updatedEntity);
+      await this.didSyncStatusRepository.save({
+        document: did,
+        status: DidSyncStatus.Synced,
+      });


I think this should be in a transaction as I think an invariant should be that there is always a didSyncStatus entity if there is a did entity.

jrhender · 2024-11-10T16:50:47Z

src/modules/did/did.service.ts

+    const staleDIDs = (
+      await this.didRepository.find({ select: ['id'] })
+    ).filter((doc) => {
+      const identity = addressOf(doc.id);
+      return changedIdentities.includes(identity);
+    });
+    await this.didSyncStatusRepository
+      .createQueryBuilder()
+      .useTransaction(true)
+      .update(DidSyncStatusEntity)
+      .set({ status: DidSyncStatus.Stale })
+      .where({ document: In(staleDIDs) })
+      .execute();


I seems to me like a better approach would be to rely on the database to do the filtering in the where clause, rather than reading all dids into application memory and then filtering in the application. I think that, if the number of DIDs became large enough, this this.didRepository.find({ select: ['id'] query would probably fail.

I suppose the challenge might be how to use the addresses returned from changedIdentities to find the didSyncStatus entities to update. I think a couple options are:

Store the address of the identity on the DidSyncStatusEntity

Join to DIDDocumentEntity in the query and filter on the id property. Of course the addresses would need to be transformed to did:ethr DID ids.

Note that I think that option 2 might be slightly less complicated if we got rid of the one-to-one join entity and just added a new column to the DidDocumentEntity.
I think we only really need a single isStale boolean column (or could also keep the enum), so the impact on the performance of the database query when reading the DID Document would be negligible, I think.

Thanks for suggestions. Option 2 seems more safe to me, because it excludes case when two status entities connected to the same document. I would also like to avoid changing document entity. It is used everywhere in ssi-hub and I am bit afraid to cause some unexpected changes. And besides property isState is not inherently part of the document.

jrhender · 2024-11-10T16:52:22Z

src/modules/did/did.service.ts

+    await this.didSyncStatusRepository
+      .createQueryBuilder()
+      .useTransaction(true)
+      .update(DidSyncStatusEntity)
+      .set({ status: DidSyncStatus.Stale })
+      .where({ document: In(staleDIDs) })
+      .execute();


Would this query work for existing cached documents (already existing in the cache I mean)? Maybe I missed it, but I don't see a migration that adds a DidSyncStatusEntity for all existing cached DID Documents.

Yes, I missed that

Co-authored-by: John Henderson <[email protected]>

With digits redactor, the logs looked like: debug [DIDService] : 2024-11-21T22:32:25.999Z - Fetched 0 DID events from interval [DIGITS, DIGITS]

JGiter added 3 commits November 6, 2024 15:37

feat: add last did sync entity

65ca95c

feat: incremental document sync

aa7287a

fix: use did update job type

aa0c6fd

JGiter temporarily deployed to dev November 6, 2024 13:47 — with GitHub Actions Inactive

JGiter commented Nov 6, 2024

View reviewed changes

JGiter temporarily deployed to dev November 6, 2024 13:52 — with GitHub Actions Inactive

JGiter temporarily deployed to dev November 6, 2024 14:52 — with GitHub Actions Inactive

JGiter temporarily deployed to dev November 6, 2024 15:20 — with GitHub Actions Inactive

JGiter temporarily deployed to dev November 7, 2024 07:51 — with GitHub Actions Inactive

JGiter force-pushed the feat/incremental-document-sync branch from 4f62bd5 to c18173f Compare November 7, 2024 07:52

JGiter temporarily deployed to dev November 7, 2024 07:52 — with GitHub Actions Inactive

JGiter force-pushed the feat/incremental-document-sync branch from c18173f to 581f885 Compare November 7, 2024 07:53

JGiter temporarily deployed to dev November 7, 2024 07:53 — with GitHub Actions Inactive

JGiter marked this pull request as ready for review November 7, 2024 08:12

JGiter requested a review from royki November 7, 2024 08:14

jrhender requested changes Nov 8, 2024

View reviewed changes

JGiter temporarily deployed to dev November 8, 2024 08:17 — with GitHub Actions Inactive

JGiter temporarily deployed to dev November 8, 2024 08:19 — with GitHub Actions Inactive

JGiter temporarily deployed to dev November 8, 2024 08:26 — with GitHub Actions Inactive

JGiter temporarily deployed to dev November 8, 2024 08:35 — with GitHub Actions Inactive

JGiter temporarily deployed to dev November 8, 2024 08:59 — with GitHub Actions Inactive

JGiter commented Nov 8, 2024

View reviewed changes

JGiter requested a review from jrhender November 8, 2024 10:00

jrhender reviewed Nov 10, 2024

View reviewed changes

JGiter temporarily deployed to dev November 12, 2024 09:30 — with GitHub Actions Inactive

JGiter temporarily deployed to dev November 12, 2024 09:44 — with GitHub Actions Inactive

JGiter temporarily deployed to dev November 12, 2024 10:13 — with GitHub Actions Inactive

JGiter force-pushed the feat/incremental-document-sync branch from 487f07f to c6e1b08 Compare November 12, 2024 10:22

JGiter temporarily deployed to dev November 12, 2024 10:23 — with GitHub Actions Inactive

JGiter and others added 26 commits November 22, 2024 08:11

refactor: mark stale documents

7b61929

fix: dont duplicate update doc job

464c866

fix: did_status_entity.document

27d5e7a

fix: update doc transactionally

91a6041

fix: did sync status migration

2ae001c

test: did service

8cd5a29

refactor: query changed did on db

5348618

feat: limit max synced docs

ee3cb8a

fix: fetch sync status with doc

ed78e5a

feat: catch failed sync

ff5affb

docs: incremental sync

dd41ef1

feat: migrate did sync status

f4d81d4

fix: did sync status column names

b74d350

fix: synced block

6ca5bbd

fix: use logger

6a4e4da

fix: mark if changed dids not empty

38e61a4

fix: commit synced doc tx

377faa1

fix: save status on new did

0140407

test: did service

4de22a0

fix: save new sync status

1443869

refactor: upsert status on did update

70db7bb

feat: create status if not found in sync

379730d

chore: discard tx mark stale

215965e

chore: log failed sync

258ae85

test: did service

1391c02

docs: getChangedIdentities

8bfd1e9

Co-authored-by: John Henderson <[email protected]>

JGiter force-pushed the feat/incremental-document-sync branch from 01ccb64 to 8bfd1e9 Compare November 22, 2024 06:11

JGiter temporarily deployed to dev November 22, 2024 06:12 — with GitHub Actions Inactive

fix(logger.service): disable digits redactor (#675)

49366bc

With digits redactor, the logs looked like: debug [DIDService] : 2024-11-21T22:32:25.999Z - Fetched 0 DID events from interval [DIGITS, DIGITS]

jrhender temporarily deployed to dev December 12, 2024 01:30 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat/incremental document sync #674

feat/incremental document sync #674

JGiter commented Nov 6, 2024

JGiter Nov 6, 2024

jrhender left a comment

jrhender Nov 8, 2024

JGiter Nov 8, 2024

jrhender Nov 8, 2024

JGiter Nov 8, 2024

JGiter Nov 8, 2024

JGiter Nov 8, 2024

jrhender left a comment

jrhender Nov 10, 2024

JGiter Nov 12, 2024

jrhender Nov 10, 2024

jrhender Nov 10, 2024

JGiter Nov 19, 2024

jrhender Nov 10, 2024

JGiter Nov 20, 2024

feat/incremental document sync #674

Are you sure you want to change the base?

feat/incremental document sync #674

Conversation

JGiter commented Nov 6, 2024

Choose a reason for hiding this comment

jrhender left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jrhender left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment