BUG | SQL> Uneven duplicated rows in pyfunceble_whois_record

[spirillen]

What is the problem you are experiencing?

AS I was working on a way to merge my pyfunceble_whois_record across my test- servers MariaDB and the whois.csv I stumbles on the following case, where their are about half a million duplicated domain (pyfunceble_whois_record.subject) but not a hole lot of other data.

id	subject	expiration_date	epoch	idna_subject	created_at	modified_at	registrar
206108	zzztube.com	04-nov-2024	1730678400	zzztube.com	2023-10-16 22:56:07.000	2023-10-16 22:56:07.000	Danesco Trading Ltd.
	zzztube.com	04-nov-2021	1635980400	zzztube.com
	zzztube.com	04-nov-2021	1635980400	zzztube.com

This problem makes it very hard to combine|concat two tables

How can we reproduce the problem?

Use PyFunceble with --database-type mariadb

Do you have a screenshot?

No response

What did you expect to happen?

Would expect a clean database where any pyfunceble_whois_record.subject are only listed once, and existing records are updated in full.

Is there a workaround?

A lot of manual labor cleaning up the table.

How did you configure PyFunceble?

Doesn't matter

Which Operating System did you use?

Ubuntu 24.04 LTS, Unix or *BSD like OS

Which Python Version did you use?

No response

Which PyFunceble Version did you use?

4.3.0a9.dev

Additional context

As mentioned other places... it's time to have a merge tool to collectively build a central WHOIS DB, this includes reading in the whois.csv to any DB and visa versa (export)

Touches:

• 2 database type/locations WHOIS and the rest 💭 #395
• [Brainstorm / Please participate!] A thought to speed up checking with json/sql #57
• Brainstorm DB structure, minimize data usage and disk I/O (requires #145) #383
• BUG | Whois date in past marked ACTIVE #403
• BUG | result and results (output file) do not match #393
• FR | mariadb = 'NoneType' #392
• BUG | --config-dir #389
• BUG: UnicodeError: label empty or too long #386
• FEATURE: Sharing WHOIS #350
• BUG: dead domain query #343
• FEATURE: Implement check the availability of website / detection of parked & for sale domains #255

[spirillen]

Now I've got this issue as well, each whois query are generating a new entry AND do not lookup in the database an using existing records...

pyfunceble --database-type mariadb --whois-lookup-only -d 0-0-----------------------------------------------------------0.com

All this, while the other DB server where the subject have an id, there the records only appears once