Exclude contact cards (and other non-editable blocks) from HIX

[charludo]

Short description

The contact cards introduced in #3169 have a severe negative impact on the HIX score. This PR excludes them from the HIX calculations.

Proposed changes

• before sending text to Textlab, remove all divs with contenteditable="false".
• this currently only affects contacts, but makes it simple to exclude other blocks in the future, should the need arise

Side effects

• html.text_content() was previously only used to check for empty pages. For convenience, I have changed the code to send the result of this operation to textlab instead of the raw HTML - but I am uncertain if there has been a decision against this in the past when you added the code in question @david-venhoff - are my changes OK?

Resolved issues

Fixes: #3268

---

Pull Request Review Guidelines

[david-venhoff]

@david-venhoff - are my changes OK?

I fear that this might change the hix scores of pages again. The last time we did this our service team had quite a nightmare dealing with municipalities that suddenly could not translate their pages anymore because the hix score was too low.
If we do this, we should probably at least do some tests that this change does not decrease the hix score in comparison to right now.

[JoeyStk]

I don't have a lot of context about this piece of code. I hope @david-venhoff knows more about it. For what I can see this piece of code looks good and the logic itself makes sense, but I might not think of all possible side effects :/

[charludo]

@david-venhoff - are my changes OK?

I fear that this might change the hix scores of pages again. The last time we did this our service team had quite a nightmare dealing with municipalities that suddenly could not translate their pages anymore because the hix score was too low. If we do this, we should probably at least do some tests that this change does not decrease the hix score in comparison to right now.

That's a very valid concern.

I'm not sure how else to progress though, tbh. Maybe passing tostring(html) would be the better option, but that is not guaranteed to achieve text == tostrong(fromstring(text)), afaik.

Removing the divs in question without parsing the HTML is an entirely differnet can of worms.

Frankly I don't think we have a choice but to risk slightly changing HIX scores :( (With prior testing still, of course)

[charludo]

I think we need @osmers to chime in on this 😅
In short: the issue is that we must remove contact cards, otherwise the HIX score worsens a lot; but the only way to reliably do so can lead to very slight changes in the HTML content we send to Textlab, and so could result in slightly changed HIX scores compared to right now, including pages not containing any contact cards. I really cannot say if that change in score would be an increase or decrease.

[osmers]

@charludo can I already test this in the test cms? I would then just go through each page for a region and see how much the HIX-score changes, even without inserting contact cards. Though I think it would make sense to also test it with inserted contact cards just to make sure (bcs there municipalities will expect an increase in the HIX-score).
As David said, we definitely need to test this and if the tests show rather bad results, in the sense of much lower HIX-scores, we need to rethink how to do this...

[charludo]

@charludo can I already test this in the test cms? I would then just go through each page for a region and see how much the HIX-score changes, even without inserting contact cards. Though I think it would make sense to also test it with inserted contact cards just to make sure (bcs there municipalities will expect an increase in the HIX-score). As David said, we definitely need to test this and if the tests show rather bad results, in the sense of much lower HIX-scores, we need to rethink how to do this...

Not really, no, since this is not merged yet :/

Contact Cards should definitely improve HIX compared to plain-text contact details though, yes.

[osmers]

Could we merge it but possibly block/exclude it from the next release? Then Salua and me can test it :)

[david-venhoff]

It is also possible to test without merging by creating a branch that ends in -publish-dev-package:

integreat-cms/.circleci/config.yml

Lines 661 to 668 in 102b45f

	- publish-package:
	name: publish-dev-package
	context: pypi-test
	filters:
	branches:
	only:
	- develop
	- /.*-publish-dev-package/

[charludo]

It is also possible to test without merging by creating a branch that ends in -publish-dev-package:

integreat-cms/.circleci/config.yml

Lines 661 to 668 in 102b45f

	- publish-package:
	name: publish-dev-package
	context: pypi-test
	filters:
	branches:
	only:
	- develop
	- /.*-publish-dev-package/

Huh, I had no idea!

The build was successfull, but I cannot dinf where I can actually access it 🤔 Do we have automated deploy for these?

[david-venhoff]

I think it will be deployed on the cms test server, when it updates the next time. I think there is also a way to update the test server manually, so that it runs the pr immediately

[charludo]

I think it will be deployed on the cms test server, when it updates the next time. I think there is also a way to update the test server manually, so that it runs the pr immediately

Ohhhh so this will take this build just because it's the newest? But there's no way to switch between versions on the test server?

[david-venhoff]

Yeah, that is my understanding