Architecture documentation

[benoit74]

Changes:

• create a high level documentation of the scraper

I've left multiple **TODO** where I need your help if possible (especially @mgautierfr) to resolve the situation.

The exercise was (as expected by all of us) very interesting to discover areas for improvements:

• probably unused Python libraries
• potentially useless transformations (JSON, JSONP and static JS url rewriting, all these are done dynamically by wombat as far as I've understood)

I'm waiting for your feedback on these last points and will add more test cases to the test website around these topics in the mean time.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.98%. Comparing base (437ed7a) to head (2f9b95c).
Report is 1 commits behind head on warc2zim2.

❗ Current head 2f9b95c differs from pull request most recent head 3de8ab8. Consider uploading reports for the commit 3de8ab8 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           warc2zim2     #226   +/-   ##
==========================================
  Coverage      85.98%   85.98%           
==========================================
  Files             13       13           
  Lines           1049     1049           
  Branches         195      195           
==========================================
  Hits             902      902           
  Misses           116      116           
  Partials          31       31           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mgautierfr]

Seems mostly good.
See my comments.

[benoit74]

Great, thank you @mgautierfr !

[benoit74]

I've fixed the documentation. Thank you @mgautierfr for these valuables feedbacks, please confirm it is now OK for you.

@kelson42 @rgaudin do you consider this is a sufficient first step towards more documentation, or did you expected something more detailed? I tried to find the subtle limit between too many details which would get out of sync with the codebase very quickly and too few which would prevent a good understanding of warc2zim behavior, but I might have failed doing so ^^

[mgautierfr]

It's ok for me.

[rgaudin]

Thank you ; I think it's a good start indeed ; please check my inline suggestions

[benoit74]

@kelson42 do you wanna review it as well or may I merge like it is, and we will always be able to open an issue to fix what needs to be later on?