Unable to use Ruby 3.x

[baka-san]

Currently unable to use Kimurai with Ruby 3.x. I'm able to use the same code when downgrading the Ruby version to 2.x. Getting the error wrong number of arguments (given 2, expected 1; required keyword: url) (ArgumentError) when trying to run the crawl/parse methods. I had issues with rbcat as well. Any plans to add Ruby 3.x support? I think this issue is likely related: #58

=== Full error ===

g:test_api/ $ ruby app/spiders/test_spider.rb                                                                                                                  [17:47:53]
I, [2021-12-17 17:48:01 +0900#15507] [M: 280]  INFO -- test_spider: Spider: started: test_spider
D, [2021-12-17 17:48:02 +0900#15507] [M: 280] DEBUG -- test_spider: BrowserBuilder (mechanize): created browser instance
I, [2021-12-17 17:48:02 +0900#15507] [M: 280]  INFO -- test_spider: Browser: started get request to: http://www.google.com
I, [2021-12-17 17:48:03 +0900#15507] [M: 280]  INFO -- test_spider: Browser: finished get request to: http://www.google.com
I, [2021-12-17 17:48:03 +0900#15507] [M: 280]  INFO -- test_spider: Info: visits: requests: 1, responses: 1
I, [2021-12-17 17:48:03 +0900#15507] [M: 280]  INFO -- test_spider: Browser: driver mechanize has been destroyed
F, [2021-12-17 17:48:03 +0900#15507] [M: 280] FATAL -- test_spider: Spider: stopped: {:spider_name=>"test_spider", :status=>:failed, :error=>"#<ArgumentError: wrong number of arguments (given 2, expected 1; required keyword: url)>", :environment=>"development", :start_time=>2021-12-17 17:48:01.404891 +0900, :stop_time=>2021-12-17 17:48:03.8518 +0900, :running_time=>"2s", :visits=>{:requests=>1, :responses=>1}, :items=>{:sent=>0, :processed=>0}, :events=>{:requests_errors=>{}, :drop_items_errors=>{}, :custom=>{}}}
app/spiders/test_spider.rb:16:in `parse': wrong number of arguments (given 2, expected 1; required keyword: url) (ArgumentError)
  from /Users/g/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/kimurai-1.4.0/lib/kimurai/base.rb:204:in `public_send'
  from /Users/g/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/kimurai-1.4.0/lib/kimurai/base.rb:204:in `request_to'
  from /Users/g/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/kimurai-1.4.0/lib/kimurai/base.rb:128:in `block in crawl!'
  from /Users/g/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/kimurai-1.4.0/lib/kimurai/base.rb:124:in `each'
  from /Users/g/.rbenv/versions/3.0.2/lib/ruby/gems/3.0.0/gems/kimurai-1.4.0/lib/kimurai/base.rb:124:in `crawl!'
  from app/spiders/test_spider.rb:91:in `<main>'

[Thornolf]

Hey ! I have the exact same problem. I don't see any commit for almost 2 years on this repo, is this project not maintained anymore ?

[n-studio]

https://github.com/n-studio/kimuraframework/tree/dev/ruby-3 <- this branch might help. I can't guarantee it's production ready though.

You also need to add: gem "rbcat", github: "deanopound/rbcat", branch: "fix/support-ruby-3.0.0" to your project.

[tkhobbes]

Similar problem here, the error message around "given 2, expected 1" argument is thrown for colorizer.rb. Have updated my Gemfile as follows:

gem "kimurai", github: "n-studio/kimuraframework", branch: "dev/ruby-3"
gem "rbcat", github: "deanopound/rbcat", branch: "fix/support-ruby-3.0.0"

Here's the full error stack:

/Users/thomas/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/rbcat-0.2.1/lib/rbcat/colorizer.rb:9:in `colorize': wrong number of arguments (given 2, expected 1) (ArgumentError)
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/rbcat-0.2.1/lib/rbcat.rb:10:in `colorize'
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/kimurai-1.4.0/lib/kimurai/base.rb:18:in `block in <class:Base>'
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/3.1.0/logger.rb:586:in `format_message'
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/3.1.0/logger.rb:476:in `add'
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/3.1.0/logger.rb:529:in `info'
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/kimurai-1.4.0/lib/kimurai/base.rb:118:in `crawl!'
        from scraper.rb:34:in `<main>'

So above solution doesn't seem to work?

[n-studio]

@tkhobbes Did you bundle update correctly? The path of the file should be /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/bundler/gems/kimuraframework-631a156a8028/lib/kimurai/base.rb

[tkhobbes]

Weird. Yes it's updated, if I do bundle info kimurai I get

  * kimurai (1.4.1 6af9500)
        Summary: Modern web scraping framework written in Ruby and based on Capybara/Nokogiri
        Homepage: https://github.com/vifreefly/kimuraframework
        Path: /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/bundler/gems/kimuraframework-6af9500a816c

But when I run my program with "ruby myprog.rb" I still get this message:

/Users/thomas/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/rbcat-0.2.1/lib/rbcat/colorizer.rb:9:in `colorize': wrong number of arguments (given 2, expected 1) (ArgumentError)
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/rbcat-0.2.1/lib/rbcat.rb:10:in `colorize'
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/kimurai-1.4.0/lib/kimurai/base.rb:18:in `block in <class:Base>'
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/3.1.0/logger.rb:586:in `format_message'
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/3.1.0/logger.rb:476:in `add'
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/3.1.0/logger.rb:529:in `info'
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/kimurai-1.4.0/lib/kimurai/base.rb:118:in `crawl!'

I simply have a "require 'kimurai'" line in my program... so I am not 100% sure why it's using the OLD gem (1.4.0) and not the NEW one. Seems like some sort of other problem going on.

[n-studio]

@tkhobbes Running ruby will run your program with the default gems, to run with your Gemfile, you have to run bundle exec ruby myprog.rb

[tkhobbes]

Thanks. Rookie mistake I guess. However the error is still there, this time with selenium-webdriver:

/Users/thomas/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/selenium-webdriver-4.1.0/lib/selenium/webdriver/chrome/options.rb:73:in `initialize': wrong number of arguments (given 1, expected 0) (ArgumentError)
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/bundler/gems/kimuraframework-6af9500a816c/lib/kimurai/browser_builder/selenium_chrome_builder.rb:33:in `new'
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/bundler/gems/kimuraframework-6af9500a816c/lib/kimurai/browser_builder/selenium_chrome_builder.rb:33:in `block in build'
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/gems/capybara-3.36.0/lib/capybara/session.rb:104:in `driver'
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/bundler/gems/kimuraframework-6af9500a816c/lib/kimurai/capybara_ext/session.rb:51:in `ensure in visit'
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/bundler/gems/kimuraframework-6af9500a816c/lib/kimurai/capybara_ext/session.rb:52:in `visit'
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/bundler/gems/kimuraframework-6af9500a816c/lib/kimurai/base.rb:207:in `request_to'
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/bundler/gems/kimuraframework-6af9500a816c/lib/kimurai/base.rb:128:in `block in crawl!'
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/bundler/gems/kimuraframework-6af9500a816c/lib/kimurai/base.rb:124:in `each'
        from /Users/thomas/.rbenv/versions/3.1.0/lib/ruby/gems/3.1.0/bundler/gems/kimuraframework-6af9500a816c/lib/kimurai/base.rb:124:in `crawl!'

[n-studio]

@tkhobbes Good catch, I fixed it: n-studio@545ced3

[tkhobbes]

Perfect - works like charm now :)

[baka-san]

@n-studio

Thanks for posting the branch. I tried it out, and it looks like the crawl method works well, but I think you have an issue with the self.parse! method. You can see my branch for the one-line fix baka-san@2ea9c88