Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mobile page urls are not getting scrapped #46

Open
suyogdilipkale opened this issue Feb 21, 2018 · 1 comment
Open

Mobile page urls are not getting scrapped #46

suyogdilipkale opened this issue Feb 21, 2018 · 1 comment

Comments

@suyogdilipkale
Copy link

I am using this package for my mobile app client.

when user browse the urls on mobile devices, we pass the url to back-end sever and use this package to scrape the metadata, but if those mobile responsive urls are not supported on desktop then it doesn't extract the metadata as the package is executed on server.

sample url: https://www.m.webmd.com/vitamins/ai/ingredientmono-483/peanut-oil
this is mobile responsive url and if you browse on desktop machine it will give an error.

I tried changing the user-agent in headers but it does not work.

var client = new MetaInspector(param.url, {timeout: 10000, headers:‘Mozilla/5.0 (Linux; Android 5.1.1; Nexus 5 Build/LMY48B; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/43.0.2357.65 Mobile Safari/537.36’});

@suyogdilipkale
Copy link
Author

suyogdilipkale commented Feb 22, 2018

I figured out solution by passing user-agent as below:

var MetaInspector = require('node-metainspector');
var client = new MetaInspector("https://www.m.webmd.com/vitamins/ai/ingredientmono-483/peanut-oil", { timeout: 5000, headers:{'User-Agent':'Mozilla/5.0 (Linux; Android 5.1.1; Nexus 5 Build/LMY48B; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/43.0.2357.65 Mobile Safari/537.36'} });

client.on("fetch",
function(){
console.log("title: " + client.title);
console.log("image: " + client.image);
});

client.on("error", function(err){
console.log(err);
});

client.fetch();

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant