Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle leading wildcard in directive path #41

Merged
merged 3 commits into from
Jun 11, 2019
Merged

Handle leading wildcard in directive path #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
264c43e
Failing test for '*/' prefix in directive path
Jun 10, 2019
85819c2
Add check for directive paths prefixed by '*/'
Jun 10, 2019
84b7d60
Adding/removing errant blank lines.
Jun 11, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 23 additions & 3 deletions src/agent.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,13 @@ namespace
{
return url.defrag().escape().fullpath();
}

std::string trim_front(const std::string& str, const char chr)
{
auto itr = std::find_if(str.begin(), str.end(),
[chr](const char c) {return c != chr;});
return std::string(itr, str.end());
}
}

namespace Rep
Expand All @@ -25,6 +32,12 @@ namespace Rep
{
return *this;
}
// leading wildcard?
if (query.front() == '*')
{
Url::Url trimmed(trim_front(query, '*'));
directives_.push_back(Directive(escape_url(trimmed), true));
}
directives_.push_back(Directive(escape_url(url), true));
sorted_ = false;
return *this;
Expand All @@ -45,6 +58,12 @@ namespace Rep
{
return *this;
}
// leading wildcard?
if (query.front() == '*')
{
Url::Url trimmed(trim_front(query, '*'));
directives_.push_back(Directive(escape_url(trimmed), false));
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you mean for there to be an else here? As this stands, you're pushing both the trimmed and untrimmed versions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that is the intent. There is an explanation in my PR comment.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

D'oh again!

directives_.push_back(Directive(escape_url(url), false));
}
sorted_ = false;
Expand All @@ -55,9 +74,10 @@ namespace Rep
{
if (!sorted_)
{
std::sort(directives_.begin(), directives_.end(), [](const Directive& a, const Directive& b) {
return b.priority() < a.priority();
});
std::sort(directives_.begin(), directives_.end(),
[](const Directive& a, const Directive& b) {
return b.priority() < a.priority();
});
sorted_ = true;
}
return directives_;
Expand Down
36 changes: 36 additions & 0 deletions test/test-robots.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -319,3 +319,39 @@ TEST(RobotsTest, NeverExternalAllowed)
Rep::Robots robot("", "http://a.com/robots.txt");
EXPECT_FALSE(robot.allowed("http://b.com/", "one"));
}

TEST(RobotsTest, LeadingWildcardAllow)
{
std::string content =
"User-agent: meow\n"
"Disallow: /\n"
"Allow: ****/cats\n"
"Allow: */kangaroos\n";
Rep::Robots robot(content);

EXPECT_FALSE(robot.allowed("/kangaroo/zebra/cat/page.html", "meow"));
EXPECT_TRUE(robot.allowed("/cats.html", "meow"));
EXPECT_TRUE(robot.allowed("/cats/page.html", "meow"));
EXPECT_TRUE(robot.allowed("/get/more/cats/page.html", "meow"));
EXPECT_TRUE(robot.allowed("/kangaroos/page.html", "meow"));
EXPECT_TRUE(robot.allowed("/heaps/of/kangaroos/page.html", "meow"));
EXPECT_TRUE(robot.allowed("/kangaroosandkoalas/page.html", "meow"));
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like a case with neither cats nor kangaroos.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first line has neither cats nor kangaroos:

EXPECT_FALSE(robot.allowed("/kangaroo/zebra/cat/page.html", "meow"));

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to add a specific test if you have one in mind?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

D'oh! Never mind.

}

TEST(RobotsTest, LeadingWildcardDisallow)
{
std::string content =
"User-agent: meow\n"
"Allow: /\n"
"Disallow: ****/cats\n"
"Disallow: */kangaroos\n";
Rep::Robots robot(content);

EXPECT_TRUE(robot.allowed("/kangaroo/zebra/cat/page.html", "meow"));
EXPECT_FALSE(robot.allowed("/cats.html", "meow"));
EXPECT_FALSE(robot.allowed("/cats/page.html", "meow"));
EXPECT_FALSE(robot.allowed("/get/more/cats/page.html", "meow"));
EXPECT_FALSE(robot.allowed("/kangaroos/page.html", "meow"));
EXPECT_FALSE(robot.allowed("/heaps/of/kangaroos/page.html", "meow"));
EXPECT_FALSE(robot.allowed("/kangaroosandkoalas/page.html", "meow"));
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above.

}