Request for vrf support and configurable out interface

[bvlarsen]

Hi,

Just started to use Jool_siit, great SW.

I wish there was vrf support, so Jool_siit would read out device from vrf instead of global routing table.
And it would also be nice if one could specify out interface in eamt, to manually control outbound interface.

Maybe no specification of interface means same behavior as today, then it will be backward compatible. And if vrf or specific interface is specified it will be used.

[ydahhrk]

Note: VRF noob on the keyboard.

I wish there was vrf support, so Jool_siit would read out device from vrf instead of global routing table.

I don't know. I'm having trouble finding VRF-related API in the kernel. Do you have a rough idea of how this would work?

Eg. What's the VRF equivalent of ip_route_output()/ip6_route_output()? One of the first things those do is throw away the input interface, so I'm guessing they're dead ends.

[bvlarsen]

Hi, thanks for getting back to me. I´m no expert at VRF´s inner workings in Linux, I have mostly used them with commercial routers. But I will investigate how their routing information could be programmatically extracted. VRF´s do use routing tables in Linux, and routing tables are a limited resource 252 available only. So in larger scale manual specification of out-interface might be more effective.

…

On Thu, Jan 20, 2022 at 7:30 PM Alberto Leiva Popper < ***@***.***> wrote: Note: VRF noob on the keyboard. I wish there was vrf support, so Jool_siit would read out device from vrf instead of global routing table. I don't know. I'm having trouble finding VRF-related API in the kernel. Do you have a rough idea of how this would work? Eg. What's the VRF equivalent of ip_route_output()/ip6_route_output()? One of the first things those do is throw away <https://elixir.bootlin.com/linux/v5.4/source/net/ipv4/route.c#L2452> the input interface <https://elixir.bootlin.com/linux/v5.4/source/net/ipv6/route.c#L2475>, so I'm guessing they're dead ends. — Reply to this email directly, view it on GitHub <#374 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO5J3JXWSXYUFTP26MQMDTDUXBIC7ANCNFSM5MJMW7BQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[hunbalazs]

Hi,

(not a kernel programming expert here)

On linux, vrf's are implemented using a special master interface which are called l3mdev (or something like that). As I know these are based on IP rules (like source based routing) so the problem here is that input interfaces must be set for it to work with jool.

Another issue is that ip_route_output/ip6_route_output are for host sourced packets which by default use main route table to find the destination (there are some hacks [ip vrf exec ...] but only for user space).

...
VRF's do use routing tables in Linux, and routing tables are a limited resource 252 available only. So in larger scale manual specification of out-interface might be more effective.
...

Routing table identifier type is uint32_t so it is MUCH more than 252 (docs are off at some places).
Also, please don't use oif because that will mean you can't use ECMP/UCMP either.

[bvlarsen]

Hi again, Yes. you're right. I also figured out the other day that the kernel supports more than 256 tables. IP rules tell the policy routing database how to handle routing. This can be source routing, but also destination routing, or a combination. Are you familiar with the free range routing (FRR) project? This is an open source router project. They also support vrf´s. They do vrf lookups here https://github.com/FRRouting/frr/blob/master/zebra/zebra_vrf.c from line 320. Maybe this can help? Manually specifying oif is a good option to also have. This impedes using ecmp, but it gives manual and simple control, and it may also increase performance. Doing a lookup pr packet probably leads to context switching in the CPU and eating CPU cycles. Some may want ecmp to use link capacity better, others may want more packets translated. Options give flexibility :)

…

On Tue, Feb 1, 2022 at 8:29 PM Balázs Kalmár ***@***.***> wrote: Hi, (not a kernel programming expert here) On linux, vrf's are implemented using a special master interface which are called l3mdev (or something like that). As I know these are based on IP rules (like source based routing) so the problem here is that input interfaces must be set for it to work with jool. Another issue is that ip_route_output/ip6_route_output are for host sourced packets which by default use main route table to find the destination (there are some hacks [ip vrf exec ...] but only for user space). ... VRF's do use routing tables in Linux, and routing tables are a limited resource 252 available only. So in larger scale manual specification of out-interface might be more effective. ... Routing table identifier type is uint32_t so it is MUCH more than 252 (docs are off at some places). Also, please don't user oif because that will mean you can't use ECMP/UCMP either. — Reply to this email directly, view it on GitHub <#374 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO5J3JUDTDGP5K4TVUUEB2DUZAYCTANCNFSM5MJMW7BQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[hunbalazs]

Hi,

I wanted to use linux's vrf implementation with jool because of FRR but it also supports namespaces as a vrf backend so I went that way instead.

FRR's zebra is a userspace implementation, it is totally different from a kernelspace implementation. Userspace can softfail if a kernel module is not available but a kernel module can only hard depend on kernel implementations (be it a module or built-in, correct me if I'm wrong here).

I'm not sure jool should do routing on its own, the user is perfectly capable of controlling it from standard kernel interfaces.
Per packet lookup must be done anyway, that's why the kernel has route caches, also context switches does not happen inside the kernel.

In my opinion the only thing needs to be done here is emulating a forwarded package because currently jool uses an API which creates host outbound packages.

[bvlarsen]

Hi again, I wanted to close this mail thread thanking you for looking into this. FRR works with VRF´s, at least the version I have running on Ubuntu 20.04. I see the same VRF´s in FRR as I find and can work with in the OS it self. The FRR doc also say VRF´s are used, not namespaces. The entire FRR SW package can be run i a namespace though. Also I noticed that IP packets passing through jool_siit looses som meta info, like in interface. This results in limitations in what routing policies can be used, like matching in interface. I.e: ip rule add iif ens33 .... table n, is not possible to use. Thank you for trying. I have worked more with jool_siit since the last e-mail and want to say it´s very good sw. But controlling it from Python made it evident that it needs an API. Maybe a GRPC API with a hart beat, so that jool_siit drops all configurations it the controlling SW fails. GRPC with protobuf makes it easy to interface with many languages. Best regards Bjørn Vegard

…

On Wed, Feb 2, 2022 at 9:29 AM Balázs Kalmár ***@***.***> wrote: Hi, I wanted to use linux's vrf implementation with jool because of FRR but it also supports namespaces as a vrf backend so I went that way instead. FRR's zebra is a userspace implementation, it is totally different from a kernelspace implementation. Userspace can softfail if a kernel module is not available but a kernel module can only hard depend on kernel implementations (be it a module or built-in, correct me if I'm wrong here). I'm not sure jool should do routing on its own, the user is perfectly capable of controlling it from standard kernel interfaces. Per packet lookup must be done anyway, that's why the kernel has route caches, also context switches does not happen inside the kernel. In my opinion the only thing needs to be done here is emulating a forwarded package because currently jool uses an API which creates host outbound packages. — Reply to this email directly, view it on GitHub <#374 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO5J3JXFBVZQFLUN2QLTWODUZDTM7ANCNFSM5MJMW7BQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[hunbalazs]

Hi,

FRR 7.1 does not have namespace as a VRF backend feature, you have to use FRR repo for that (https://deb.frrouting.org/). I tried it, and it is working with any jool module.

Every packet is basically recreated by jool, no metadata is persisted (MARK, in interface, etc.)

Since jool uses netlink interface you can modify pyroute2 to talk with jool. Not an easy task but at least pyroute implements basic netlink API, you only have to add jool specific things.

[ydahhrk]

Sorry; I wish I could help, but working up routing would be more of a feature than a fix, and the current status does not allow for new features.

Every packet is basically recreated by jool, no metadata is persisted (MARK, in interface, etc.)

You sure mark doesn't persist? The code seems to copy it: 0 1 2 3

[hunbalazs]

Every packet is basically recreated by jool, no metadata is persisted (MARK, in interface, etc.)

You sure mark doesn't persist? The code seems to copy it: 0 1 2 3

Indeed, sorry for the noise

[bvlarsen]

"Since jool uses netlink interface you can modify pyroute2 to talk with jool. Not an easy task but at least pyroute implements basic netlink API, you only have to add jool specific things." Thanks for the tip :)

…

On Wed, Mar 9, 2022 at 10:52 PM Balázs Kalmár ***@***.***> wrote: Hi, FRR 7.1 does not have namespace as a VRF backend feature, you have to use FRR repo for that (https://deb.frrouting.org/). I tried it, and it is working with any jool module. Every packet is basically recreated by jool, no metadata is persisted (MARK, in interface, etc.) Since jool uses netlink interface you can modify pyroute2 to talk with jool. Not an easy task but at least pyroute implements basic netlink API, you only have to add jool specific things. — Reply to this email directly, view it on GitHub <#374 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO5J3JTBSECEWYLCEP2PFZTU7EMQ3ANCNFSM5MJMW7BQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[cwmos]

If #140 was implemented, then VRFs would be supported directly - you would just need to set the VRF as master device of the Jool device.