[HLD] bmp for monitoring SONiC BGP info #1621

FengPan-Frank · 2024-02-22T14:41:51Z

[HLD] bmp for monitoring SONiC BGP info

Description	Repo	PR link
Yang model definition	sonic-buildimage	sonic-net/sonic-buildimage#18530
Add BMP_STATE_DB dedicated instance	sonic-buildimage	sonic-net/sonic-buildimage#19016
Add BMP_STATE_DB support in swss-common	sonic-swss-common	sonic-net/sonic-swss-common#879
bmpcfgd service implementation	sonic-buildimage	sonic-net/sonic-buildimage#18940
config cli implementation	sonic-utility	sonic-net/sonic-utilities#3286
show cli implementation	sonic-utility	sonic-net/sonic-utilities#3289
openbmpd service implementation	sonic-bmp	sonic-net/sonic-bmp#7
Add sonic-bmp submodule	sonic-buildimage	sonic-net/sonic-buildimage#20780
Build bmp container into sonic-buildimage	sonic-buildimage	sonic-net/sonic-buildimage#18946
Add bmp dynamic feature switch in frr side config.	sonic-buildimage	sonic-net/sonic-buildimage#20895

qiluo-msft · 2024-02-29T01:43:52Z

doc/bmp/bmp.md

+[OpenBMP](#https://www.openbmp.org/) is open source BGP monitor framework, which supports collecting, aggregating, data persisting, as well as dashboard, etc. For this project, we will fork this project with some internal code changes which populates the required data into Redis for our usage.
+
+## 2.3 BGP/FRR update
+Current [BGP](#https://github.com/sonic-net/sonic-frr) container has been forked from [FRR](#https://github.com/FRRouting/frr), which supports BMP protocol from version frr/7.2


Current [BGP] container has been forked from [FRR] -> Current BGP container uses a implementation (sonic-frr) forked from FRR #Closed

zhangyanzhao · 2024-03-12T15:05:18Z

community review recording https://zoom.us/rec/share/t7kdlkY0uSCaf9fVifFaeS2rY8RSlWrQlIcO7CaZsaE0HU1bnp8-orK5IN81SXup.QdSvaujeNaOagbml

venkatmahalingam · 2024-03-12T15:42:08Z

doc/bmp/bmp.md

+This table will capture BGP Neighbor attributes supported on BGP neighbor, including capability it support.
+
+```
+admin@str-7050qx-32s-acs-04:~$ redis-cli -n 0 HGETALL BGP_NEIGHBOR_TABLE:10.0.0.61


https://github.com/sonic-net/SONiC/pull/1621/files#diff-0649e404e5d4496d9d6af8f6b9411cb483e3a6d376a927d8660baf476465d7b3R138
"SONiC to stream events occurring in SONiC switch (e.g BGP flap, process not running, ...) in real time via gNMI subscription model.
The events are defined using versioned YANG schema."

We have the above YANG model for streaming the BGP states today and looks like BGP_NEIGHBOR_TABLE supports extensive state information, hope we'll migrate to use one streaming method i.e BMP or events infra.

praveenraja1 · 2024-03-12T15:52:48Z

doc/bmp/bmp.md

+As above context mentioned, we need to bring up new container as bmp on SONiC, after change the relevant component architecture is as below:
+<img src="images/architecture_diagram.png" alt="Architecture Diagram" width="800">  
+
+- Add new bmp container, which has limited resource control. 


Do we new docker for running the "openBMPd" daemon?

I understand that the plan is to have,

config "feature" command to enable/disable this.

Limit CPU resource usage of the new docker .

Even if we leverage the existing frr container to run openBMPd, there are alternate ways to achieve the above, for instance #1 we could have a "config bmp <enable|disable>" which could internally start/stop the openBMPd process. And for #2 we could use command such as "nice/cpulimit" to set process usage limits.

From what i have seen adding a new container (base docker, with no overhead app) increases RAM usage by 20 to 30 MB, so just want to ensure if we really need a new container here.

Thanks, it's indeed feasible as either way, and some tradeoff just exists as consideration.
The advantage of using dedicated container is that we can avoid packaging bmp feature onto some slim images if DUT resource is small and expensive relatively.

rathnasabapathyv · 2024-03-12T18:08:09Z

doc/bmp/bmp.md

+Create below config items list for enabling and disabling different table.
+
+```
+   127.0.0.1:6379[4]> HGETALL FEATURE|bmp


Since FRR does support bmp enable/disable as a cmd-line arg (/usr/lib/frr/bgpd -A 127.0.0.1 -M bmp), to support per table level enable/disable as shown below, it's better to discuss this with FRR-community and introduce table level enable/disable support via FRR CLI knob, so that zebra/bgpd doesn't have to spend any cycle in sending change notifications for those tables to BMP ; Otherwise zebra/bgpd will always send notifications to BMP for the disabled tables and bmp ignores it later based on this config-db.

Please discuss with FRR community on this, as discussed in the HLD review community meeting (March-12-2024)

Sure, currently FRR just post data to BMP following standard BGP protocol, thus neighbor table will be sent in BGP_MSG_OPEN message (#https://datatracker.ietf.org/doc/html/rfc4271#page-13) and all other tables including rib/ls_state... will be sent in BGP_MSG_UPDATE. A better option is to update in FRR side and controlled by the config/cli together, so that no extra data is transmitted between FRR and BMP. Will follow up with FRR community.

Thanks. Let's discuss with FRR community and enhance FRR capability to achieve this, as part of bringing in this feature to community-sonic.

venkatmahalingam · 2024-03-12T18:36:17Z

doc/bmp/bmp.md

+
+## 1.1 Functional requirements
+
+Now SONiC supports standard BGP session established with BGP neighbors, but there's only limited BGP info readable in database, which makes some cases difficult to debug, we need to dump more BGP data, meanwhile we're also expecting some live debug option which could dump BGP related info on demand and improve the debuggability.


VTYSH show commands provide extensive dumps today for the debuggability, trying to understand the new information that we support part of openBMP.

On-change subscription for BGP data is clearly an use-case, I would like to see more filtering options (e.g RIB-in or our per neighbor) in the HLD so that we can limit the number entries going into the DB.

Are you asking smaller granularity control like per neighbor? instead of per table?.

I'm asking per neighbor level filtering at least and if possible, per table level filtering is good as well.

Sure, we will support table level filtering first, later could append neighbor level filtering if specific requirement asks.

venkatmahalingam · 2024-03-12T18:45:16Z

doc/bmp/bmp.md

+    bmp stats interval 1000 
+    bmp monitor ipv4 unicast pre-policy 
+    bmp monitor ipv6 unicast pre-policy 
+    bmp connect 127.0.0.1 port 5000 min-retry 1000 max-retry 2000


One option is to connect to the local collector and other option is to connect to the remote collector, if we have an use-case not to store into the redis DB but to only stream into the remote collector, are we planning to give such an option?

Correct, that will require dedicated BMP service being deployed and maintained, for now we're not planning of this.

venkatmahalingam · 2024-03-12T18:50:04Z

doc/bmp/bmp.md

+
+## 2.4 Database Design
+
+BMP will continually populate existing redis table APPL_DB, just with different keys defined to cover the data so that functional requirement could be supported.


Please consider pushing this state information into STATE_DB or new DB instance if that makes our life easy.

venkatmahalingam · 2024-03-12T18:53:16Z

doc/bmp/bmp.md

+# 4 Resource usage and Test plan
+
+### Resource usage
+Since BMP is a sidecar daemon for improve the SONiC debuggability and monitor efficiency, we don't expect it take too much CPU usage, and prefer it could be able to work under low priority but without break. Now FRR to openbmp is working under push model, if limit cpu of bmp into 5%, it just operate slow, but without missing update, when we flap the bgp several times, all the flap updates will pushed to bmp finally. Thus we should be able to restrist CPU of BMP to lower priority without


There was an exercise done to optimize the DB memory usage for storing the entries in DASH WG. Since we may store lot of RIB information into the DB, may be it's ideal to use protobuf format rather than text format for DB entries (refer the below link for more information).
https://github.com/sonic-net/DASH/blob/main/documentation/general/data/AppDBMemoryEstimation.xlsx

Yes, will make some comparison for this, using pb should be feasible just the data will not be readable if we use redis-cli directly, seems now most of sonic database are text? I'm not sure @qiluo-msft may you comment?

yes, most database contents are in text format, we may need to check the possibility of memory optimization as we may push lot of RIB information into the DB.

…nto fenpan_bmp

Why I did it Add yang model support for feature sonic-net/SONiC#1621 Microsoft ADO (number only):27332639

forrestchu · 2024-06-28T02:06:01Z

doc/bmp/images/architecture_diagram.png

I have some concerns about this architecture design:
1）Why not BMP send route to external OpenBMP servers, which may keep more detailed BGP history based on Kafka.
2) Can BMP module in FRR write route to Redis directly? It is more straightforward.

1> That's will require us to maintain dedicated OpenBMP service, which is not in current scope, maybe later we can do that, if so it will be configurable simply from bmp perspective as well.
2> I think it will introduce unnecessary complexities into FRR, if like that FRR can persist Rib info directly instead of BMP, right? We choose BMP path now because BMP was natively supported in FRR in ASYNC mode, and not much extra cost added into FRR.

OK, Got the concept behind your design. Thanks~

#### Why I did it sonic-net/SONiC#1621, need dedicated daemon to monitor config_db and manage openbmpd state. #### How I did it Use swss-common lib to monitor config_db change.

…nto fenpan_bmp

FengPan-Frank requested review from qiluo-msft and zhangyanzhao February 22, 2024 14:42

FengPan-Frank force-pushed the fenpan_bmp branch 3 times, most recently from 6427fc8 to 23d402e Compare February 27, 2024 12:34

FengPan-Frank requested a review from lguohan February 27, 2024 12:34

FengPan-Frank force-pushed the fenpan_bmp branch 5 times, most recently from 7edbcda to b81dfcf Compare February 29, 2024 01:26

qiluo-msft reviewed Feb 29, 2024

View reviewed changes

FengPan-Frank force-pushed the fenpan_bmp branch 2 times, most recently from 1673a55 to d075afa Compare February 29, 2024 12:33

FengPan-Frank force-pushed the fenpan_bmp branch 5 times, most recently from 6f2ae6c to 68234b1 Compare March 12, 2024 14:41

venkatmahalingam reviewed Mar 12, 2024

View reviewed changes

praveenraja1 reviewed Mar 12, 2024

View reviewed changes

rathnasabapathyv reviewed Mar 12, 2024

View reviewed changes

venkatmahalingam reviewed Mar 12, 2024

View reviewed changes

FengPan-Frank force-pushed the fenpan_bmp branch 2 times, most recently from edbe16c to 16f287f Compare March 14, 2024 14:57

[HLD] bmp for monitoring SONiC BGP info

ac631e3

FengPan-Frank force-pushed the fenpan_bmp branch from 16f287f to ac631e3 Compare March 14, 2024 15:02

This was referenced Mar 27, 2024

Add yang models for bmp service sonic-net/sonic-buildimage#18482

Closed

Add support for redis population. sonic-net/sonic-bmp#4

Closed

Add yang models for bmp service. sonic-net/sonic-buildimage#18530

Merged

FengPan-Frank added 3 commits April 4, 2024 09:30

[HLD] bmp for monitoring SONiC BGP info

99fcc30

Merge branch 'sonic-net:master' into fenpan_bmp

d28e315

Merge branch 'fenpan_bmp' of https://github.com/FengPan-Frank/SONiC i…

50cb559

…nto fenpan_bmp

lguohan pushed a commit to sonic-net/sonic-buildimage that referenced this pull request Apr 23, 2024

[yang]: Add yang models for bmp service. (#18530)

b37c26f

Why I did it Add yang model support for feature sonic-net/SONiC#1621 Microsoft ADO (number only):27332639

FengPan-Frank mentioned this pull request May 11, 2024

Add bmpcfgd for monitoring config_db state change. sonic-net/sonic-buildimage#18940

Merged

11 tasks

forrestchu reviewed Jun 28, 2024

View reviewed changes

qiluo-msft assigned FengPan-Frank Jul 9, 2024

FengPan-Frank added 2 commits December 4, 2024 03:20

Merge branch 'fenpan_bmp' of https://github.com/FengPan-Frank/SONiC i…

4b33274

…nto fenpan_bmp

Merge branch 'fenpan_bmp' of https://github.com/FengPan-Frank/SONiC i…

bb04424

…nto fenpan_bmp

qiluo-msft approved these changes Dec 7, 2024

View reviewed changes

qiluo-msft merged commit 2ca5103 into sonic-net:master Dec 7, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HLD] bmp for monitoring SONiC BGP info #1621

[HLD] bmp for monitoring SONiC BGP info #1621

FengPan-Frank commented Feb 22, 2024 •

edited

Loading

qiluo-msft Feb 29, 2024 •

edited

Loading

FengPan-Frank Feb 29, 2024

zhangyanzhao commented Mar 12, 2024

venkatmahalingam Mar 12, 2024

praveenraja1 Mar 12, 2024

FengPan-Frank Apr 15, 2024

rathnasabapathyv Mar 12, 2024

FengPan-Frank Mar 13, 2024

rathnasabapathyv Mar 13, 2024

venkatmahalingam Mar 12, 2024

FengPan-Frank Mar 13, 2024

venkatmahalingam Mar 13, 2024

FengPan-Frank Apr 12, 2024

venkatmahalingam Mar 12, 2024

FengPan-Frank Mar 13, 2024

venkatmahalingam Mar 12, 2024

FengPan-Frank Mar 13, 2024

venkatmahalingam Mar 12, 2024 •

edited

Loading

FengPan-Frank Mar 13, 2024 •

edited

Loading

venkatmahalingam Mar 13, 2024

forrestchu Jun 28, 2024

FengPan-Frank Jun 28, 2024

forrestchu Jul 1, 2024


		## 1.1 Functional requirements

		Now SONiC supports standard BGP session established with BGP neighbors, but there's only limited BGP info readable in database, which makes some cases difficult to debug, we need to dump more BGP data, meanwhile we're also expecting some live debug option which could dump BGP related info on demand and improve the debuggability.


		## 2.4 Database Design

		BMP will continually populate existing redis table APPL_DB, just with different keys defined to cover the data so that functional requirement could be supported.

[HLD] bmp for monitoring SONiC BGP info #1621

[HLD] bmp for monitoring SONiC BGP info #1621

Conversation

FengPan-Frank commented Feb 22, 2024 • edited Loading

qiluo-msft Feb 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhangyanzhao commented Mar 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

venkatmahalingam Mar 12, 2024 • edited Loading

Choose a reason for hiding this comment

FengPan-Frank Mar 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FengPan-Frank commented Feb 22, 2024 •

edited

Loading

qiluo-msft Feb 29, 2024 •

edited

Loading

venkatmahalingam Mar 12, 2024 •

edited

Loading

FengPan-Frank Mar 13, 2024 •

edited

Loading