[SRv6] swss vs test case TestSrv6VpnFpmsyncd failed

[shuaishang]

The vs test case "test_srv6.py::TestSrv6VpnFpmsyncd::test_AddRemoveSrv6SteeringRouteIpv6" (#3123 ) has the probability to fail.

Here is the log:

shuai@shuai:~/sonic-buildimage/src/sonic-swss/tests$ sudo pytest --dvsname=vs test_srv6.py::TestSrv6VpnFpmsyncd::test_AddRemoveSrv6SteeringRouteIpv6
==================================== test session starts ====================================
platform linux -- Python 3.10.12, pytest-8.3.4, pluggy-1.5.0
rootdir: /home/shuai/sonic-buildimage/src/sonic-swss/tests
plugins: flaky-3.8.1
collected 1 item

test_srv6.py

F                                                                        [100%]

========================================= FAILURES ==========================================
__________________ TestSrv6VpnFpmsyncd.test_AddRemoveSrv6SteeringRouteIpv6 __________________

self = <test_srv6.TestSrv6VpnFpmsyncd object at 0x7ff98924b730>
dvs = <conftest.DockerVirtualSwitch object at 0x7ff98924b910>, testlog = None

    def test_AddRemoveSrv6SteeringRouteIpv6(self, dvs, testlog):

        _, output = dvs.runcmd(f"vtysh -c 'show zebra dplane providers'")
        if 'dplane_fpm_sonic' not in output:
            pytest.skip("'dplane_fpm_sonic' required for this test is not available, skipping", allow_module_level=True)

        self.setup_srv6(dvs)

        dvs.runcmd("vtysh -c \"configure terminal\" -c \"interface lo\" -c \"ip address fc00:0:2::1/128\"")

        # configure srv6 usid locator
        dvs.runcmd("vtysh -c \"configure terminal\" -c \"segment-routing\" -c \"srv6\" -c \"locators\" -c \"locator loc1\" -c \"prefix fc00:0:2::/48 block-len 32 node-len 16 func-bits 16\" -c \"behavior usid\"")

        # save exist asic db entries
        tunnel_entries = get_exist_entries(self.adb.db_connection, "ASIC_STATE:SAI_OBJECT_TYPE_TUNNEL")
        nexthop_entries = get_exist_entries(self.adb.db_connection, "ASIC_STATE:SAI_OBJECT_TYPE_NEXT_HOP")
        route_entries = get_exist_entries(self.adb.db_connection, "ASIC_STATE:SAI_OBJECT_TYPE_ROUTE_ENTRY")
        sidlist_entries = get_exist_entries(self.adb.db_connection, "ASIC_STATE:SAI_OBJECT_TYPE_SRV6_SIDLIST")

        # create v6 route with vpn sid
        dvs.runcmd("ip -6 route add 2001:db8:1:1::/64 encap seg6 mode encap segs fc00:0:1:e000:: dev sr0 vrf Vrf13")

        time.sleep(3)

        # check application database
        self.pdb.wait_for_entry("ROUTE_TABLE", "Vrf13:2001:db8:1:1::/64")
        expected_fields = {"segment": "Vrf13:2001:db8:1:1::/64", "seg_src": "fc00:0:2::1"}
        self.pdb.wait_for_field_match("ROUTE_TABLE", "Vrf13:2001:db8:1:1::/64", expected_fields)

        self.pdb.wait_for_entry("SRV6_SID_LIST_TABLE", "Vrf13:2001:db8:1:1::/64")
        expected_fields = {"path": "fc00:0:1:e000::"}
        self.pdb.wait_for_field_match("SRV6_SID_LIST_TABLE", "Vrf13:2001:db8:1:1::/64", expected_fields)

        # verify that the route has been programmed into the ASIC
        self.adb.wait_for_n_keys("ASIC_STATE:SAI_OBJECT_TYPE_TUNNEL", len(tunnel_entries) + 1)
        self.adb.wait_for_n_keys("ASIC_STATE:SAI_OBJECT_TYPE_NEXT_HOP", len(nexthop_entries) + 1)
        self.adb.wait_for_n_keys("ASIC_STATE:SAI_OBJECT_TYPE_SRV6_SIDLIST", len(sidlist_entries) + 1)
        self.adb.wait_for_n_keys("ASIC_STATE:SAI_OBJECT_TYPE_ROUTE_ENTRY", len(route_entries) + 1)

        # get created entries
        route_key = get_created_entry(self.adb.db_connection, "ASIC_STATE:SAI_OBJECT_TYPE_ROUTE_ENTRY", route_entries)
        nexthop_id = get_created_entry(self.adb.db_connection, "ASIC_STATE:SAI_OBJECT_TYPE_NEXT_HOP", nexthop_entries)
        tunnel_id = get_created_entry(self.adb.db_connection, "ASIC_STATE:SAI_OBJECT_TYPE_TUNNEL", tunnel_entries)
        sidlist_id = get_created_entry(self.adb.db_connection, "ASIC_STATE:SAI_OBJECT_TYPE_SRV6_SIDLIST", sidlist_entries)

        # check ASIC SAI_OBJECT_TYPE_SRV6_SIDLIST database
        tbl = swsscommon.Table(self.adb.db_connection, "ASIC_STATE:SAI_OBJECT_TYPE_SRV6_SIDLIST")
        (status, fvs) = tbl.get(sidlist_id)
        assert status == True
        for fv in fvs:
            if fv[0] == "SAI_SRV6_SIDLIST_ATTR_SEGMENT_LIST":
                assert fv[1] == "1:fc00:0:1:e000::"
            elif fv[0] == "SAI_SRV6_SIDLIST_ATTR_TYPE":
                assert fv[1] == "SAI_SRV6_SIDLIST_TYPE_ENCAPS_RED"

        # check ASIC SAI_OBJECT_TYPE_ROUTE_ENTRY database
        tbl = swsscommon.Table(self.adb.db_connection, "ASIC_STATE:SAI_OBJECT_TYPE_ROUTE_ENTRY")
        (status, fvs) = tbl.get(route_key)
        assert status == True
        for fv in fvs:
            if fv[0] == "SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID":
                assert fv[1] == nexthop_id

        # check ASIC SAI_OBJECT_TYPE_NEXT_HOP database
        tbl = swsscommon.Table(self.adb.db_connection, "ASIC_STATE:SAI_OBJECT_TYPE_NEXT_HOP")
        (status, fvs) = tbl.get(nexthop_id)
        assert status == True
        for fv in fvs:
            if fv[0] == "SAI_NEXT_HOP_ATTR_TYPE":
                assert fv[1] == "SAI_NEXT_HOP_TYPE_SRV6_SIDLIST"
            if fv[0] == "SAI_NEXT_HOP_ATTR_SRV6_SIDLIST_ID":
>               assert fv[1] == sidlist_id
E               AssertionError: assert 'oid:0x0' == 'oid:0x3d000000000617'
E
E                 - oid:0x3d000000000617
E                 + oid:0x0

test_srv6.py:1516: AssertionError
----------------------------------- Captured stdout setup -----------------------------------
remove extra link dummy
===================================== warnings summary ======================================
test_srv6.py:10
  /home/shuai/sonic-buildimage/src/sonic-swss/tests/test_srv6.py:10: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
    from distutils.version import LooseVersion

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================== short test summary info ==================================
FAILED test_srv6.py::TestSrv6VpnFpmsyncd::test_AddRemoveSrv6SteeringRouteIpv6 - AssertionError: assert 'oid:0x0' == 'oid:0x3d000000000617'
========================== 1 failed, 1 warning in 63.59s (0:01:03) ==========================

Here is the "swss.rec":

2024-12-02.08:14:18.102806|ROUTE_TABLE:Vrf13:2001:db8:1:1::/64|SET|segment:Vrf13:2001:db8:1:1::/64|seg_src:fc00:0:2::1
2024-12-02.08:14:18.107376|SRV6_SID_LIST_TABLE:Vrf13:2001:db8:1:1::/64|SET|path:fc00:0:1:e000::

Here is the "sairedis.rec":

2024-12-02.08:05:01.754828|c|SAI_OBJECT_TYPE_VIRTUAL_ROUTER:oid:0x3000000000614|NULL=NULL
2024-12-02.08:05:01.991263|c|SAI_OBJECT_TYPE_TUNNEL:oid:0x2a000000000615|SAI_TUNNEL_ATTR_TYPE=SAI_TUNNEL_TYPE_SRV6|SAI_TUNNEL_ATTR_UNDERLAY_INTERFACE=oid:0x60000000005c9|SAI_TUNNEL_ATTR_ENCAP_SRC_IP=fc00:0:2::1
2024-12-02.08:05:01.992157|c|SAI_OBJECT_TYPE_NEXT_HOP:oid:0x4000000000616|SAI_NEXT_HOP_ATTR_TYPE=SAI_NEXT_HOP_TYPE_SRV6_SIDLIST|SAI_NEXT_HOP_ATTR_SRV6_SIDLIST_ID=oid:0x0|SAI_NEXT_HOP_ATTR_TUNNEL_ID=oid:0x2a000000000615
2024-12-02.08:05:01.992793|C|SAI_OBJECT_TYPE_ROUTE_ENTRY||{"dest":"2001:db8:1:1::/64","switch_id":"oid:0x21000000000000","vr":"oid:0x3000000000614"}|SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID=oid:0x4000000000616
2024-12-02.08:05:01.993867|c|SAI_OBJECT_TYPE_SRV6_SIDLIST:oid:0x3d000000000617|SAI_SRV6_SIDLIST_ATTR_SEGMENT_LIST=1:fc00:0:1:e000::|SAI_SRV6_SIDLIST_ATTR_TYPE=SAI_SRV6_SIDLIST_TYPE_ENCAPS_RED

According to "swss.rec", the SID_LIST may be created after ROUTE_TABLE. Then the case failed.

[shuaishang]

@cscarpitta could you please help to check this issue?