K8S client issues

[alacuku]

Motivation

Falco has a built-in functionality called Kubernetes Metadata Enrichment. It provides k8s metadata, fetched from the k8s api-server, used by Falco to enrich the system-call events. Furthermore, these metadata are available to users as events fields to be used in the conditions and 'outputs' of Falco rules.

Current implementation

The built-in k8s client is written in C++ and lives in the libs repo. It has a number of issues discussed in the next sections.

Poor performances

The metadata are fetched asynchronously from the api-server by opening a connection to the watch endpoint. Periodically, it checks for new events and processes them. The processing is done in the main loop, bloking the processing of syscall stream coming from the driver/probe. In large environments this causes syscall events to be dropped. For more info, please see: #2129. The following scheme shows how the client handles events coming from the api-server (some calls have been ommitted for the sake of clarity):

Stability issues

The network connections to the api-server are far from being stable. It does not check for temporary network failures causing Falco to crash and restart. When deploying Falco in large clusters, all the Falco instances( one per node) connects to the api-server, and it could happen that some of them are throtled and the connection is closed. In such cases Falco pods will crash and restart. For more info see: #1909

Scalability issues

A watch is opened toward the api-serverfor each resource. Currently, we collect metadata for the following resources:

• pods;
• deployments;
• replictioncontrollers;
• replicasets;
• daemonsets;
• namespaces;
• services;
• nodes.

In a 5K nodes cluster the Falco instances will open 40000 watches. This approach does not scale well and contributes to control plane outages in large clusters.

Only Pods objects are efficiently retrieved by using the NodeSelector. For the other resource types, there is no way to filter them out, but we cache them locally leading to a significant waste of resources, especially memory.

Known issues:

• Falco runtime error in k8s_replicationcontroller_handler_state for large k8s clusters (400+ nodes) #1909
• Kubernetes feature consumes high CPU and creates significant load on API server (consider optimizing and caching for high scale) #2346
• Lots of "syscall event drop" since 0.32.1 #2129
• Falco using tokens past their lifetime in Kubernetes 1.21 #1781
• metadata_download.max_mb config not working #2463
• IPv6 Kubernetes API server address parsed as invalid URI #2975

Native libs (client-go) based implementation

The idea is to introduce a centralized component that retrieves the K8s metadata required by Falco instances. The new component should be based on modern libraries such as the one provided by the k8s project.

Motivation

See the section about current implementation.

Goals

• Fetch metadata from the k8s api-server using a centralized component;
• Keep the latency of the updates as short as possible;
• Send only the required metadata to Falco instances ready to be used without the need for further processing on Falco's side;

Non-Goals

• Sharding the K8s state, the instance of the component will be responsible for the entire cluster;
• Adaptive metadata fetching based on requests by the Falco instances. We could think about it in the next iterations.

Proposal

Provide a new component to be deployed in the K8s cluster responsible for providing the metadata required by each Falco instance. Metadata to be sent to Falco:

• Pod's metadata for a given node;
• Namespace metadata only for the ones where local pods reside;
• Workloads metadata: Deployment, replicasets, daemonsets, replicationcontrollers;
• Services metadata only for the ones serving the local pods;

Each Falco instance will subscribe to the centralized component providing the node name where they reside and will receive metadata through a stream of updates. The communication between the Falco instances and the centralized component uses GRPC over HTTP2.

Design Details

The Meta Collector will be implemented using Go relying on controller-runtime to efficiently retrieve and cache K8s resources. An operator/controller will be implemented for each resource needed by Falco. Each operator will connect to the api-server and fetch the initial state for the watched resource. It will create a state in memory and send updates to the Falco instances that need the data. Since we are limited to a subset of the fields of the watched resources update events will be sent only when those fields change. Updates should be sent to Falco as soon as possible. Each operator will:

• Check if a Modified event received from the api-server produces an update to be sent to the Falco instances;
• Generate the event to be sent;
• Send the event to the message broker which will send it to the relevant Falco instances.

After the operators complete the initialization a GRPC server (message broker) will start and wait for Falco subscriptions. The message broker will compute an initial list of events/updates to send to the Falco instance upon a new subscription. Remember that the state is cached, so subscriptions will not generate requests to the api-server. Then the updates will be sent to the Falco instances as they come from the api-server.

NOTE This issue will be updated with more implementation details in the upcoming weeks.

[alacuku]

Anyway, I would be happy to work on the new implementation.

[alacuku]

/assign alacuku

[Andreagit97]

Thank you very much for this! We really need it as soon as possible for this reason I will put a milestone /0.11.0! This is huge so of course, in case of blockers we can move it to the next milestone!

/milestone 0.11.0

BTW thank you very much for taking care of this!

[Dentrax]

What a well-written proposal. So kudos! It would definitely resolve our performance issues. Strong +1 from me.

[tspearconquest]

I will be happy to help any way that I can help on this. Looking forward to seeing this implemented to resolve #2485. Thanks in advance :)

[FedeDP]

/milestone 0.12.0

[leogr]

I think we are too late to have this in Falco 0.35, so
/milestone 0.13.0

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[tspearconquest]

/remove-lifecycle stale

[leogr]

Cross-linking falcosecurity/plugins#378

[Andreagit97]

yep to be honest I would move it to Falco because we will address it in Falco 0.37.0 rather than in libs 0.14.0

[tspearconquest]

The Meta Collector will be implemented using Go relying on controller-runtime to efficiently retrieve and cache K8s resources. An operator/controller will be implemented for each resource needed by Falco. Each operator will connect to the api-server and fetch the initial state for the watched resource. It will create a state in memory and send updates to the Falco instances that need the data. Since we are limited to a subset of the fields of the watched resources update events will be sent only when those fields change.

Hi, as this is to be a few separate components, and it will be talking with the API server, I wanted to ask to consider putting these components into a separate namespace from the main falco pods. Better still would be a separate namespace for each component, but I realize that the single shared namespace is a common pattern, so I hope to simply keep the syscall pods separate from the API server pods as it will greatly simplify the implementation of these (due to security policies in our clusters) for our org once the new components are released.

[Andreagit97]

so I hope to simply keep the syscall pods separate from the API server pods as it will greatly simplify the implementation of these (due to security policies in our clusters) for our org once the new components are released.

yes, this is the idea, we will see if during the integration we will face some issues in doing this. We will keep you posted!

[Issif]

I like the principle too.
I already discussed about that with @leogr @LucaGuerra and @alacuku, but this central component could be extended in the future to enrich the events with fields from the cloud provider metadata or just static elements (like falcosidekick does).
Moreover, it will imply that Falco inits the gRPC connection to the metadata-collector, right now, it's the opposite for the falco-exporter. I guess it could imply in the future the same direction for the output, and falco could be connected to sidekick or else with gRPC.

[mateuszdrab]

Hey @mateuszdrab

I think I'm a bit confused now.

You're right, this may be confusing. I guess the lack of clarity is due to the missing documentation (we are working on it right now). Meanwhile, I can give you an overview, hoping it will help with make more clarity.

The k8s client is being removed and replaced by k8smeta and k8s-metacollector?
Yes.

By default, both will be disabled? (As per #2995)
Yes, they will. Please note that you can see k8smeta and k8s-metacollector as two components of a single feature. The k8smeta does not work without the k8s-metacollector. So let's say these two components together are the new _advanced k8s metadata enrichment_ mechanism for Falco (below it will clear why I put emphasis on the _advanced word.

I use fields like container.image.repository, container.id and k8s.ns.name and don't think I need the new ones;

All container.*, k8s.pod.* and k8s.ns.name fields are populated with data gathered from the container runtime. So they work out-of-the-box in Falco without the need for the old k8s client nor the new advanced k8s metadata enrichment. Thus, these fields will continue to work as before. Nothing changed on this front.

however, I do want to address the issue of metadata being suddenly lost resulting in an increase in alerts like I mentioned above.

Starting with Falco 0.37 (due by Jan 29th), you will be able to enable the new advanced k8s metadata enrichment (k8smeta and k8s-metacollector) as well.
The new fields provided by the new advanced k8s metadata enrichment are in addition and do not replace the existing k8s.pod.* and k8s.ns.name fields. This also implies you will be able to use both k8s.pod.name and k8smeta.pod.name at the same time. In most cases, they will return the same value, but since the data is collected from different places (the container runtime for k8s ones, the K8s API server, for k8smeta), they availability and reliability can be different during the lifecycle of an application. Although this solution might seem redundant, it will offer a lot of flexibility to the users. In addition, the new k8s metadata enrichment will provide a lot of other fields (you can find them here: https://github.com/falcosecurity/plugins/tree/master/plugins/k8smeta#supported-fields). Note that the new advanced k8s metadata enrichment requires a cluster component (i.e. the k8smetacollector) to be deployed in the cluster, so it might not fit all users.

For these reasons, and since the new advanced k8s metadata enrichment might not be required by all users, we consider it an "advanced" mechanism, and it is not enabled by default.

To wrap up:

• If container.*, k8s.pod.*, and k8s.ns.name fields are enough for your use case, you should evaluate and stick with them.
• If container.*, k8s.pod.*, and k8s.ns.name fields are not enough for you (because they do not work well in your use case or you need more metadata) you should evaluate to opt-in to the new k8s metadata enrichment mechanism. This will require a bit of experimentation (since it's a fresh new approach), but you can do it in parallel.

Hoping this helps. Let me know if you have any other doubts!

Thank you for clarifying

It looks like the issue I might be having is not going to be addressed by this change. For me, it is those basic K8s.* labels I mentioned that disappear resulting in excessive alerts. I can resolve it by restarting the Falco pods.

I am surprised the K8s namespace metadata comes from the container runtime and not the API server itself.

I guess I may need to raise a separate issue for that if I determine the exact steps to reproduce it.

[leogr]

It looks like the issue I might be having is not going to be addressed by this change. For me, it is those basic K8s.* labels I mentioned that disappear resulting in excessive alerts. I can resolve it by restarting the Falco pods.

This might be related to drops or other issues related to the container runtime. Or perhaps, some conflict with the old k8s client (have you tried to remove -k/-K?). In any case, I agree yours is likely a different issue.

I guess I may need to raise a separate issue for that if I determine the exact steps to reproduce it.

This would be great 👍

I am surprised the K8s namespace metadata comes from the container runtime and not the API server itself.

Magic 🤯 This was introduced a couple of years ago in Falco and is likely not well-documented. I hope we will do better in documenting the feature coming with the new release (cc @LucaGuerra ).

[Andreagit97]

This should be solved by Falco 0.37.0! Feel free to reopen if this is still an issue

[mateuszdrab]

Hi @Andreagit97

Since the update, it looks like Falco isn't populating the k8s.ns.name and k8s.pod fields - they're all set to null.
All the rule exceptions that used those fields are not working and result in thousands of alerts being generated.

Restarted the daemonset but the situation continues.
Anything else you'd recommend me check or shall I raise a new issue?

[Andreagit97]

ei @mateuszdrab some questions:

• can you provide us with the helm chart command you are using to deploy Falco?
• which Falco version are you using?
• before the update were these fields available?

[mateuszdrab]

Hi @Andreagit97

Values for the deployment are:

controller:
  daemonset:
    updateStrategy:
      rollingUpdate:
        maxUnavailable: 100%
customRules:
  rules-custom.yaml: <redacted>
driver:
  enabled: true
  kind: ebpf
falco:
  grpc:
    enabled: true
  grpc_output:
    enabled: true
  rules_file:
  - /etc/falco/falco_rules.yaml
  - /etc/falco/falco-incubating_rules.yaml
  - /etc/falco/falco-sandbox_rules.yaml
  - /etc/falco/rules.d
  syscall_event_drops:
    actions:
    - log
falcoctl:
  config:
    artifact:
      follow:
        refs:
        - falco-rules:2
        - falco-incubating-rules:2
        - falco-sandbox-rules:2
      install:
        refs:
        - falco-rules:2
        - falco-incubating-rules:2
        - falco-sandbox-rules:2
falcosidekick:
  config:
    syslog:
      host: <redacted>
      port: "514"
    teams:
      webhookurl: <redacted>
  enabled: true
  webui:
    enabled: false
tolerations:
- effect: NoSchedule
  key: node-role.kubernetes.io/master
- effect: NoSchedule
  key: node-role.kubernetes.io/control-plane
- key: environment
  operator: Exists
- key: worker-role
  operator: Exists

Version is latest so 0.37.1, and indeed, before update the fields were available so the rules were working fine which can be seen in the screenshot below:

Alert count/rate:

In the query below, you can see a sample alert that used to have a namespace in the json fields but now no longer has it

[Andreagit97]

Not sure which container runtime you are using but k8s.pod.* and k8s.ns.name should be available through the container runtime https://github.com/falcosecurity/charts/blob/falco-4.2.5/charts/falco/values.yaml#L268 so if you have the container collector enabled there is something strange, you should see these fields...can you provide more info about your setup?

As an alternative, you can enable the k8smeta plugin for collecting the same Kubernetes fields (https://github.com/falcosecurity/charts/blob/68db0665712d657be00be36e5b7933626220d3e7/charts/falco/values.yaml#L289) but it in this case, these fields will be provided under different names, k8smeta.pod.* and k8smeta.ns.name so you need to change your rules. You can find more details here https://falco.org/blog/falco-0-37-0/#fields-supported-by-the-new-k8smeta-plugin

[mateuszdrab]

Hey Andrea
I'm running k3s version v1.27.7+k3s2 with containerd as the runtime.

From our previous conversation I thought k8s.pod.* and k8s.ns.name should be available so I did not enable the new k8smeta plugin.

I can see the socket is mounted in /host/run/containerd/containerd.sock and it's available; however, I can't see this path being referenced anywhere in Falcos config. Is it hardcoded?

Startup log:

Mon Mar 18 15:14:50 2024: Falco version: 0.37.1 (x86_64)
Mon Mar 18 15:14:50 2024: Falco initialized with configuration file: /etc/falco/falco.yaml
Mon Mar 18 15:14:50 2024: System info: Linux version 5.4.0-173-generic (buildd@lcy02-amd64-101) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)) #191-Ubuntu SMP Fri Feb 2 13:55:07 UTC 2024
Mon Mar 18 15:14:50 2024: Loading rules from file /etc/falco/falco_rules.yaml
Mon Mar 18 15:14:50 2024: Loading rules from file /etc/falco/falco-incubating_rules.yaml
Mon Mar 18 15:14:50 2024: Loading rules from file /etc/falco/falco-sandbox_rules.yaml
Mon Mar 18 15:14:50 2024: Loading rules from file /etc/falco/rules.d/rules-custom.yaml
Mon Mar 18 15:14:51 2024: /etc/falco/rules.d/rules-custom.yaml: Ok, with warnings
11 Warnings:
In rules content: (/etc/falco/rules.d/rules-custom.yaml:0:0)
    rule 'Write below etc': (/etc/falco/rules.d/rules-custom.yaml:0:2)
------
- rule: Write below etc
  ^
------
LOAD_DEPRECATED_ITEM (Used deprecated item): 'append' key is deprecated. Add an 'append' entry (e.g. 'condition: append') under 'override' instead.
In rules content: (/etc/falco/rules.d/rules-custom.yaml:0:0)
    rule 'Launch Suspicious Network Tool in Container': (/etc/falco/rules.d/rules-custom.yaml:21:2)
------
- rule: Launch Suspicious Network Tool in Container
  ^
------
LOAD_DEPRECATED_ITEM (Used deprecated item): 'append' key is deprecated. Add an 'append' entry (e.g. 'condition: append') under 'override' instead.
In rules content: (/etc/falco/rules.d/rules-custom.yaml:0:0)
    rule 'Write below root': (/etc/falco/rules.d/rules-custom.yaml:29:2)
------
- rule: Write below root
  ^
------
LOAD_DEPRECATED_ITEM (Used deprecated item): 'append' key is deprecated. Add an 'append' entry (e.g. 'condition: append') under 'override' instead.
In rules content: (/etc/falco/rules.d/rules-custom.yaml:0:0)
    rule 'Launch Package Management Process in Container': (/etc/falco/rules.d/rules-custom.yaml:43:2)
------
- rule: Launch Package Management Process in Container
  ^
------
LOAD_DEPRECATED_ITEM (Used deprecated item): 'append' key is deprecated. Add an 'append' entry (e.g. 'condition: append') under 'override' instead.
In rules content: (/etc/falco/rules.d/rules-custom.yaml:0:0)
    rule 'Redirect STDOUT/STDIN to Network Connection in Container': (/etc/falco/rules.d/rules-custom.yaml:51:2)
------
- rule: Redirect STDOUT/STDIN to Network Connection in Container
  ^
------
LOAD_DEPRECATED_ITEM (Used deprecated item): 'append' key is deprecated. Add an 'append' entry (e.g. 'condition: append') under 'override' instead.
In rules content: (/etc/falco/rules.d/rules-custom.yaml:0:0)
    rule 'Run shell untrusted': (/etc/falco/rules.d/rules-custom.yaml:59:2)
------
- rule: Run shell untrusted
  ^
------
LOAD_DEPRECATED_ITEM (Used deprecated item): 'append' key is deprecated. Add an 'append' entry (e.g. 'condition: append') under 'override' instead.
In rules content: (/etc/falco/rules.d/rules-custom.yaml:0:0)
    rule 'Non sudo setuid': (/etc/falco/rules.d/rules-custom.yaml:67:2)
------
- rule: Non sudo setuid
  ^
------
LOAD_DEPRECATED_ITEM (Used deprecated item): 'append' key is deprecated. Add an 'append' entry (e.g. 'condition: append') under 'override' instead.
In rules content: (/etc/falco/rules.d/rules-custom.yaml:0:0)
    rule 'Launch Remote File Copy Tools in Container': (/etc/falco/rules.d/rules-custom.yaml:85:2)
------
- rule: Launch Remote File Copy Tools in Container
  ^
------
LOAD_DEPRECATED_ITEM (Used deprecated item): 'append' key is deprecated. Add an 'append' entry (e.g. 'condition: append') under 'override' instead.
In rules content: (/etc/falco/rules.d/rules-custom.yaml:0:0)
    rule 'Unexpected UDP Traffic': (/etc/falco/rules.d/rules-custom.yaml:93:2)
------
- rule: Unexpected UDP Traffic
  ^
------
LOAD_DEPRECATED_ITEM (Used deprecated item): 'append' key is deprecated. Add an 'append' entry (e.g. 'condition: append') under 'override' instead.
In rules content: (/etc/falco/rules.d/rules-custom.yaml:0:0)
    rule 'Set Setuid or Setgid bit': (/etc/falco/rules.d/rules-custom.yaml:122:2)
------
- rule: Set Setuid or Setgid bit
  ^
------
LOAD_DEPRECATED_ITEM (Used deprecated item): 'append' key is deprecated. Add an 'append' entry (e.g. 'condition: append') under 'override' instead.
In rules content: (/etc/falco/rules.d/rules-custom.yaml:0:0)
    rule 'Change thread namespace': (/etc/falco/rules.d/rules-custom.yaml:135:2)
------
- rule: Change thread namespace
  ^
------
LOAD_DEPRECATED_ITEM (Used deprecated item): 'append' key is deprecated. Add an 'append' entry (e.g. 'condition: append') under 'override' instead.
Mon Mar 18 15:14:51 2024: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Mon Mar 18 15:14:51 2024: gRPC server threadiness equals to 8
Mon Mar 18 15:14:51 2024: Starting health webserver with threadiness 8, listening on 0.0.0.0:8765
Mon Mar 18 15:14:51 2024: Loaded event sources: syscall
Mon Mar 18 15:14:51 2024: Enabled event sources: syscall
Mon Mar 18 15:14:51 2024: Opening 'syscall' source with BPF probe. BPF probe path: /root/.falco/falco-bpf.o
Mon Mar 18 15:14:51 2024: Starting gRPC server at unix:///run/falco/falco.sock
{"hostname":"falco-4cvkq","output":"15:15:10.823823828: Notice Unexpected UDP Traffic Seen (connection=10.42.8.48:35662->216.58.215.68:65535 lport=65535 rport=35662 fd_type=ipv4 fd_proto=fd.l4proto evt_type=connect user=haproxy user_uid=1000 user_loginuid=-1 process=haproxy proc_exepath=/usr/local/sbin/haproxy parent=haproxy-ingress command=haproxy -f /etc/haproxy/haproxy.cfg.3dbbea03-1ebe-4d0a-876c-441ac4f102ab -c -f /etc/haproxy/haproxy-aux.cfg terminal=0 exe_flags=<NA> container_id=69d22f20d812 container_image=<NA> container_image_tag=<NA> container_name=<NA> k8s_ns=<NA> k8s_pod_name=<NA>)","priority":"Notice","rule":"Unexpected UDP Traffic","source":"syscall","tags":["TA0011","container","host","maturity_incubating","mitre_exfiltration","network"],"time":"2024-03-18T15:15:10.823823828Z", "output_fields": {"container.id":"69d22f20d812","container.image.repository":null,"container.image.tag":null,"container.name":null,"evt.arg.flags":null,"evt.time":1710774910823823828,"evt.type":"connect","fd.lport":65535,"fd.name":"10.42.8.48:35662->216.58.215.68:65535","fd.rport":35662,"fd.type":"ipv4","k8s.ns.name":null,"k8s.pod.name":null,"proc.cmdline":"haproxy -f /etc/haproxy/haproxy.cfg.3dbbea03-1ebe-4d0a-876c-441ac4f102ab -c -f /etc/haproxy/haproxy-aux.cfg","proc.exepath":"/usr/local/sbin/haproxy","proc.name":"haproxy","proc.pname":"haproxy-ingress","proc.tty":0,"user.loginuid":-1,"user.name":"haproxy","user.uid":1000}}

Looks like I need to fix up the deprecations in the rules as well

[Andreagit97]

From our previous conversation I thought k8s.pod.* and k8s.ns.name should be available so I did not enable the new k8smeta plugin.

Ok, I updated my previous answer because it was not clear. Yes, k8s.pod.* and k8s.ns.name fields are available through the container runtime so you should see these... Some additional questions:

• Are these fields available sometimes or are they never filled?
• Which was the previous version of Falco with these fields available? 0.36.0?
• For what concern other container.* fields, such as container.name, container.id, do you see them, or again are they never filled? I would like to understand if the container enrichment doesn't work at all with this setup or if it doesn't work just for k8s fields.

I can see the socket is mounted in /host/run/containerd/containerd.sock and it's available; however, I can't see this path being referenced anywhere in Falcos config. Is it hardcoded?

the helm chart do the magic under the hood, passing this value through the Falco command line https://github.com/falcosecurity/charts/blob/68db0665712d657be00be36e5b7933626220d3e7/charts/falco/templates/pod-template.tpl#L67

[mateuszdrab]

Are these fields available sometimes or are they never filled?

From what I can see in Grafana, since Feb 6th the field was never populated.

Which was the previous version of Falco with these fields available? 0.36.0?

Indeed that's correct

For what concern other container.* fields, such as container.name, container.id, do you see them, or again are they never filled? I would like to understand if the container enrichment doesn't work at all with this setup or if it doesn't work just for k8s fields.

container.id is there

the helm chart do the magic under the hood, passing this value through the Falco command line https://github.com/falcosecurity/charts/blob/68db0665712d657be00be36e5b7933626220d3e7/charts/falco/templates/pod-template.tpl#L67

That's weird because the socket is mounted in /host/run/containerd/containerd.sock but the --cri argument in Falco is pointing to /run/containerd/containerd.sock. I tried changing it and also added --disable-cri-async but no luck.

[Andreagit97]

container.id is there

Okay this is fine because it's not extracted directly from the container runtime, what about the container.name or container.image?

That's weird because the socket is mounted in /host/run/containerd/containerd.sock but the --cri argument in Falco is pointing to /run/containerd/containerd.sock.

Yep this is fine because internally Falco will add /host to the provided path, so the result will be host/run/containerd/containerd.sock

[mateuszdrab]

container.id is there

Okay this is fine because it's not extracted directly from the container runtime, what about the container.name or container.image?

That's weird because the socket is mounted in /host/run/containerd/containerd.sock but the --cri argument in Falco is pointing to /run/containerd/containerd.sock.

Yep this is fine because internally Falco will add /host to the provided path, so the result will be host/run/containerd/containerd.sock

That makes sense, and no, the container image and name are missing.

Example log line (sorry I posted it above but it was hard to notice under the Falco startup warnings):

{
    "hostname": "falco-4cvkq",
    "output": "15:15:10.823823828: Notice Unexpected UDP Traffic Seen (connection=10.42.8.48:35662->216.58.215.68:65535 lport=65535 rport=35662 fd_type=ipv4 fd_proto=fd.l4proto evt_type=connect user=haproxy user_uid=1000 user_loginuid=-1 process=haproxy proc_exepath=/usr/local/sbin/haproxy parent=haproxy-ingress command=haproxy -f /etc/haproxy/haproxy.cfg.3dbbea03-1ebe-4d0a-876c-441ac4f102ab -c -f /etc/haproxy/haproxy-aux.cfg terminal=0 exe_flags=<NA> container_id=69d22f20d812 container_image=<NA> container_image_tag=<NA> container_name=<NA> k8s_ns=<NA> k8s_pod_name=<NA>)",
    "priority": "Notice",
    "rule": "Unexpected UDP Traffic",
    "source": "syscall",
    "tags": [
        "TA0011",
        "container",
        "host",
        "maturity_incubating",
        "mitre_exfiltration",
        "network"
    ],
    "time": "2024-03-18T15:15:10.823823828Z",
    "output_fields": {
        "container.id": "69d22f20d812",
        "container.image.repository": null,
        "container.image.tag": null,
        "container.name": null,
        "evt.arg.flags": null,
        "evt.time": 1710774910823823828,
        "evt.type": "connect",
        "fd.lport": 65535,
        "fd.name": "10.42.8.48:35662->216.58.215.68:65535",
        "fd.rport": 35662,
        "fd.type": "ipv4",
        "k8s.ns.name": null,
        "k8s.pod.name": null,
        "proc.cmdline": "haproxy -f /etc/haproxy/haproxy.cfg.3dbbea03-1ebe-4d0a-876c-441ac4f102ab -c -f /etc/haproxy/haproxy-aux.cfg",
        "proc.exepath": "/usr/local/sbin/haproxy",
        "proc.name": "haproxy",
        "proc.pname": "haproxy-ingress",
        "proc.tty": 0,
        "user.loginuid": -1,
        "user.name": "haproxy",
        "user.uid": 1000
    }
}

Here is a log line from January (during time when sometimes the labels were missing until I restarted Falco):

{
    "hostname": "falco-5gr45",
    "output": "23:22:39.860302578: Notice Unexpected UDP Traffic Seen (connection=10.42.11.96:57006->172.217.16.4:65535 lport=65535 rport=57006 fd_type=ipv4 fd_proto=fd.l4proto evt_type=connect user=haproxy user_uid=1000 user_loginuid=-1 process=haproxy proc_exepath=/usr/local/sbin/haproxy parent=haproxy-ingress command=haproxy -f /etc/haproxy/haproxy.cfg.41e7ba76-e4b7-4e6e-b1fe-e9e596b4705e -c -f /etc/haproxy/haproxy-aux.cfg terminal=0 exe_flags=<NA> container_id=628333d134f0 container_image=<NA> container_image_tag=<NA> container_name=<NA> k8s_ns=<NA> k8s_pod_name=<NA>)",
    "priority": "Notice",
    "rule": "Unexpected UDP Traffic",
    "source": "syscall",
    "tags": [
        "TA0011",
        "container",
        "host",
        "maturity_incubating",
        "mitre_exfiltration",
        "network"
    ],
    "time": "2024-01-02T23:22:39.860302578Z",
    "output_fields": {
        "container.id": "628333d134f0",
        "container.image.repository": null,
        "container.image.tag": null,
        "container.name": null,
        "evt.arg.flags": null,
        "evt.time": 1704237759860302578,
        "evt.type": "connect",
        "fd.lport": 65535,
        "fd.name": "10.42.11.96:57006->172.217.16.4:65535",
        "fd.rport": 57006,
        "fd.type": "ipv4",
        "k8s.ns.name": null,
        "k8s.pod.name": null,
        "proc.cmdline": "haproxy -f /etc/haproxy/haproxy.cfg.41e7ba76-e4b7-4e6e-b1fe-e9e596b4705e -c -f /etc/haproxy/haproxy-aux.cfg",
        "proc.exepath": "/usr/local/sbin/haproxy",
        "proc.name": "haproxy",
        "proc.pname": "haproxy-ingress",
        "proc.tty": 0,
        "user.loginuid": -1,
        "user.name": "haproxy",
        "user.uid": 1000
    }
}

Here is a log line from January (during time when labels were working):

{
    "hostname": "falco-rbhwh",
    "output": "17:52:12.097741890: Notice Unexpected UDP Traffic Seen (connection=10.42.7.209:57044->140.82.121.3:443 lport=443 rport=57044 fd_type=ipv4 fd_proto=fd.l4proto evt_type=connect user=nobody user_uid=65534 user_loginuid=-1 process=git-remote-http proc_exepath=/usr/libexec/git-core/git-remote-http parent=git command=git-remote-http https://github.com/ansible/awx-operator https://github.com/ansible/awx-operator terminal=0 exe_flags=<NA> container_id=5c9a5e69c74f container_image=<NA> container_image_tag=<NA> container_name=<NA> k8s_ns=flux-system k8s_pod_name=kustomize-controller-7768d9d4dd-l62x7)",
    "priority": "Notice",
    "rule": "Unexpected UDP Traffic",
    "source": "syscall",
    "tags": [
        "TA0011",
        "container",
        "host",
        "maturity_incubating",
        "mitre_exfiltration",
        "network"
    ],
    "time": "2024-01-02T17:52:12.097741890Z",
    "output_fields": {
        "container.id": "5c9a5e69c74f",
        "container.image.repository": null,
        "container.image.tag": null,
        "container.name": null,
        "evt.arg.flags": null,
        "evt.time": 1704217932097741890,
        "evt.type": "connect",
        "fd.lport": 443,
        "fd.name": "10.42.7.209:57044->140.82.121.3:443",
        "fd.rport": 57044,
        "fd.type": "ipv4",
        "k8s.ns.name": "flux-system",
        "k8s.pod.name": "kustomize-controller-7768d9d4dd-l62x7",
        "proc.cmdline": "git-remote-http https://github.com/ansible/awx-operator https://github.com/ansible/awx-operator",
        "proc.exepath": "/usr/libexec/git-core/git-remote-http",
        "proc.name": "git-remote-http",
        "proc.pname": "git",
        "proc.tty": 0,
        "user.loginuid": -1,
        "user.name": "nobody",
        "user.uid": 65534
    }
}

[Andreagit97]

Ok, thank you for all the info! yes, this seems a regression on k3s v1.27.7+k3s2 with containerd since Falco 0.36.0 worked on this same env, we will take a look! (@incertum any idea? you probably have more experience with container runtime)

As a temp workaround, you could try to enable the k8smeta plugin to collect the k8s metadata, as explained here #2973 (comment)

[mateuszdrab]

Sure @Andreagit97

Just did a quick (slow) Loki query over the past 90 days and we can see the below

Since Feb 6th, there have been thousands of alerts with no namespace and none with namespace.
Before that, there were very little alerts because they were being suppressed successfully.
You can also see a spike of alerts before Feb 6th, this was actually the reason I started this investigation which I mentioned earlier #2973 (comment).
Essentially, there used to be periods when Falco would no longer respect my overrides, turned out the pods had to be restarted as the k8s fields were missing.

[Andreagit97]

Ok got it and Feb 6th is the date on which you switched from 0.36.0 to 0.37.1... pretty interesting

You can also see a spike of alerts before Feb 6th, this was actually the reason I started this investigation which I mentioned earlier #2973 (comment).
Essentially, there used to be periods when Falco would no longer respect my overrides, turned out the pods had to be restarted as the k8s fields were missing.

Unfortunately this seems yet another issue :/

[mateuszdrab]

k8smeta

True, let's focus on this one for now as it's not working at all.

I've enabled k8smeta and I'm still not seeing any metadata added.

I'm upgrading k3s now to v1.27.11-k3s1 - will report back soon.

[Andreagit97]

Please note that when you enable the k8smeta plugin the name of the fields to use is different, so no more k8s.pod.* but you need to use k8smeta.pod.* so probably you need to change these fields in your rules

[alacuku]

@mateuszdrab here you can find all the fields exposed by the k8smeta plugin: https://github.com/falcosecurity/plugins/tree/master/plugins/k8smeta#supported-fields

[Andreagit97]

I can confirm Falco 0.37.1 doesn't extract k8s* fields on K3s, while Falco 0.36.0 seems to do the job. Both versions don't extract the container.name and container.image maybe we could evaluate exposing all this info through the k8smeta plugin and use it as the standard collection method in k8s environments.

[mateuszdrab]

@mateuszdrab here you can find all the fields exposed by the k8smeta plugin: https://github.com/falcosecurity/plugins/tree/master/plugins/k8smeta#supported-fields

Unfortunately, despite enabling the plugin, no k8smeta labels are present.

[incertum]

Question: Talking about k3s shouldn't the socket be located at /run/k3s/containerd/containerd.sock instead? If not too much trouble we would appreciate feedback on the above.

For example the CNCF Green Reviews TAG has k3s containerd on their cluster and I am getting the container information. See the setup here: We explicitly mount the k3s path as well https://github.com/falcosecurity/cncf-green-review-testing/blob/main/kustomize/falco-driver/modern_ebpf/daemonset.yaml#L77 and https://github.com/falcosecurity/cncf-green-review-testing/blob/main/kustomize/falco-driver/modern_ebpf/daemonset.yaml#L124 plus it's one of the default locations, see our Troubleshooting Guide https://falco.org/docs/troubleshooting/missing-fields/#missing-container-images therefore no need to pass the socket location with the --cri flag unless it's a custom socket location after all.

[mateuszdrab]

That was it, the /run/containerd/containerd.sock belongs to Docker's containerd on the same node.
K3s runs it's own containerd in /run/k3s/containerd/containerd.sock.

ctr ns ls
NAME LABELS
moby

ctr -a /run/k3s/containerd/containerd.sock ns ls
NAME   LABELS
k8s.io

I was checking the paths earlier and I was trying to remember how it was set up since I've not touched it in ages. I think the reason why I left it like that is because I wanted containerd to follow K3s's version as sharing them required extra configuration.

It remains bizzare how it used to work and now it doesn't.

Thank you @incertum!

Maybe worth adding some sort of validation at start-up that the socket is working but has no pods or something like that?

Btw, k8smeta collector still isn't populating its fields

{
    "hostname": "falco-6hjsd",
    "output": "22:18:09.254710323: Notice Unexpected UDP Traffic Seen (connection=10.42.8.141:35588->104.16.60.181:443 lport=35588 rport=443 fd_type=ipv4 fd_proto=fd.l4proto evt_type=connect user=root user_uid=0 user_loginuid=-1 process=stats_thread proc_exepath=/fr24feed/fr24feed/fr24feed parent=bash command=stats_thread terminal=0 exe_flags=<NA> container_id=a5613d461804 container_image=docker.io/thomx/fr24feed-piaware container_image_tag=latest container_name=fr24feed-piaware k8s_ns=flight-tracking k8s_pod_name=fr24feed-piaware-ashford-67f544d679-b2qbx)",
    "priority": "Notice",
    "rule": "Unexpected UDP Traffic",
    "source": "syscall",
    "tags": [
        "TA0011",
        "container",
        "host",
        "maturity_incubating",
        "mitre_exfiltration",
        "network"
    ],
    "time": "2024-03-19T22:18:09.254710323Z",
    "output_fields": {
        "container.id": "a5613d461804",
        "container.image.repository": "docker.io/thomx/fr24feed-piaware",
        "container.image.tag": "latest",
        "container.name": "fr24feed-piaware",
        "evt.arg.flags": null,
        "evt.time": 1710886689254710323,
        "evt.type": "connect",
        "fd.lport": 35588,
        "fd.name": "10.42.8.141:35588->104.16.60.181:443",
        "fd.rport": 443,
        "fd.type": "ipv4",
        "k8s.ns.name": "flight-tracking",
        "k8s.pod.name": "fr24feed-piaware-ashford-67f544d679-b2qbx",
        "proc.cmdline": "stats_thread",
        "proc.exepath": "/fr24feed/fr24feed/fr24feed",
        "proc.name": "stats_thread",
        "proc.pname": "bash",
        "proc.tty": 0,
        "user.loginuid": -1,
        "user.name": "root",
        "user.uid": 0
    }
}

[Andreagit97]

I was checking the paths earlier and I was trying to remember how it was set up since I've not touched it in ages. I think the reason why I left it like that is because I wanted containerd to follow K3s's version as sharing them required extra configuration.
It remains bizzare how it used to work and now it doesn't.

Oh ok now it is clear, Falco 0.36.2 works also without the right container path because these k8s.* fields are collected by the old k8s client that is no more present in Falco 0.37.0. This also explains why both versions don't extract the container.name and container.image... the old k8s client cannot extract these container fields they are extracted only from the container runtime, but the path was wrong...

Btw, k8smeta collector still isn't populating its fields

I don't see any k8smeta field in your rule. To test the plugin you need to add something to your rules, for example, some output fields like k8s_pod_name=%k8smeta.pod.name k8s_service_name=%k8smeta.svc.name

[mateuszdrab]

Oh ok now it is clear, Falco 0.36.2 works also without the right container path because these k8s.* fields are collected by the old k8s client that is no more present in Falco 0.37.0. This also explains why both versions don't extract the container.name and container.image... the old k8s client cannot extract these container fields they are extracted only from the container runtime, but the path was wrong...

That makes sense regarding the old client - I'm glad I got this working. Hopefully the other issue about the k8s fields going missing after some time will no longer be present due to the switch of the metadata sourcing.

I don't see any k8smeta field in your rule. To test the plugin you need to add something to your rules, for example, some output fields like k8s_pod_name=%k8smeta.pod.name k8s_service_name=%k8smeta.svc.name

Shouldn't the alerts have those fields in the json though, even if the rules are not specifically using it at the moment?

[Andreagit97]

Shouldn't the alerts have those fields in the json though, even if the rules are not specifically using it at the moment?

Nope at the moment you can use them just adding them to the rules

[mateuszdrab]

Shouldn't the alerts have those fields in the json though, even if the rules are not specifically using it at the moment?

Nope at the moment you can use them just adding them to the rules

Ahh, okay.

Would be nice to see them in the log.
I run some Loki queries and figure out what to exempt based on the most noisy and false-positive alerts.
However, I should be fine with the basic K8s.* labels.

Thanks for your time and support