DHTDiscoveryCommunity IPv6 support

[synctext]

Running latest v8.0.4 on Mac.

DiscoveryCommunity and all others show 50% of overlay peers with ::ffff:192.42.1.1 like IPv6 adress. Quite a lot! 10 out of 21 peers in one overlay is IPv6.

DHT seems to fail. At least no display of them.

[qstokkink]

The DHT community has its own separate pool of peers. It is expected that the peers in other communities do not appear in the DHT community. However, it is not expected that this community never gets any peers.

Currently, the DHT community is not actually used by any Tribler components, making this a lower priority bug. I'll schedule this for 8.1.0.

[qstokkink]

@egbertbouman I assigned you, because as the author of the DHT community you can authoritatively determine if there is indeed something wrong.

[qstokkink]

I am seeing the following tracebacks for IPv6:

Traceback (most recent call last):
  File "tribler/pyipv8/ipv8/messaging/serialization.py", line 510, in pack_serializable
    packed += self._packers[packable[0]].pack(*packable[1:])
  File "tribler/pyipv8/ipv8/messaging/serialization.py", line 238, in pack
    return pack('>4sH', socket.inet_aton(data[0]), data[1])
OSError: illegal IP address string passed to inet_aton

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "tribler/pyipv8/ipv8/community.py", line 599, in on_packet
    result = cast(Callable[[Address, bytes], None], handler)(source_address, data)
  File "tribler/pyipv8/ipv8/lazy_community.py", line 111, in wrapper
    return func(self, peer or Peer(auth.public_key_bin, source_address), *unpacked)  # type: ignore[arg-type]
  File "tribler/pyipv8/ipv8/community.py", line 425, in on_old_introduction_request
    self.on_introduction_request(peer, dist, payload)
  File "tribler/pyipv8/ipv8/community.py", line 456, in on_introduction_request
    packet = self.create_introduction_response(payload.destination_address, peer.address, payload.identifier,
  File "tribler/pyipv8/ipv8/messaging/anonymization/community.py", line 696, in create_introduction_response
    return super().create_introduction_response(lan_socket_address, socket_address,
  File "tribler/pyipv8/ipv8/community.py", line 347, in create_introduction_response
    return self._ez_pack(prefix or self._prefix, payload.msg_id, [auth, dist, payload])
  File "tribler/pyipv8/ipv8/lazy_community.py", line 263, in _ez_pack
    packet = prefix + bytes([msg_num]) + self.serializer.pack_serializable_list(payloads)
  File "tribler/pyipv8/ipv8/messaging/serialization.py", line 524, in pack_serializable_list
    return b''.join(self.pack_serializable(serializable) for serializable in serializables)
  File "tribler/pyipv8/ipv8/messaging/serialization.py", line 524, in <genexpr>
    return b''.join(self.pack_serializable(serializable) for serializable in serializables)
  File "tribler/pyipv8/ipv8/messaging/serialization.py", line 513, in pack_serializable
    raise PackError(msg) from e
ipv8.messaging.serialization.PackError: Could not pack item: ('ipv4', UDPv6Address(ip='::ffff:XXX.XX.XXX.XXX', port=XXXXX))

[Mario-o-o-o]

Hello

I see a lot of equivalent errors in the logs of the docker container "image: ghcr.io/tribler/tribler:latest" installed a week ago ...

ERROR:TriblerTunnelCommunity:Exception occurred while handling packet!
Traceback (most recent call last):
  File "/home/user/tribler/pyipv8/ipv8/messaging/serialization.py", line 510, in pack_serializable
    packed += self._packers[packable[0]].pack(*packable[1:])
  File "/home/user/tribler/pyipv8/ipv8/messaging/serialization.py", line 238, in pack
    return pack('>4sH', socket.inet_aton(data[0]), data[1])
OSError: illegal IP address string passed to inet_aton
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/user/tribler/pyipv8/ipv8/community.py", line 599, in on_packet
    result = cast(Callable[[Address, bytes], None], handler)(source_address, data)
  File "/home/user/tribler/pyipv8/ipv8/lazy_community.py", line 111, in wrapper
    return func(self, peer or Peer(auth.public_key_bin, source_address), *unpacked)  # type: ignore[arg-type]
  File "/home/user/tribler/pyipv8/ipv8/community.py", line 425, in on_old_introduction_request
    self.on_introduction_request(peer, dist, payload)
  File "/home/user/tribler/pyipv8/ipv8/community.py", line 456, in on_introduction_request
    packet = self.create_introduction_response(payload.destination_address, peer.address, payload.identifier,
  File "/home/user/tribler/pyipv8/ipv8/messaging/anonymization/community.py", line 696, in create_introduction_response
    return super().create_introduction_response(lan_socket_address, socket_address,
  File "/home/user/tribler/pyipv8/ipv8/community.py", line 347, in create_introduction_response
    return self._ez_pack(prefix or self._prefix, payload.msg_id, [auth, dist, payload])
  File "/home/user/tribler/pyipv8/ipv8/lazy_community.py", line 263, in _ez_pack
    packet = prefix + bytes([msg_num]) + self.serializer.pack_serializable_list(payloads)
  File "/home/user/tribler/pyipv8/ipv8/messaging/serialization.py", line 524, in pack_serializable_list
    return b''.join(self.pack_serializable(serializable) for serializable in serializables)
  File "/home/user/tribler/pyipv8/ipv8/messaging/serialization.py", line 524, in <genexpr>
    return b''.join(self.pack_serializable(serializable) for serializable in serializables)
  File "/home/user/tribler/pyipv8/ipv8/messaging/serialization.py", line 513, in pack_serializable
    raise PackError(msg) from e
ipv8.messaging.serialization.PackError: Could not pack item: ('ipv4', UDPv6Address(ip='::ffff:76.39.54.145', port=60432))
OSError: illegal IP address string passed to inet_aton
Traceback (most recent call last):
  File "/home/user/tribler/pyipv8/ipv8/messaging/serialization.py", line 510, in pack_serializable
    packed += self._packers[packable[0]].pack(*packable[1:])
  File "/home/user/tribler/pyipv8/ipv8/messaging/serialization.py", line 238, in pack
    return pack('>4sH', socket.inet_aton(data[0]), data[1])
OSError: illegal IP address string passed to inet_aton
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/user/tribler/pyipv8/ipv8/community.py", line 599, in on_packet
    result = cast(Callable[[Address, bytes], None], handler)(source_address, data)
  File "/home/user/tribler/pyipv8/ipv8/lazy_community.py", line 111, in wrapper
    return func(self, peer or Peer(auth.public_key_bin, source_address), *unpacked)  # type: ignore[arg-type]
  File "/home/user/tribler/pyipv8/ipv8/community.py", line 425, in on_old_introduction_request
    self.on_introduction_request(peer, dist, payload)
  File "/home/user/tribler/pyipv8/ipv8/community.py", line 456, in on_introduction_request
    packet = self.create_introduction_response(payload.destination_address, peer.address, payload.identifier,
  File "/home/user/tribler/pyipv8/ipv8/messaging/anonymization/community.py", line 696, in create_introduction_response
    return super().create_introduction_response(lan_socket_address, socket_address,
  File "/home/user/tribler/pyipv8/ipv8/community.py", line 347, in create_introduction_response
    return self._ez_pack(prefix or self._prefix, payload.msg_id, [auth, dist, payload])
  File "/home/user/tribler/pyipv8/ipv8/lazy_community.py", line 263, in _ez_pack
    packet = prefix + bytes([msg_num]) + self.serializer.pack_serializable_list(payloads)
  File "/home/user/tribler/pyipv8/ipv8/messaging/serialization.py", line 524, in pack_serializable_list
    return b''.join(self.pack_serializable(serializable) for serializable in serializables)
  File "/home/user/tribler/pyipv8/ipv8/messaging/serialization.py", line 524, in <genexpr>
    return b''.join(self.pack_serializable(serializable) for serializable in serializables)
  File "/home/user/tribler/pyipv8/ipv8/messaging/serialization.py", line 513, in pack_serializable
    raise PackError(msg) from e
ipv8.messaging.serialization.PackError: Could not pack item: ('ipv4', UDPv6Address(ip='::ffff:76.39.54.145', port=60432))

it never seems to involve private IP addresses.

I have a feeling it has something to do with this issue, but maybe not.

Thx for your help

[egbertbouman]

This issue seems to be about multiple things: too many mapped IPv4 addresses, the DHT not working, and peer discovery tracebacks in the TunnelCommunity. I'll stick with the latter, as it's the most clearly described problem.

The following seems to be happening. We receive an old-style introduction-request from a peer we haven't met before. The introduction-request has supports_new_style=False, as that seems to be the default when walking to a new address. We receive this request over our IPv6 socket, which is why it shows as mapped IPv4. Next, we try to send an introduction-response. However, since the supports_new_style is set to false, we create an old-style introduction-response, using the mapped IPv4 address. Since the old-style payloads don't support this address type, this produces the error.

To me, this looks more like a general IPv8 issue. I've been looking at the tracebacks for a while now, and I don't really understand why only the TunnelCommunity seems affected. Maybe it's because the TunnelCommunity sends introduction-requests to all peers in order to discover the peer flags.

I'm not sure why we announce that we don't support new-style introduction-messages when meeting a new peer. I'd think that this would be hard-coded to true, since our current code-base does support it. But maybe I'm missing something. Anyway, an obvious way to fix would be to just check the address before packing it as IPv4.

[qstokkink]

Interesting 🤔 The assumption inside IPv8 is that old peers and old-style peers don't support anything but IPv4 and that it is, therefore, impossible for old peers to send anything to us using IPv6 addresses. However, clearly, what you say contradicts that:

We receive this request over our IPv6 socket

I guess that still means the source is secretly IPv4, but it got remapped into the IPv6 socket (somehow? 😕), and we can safely "un-remap" it back to IPv4 when sending a response. I'm still not sure how it is possible to send to an IPv6 address from an old IPv4 endpoint, but I agree with the approach to check the address and pack it as IPv4.