scapy3k

[phaethon]

Have you considered porting this to python3 and scapy-python3 (http://github.com/phaethon/scapy)?
I am ready to contribute 50% of the effort.

[alexmgr]

Hi,

I thought scapy wasn't python3 compatible until I saw your fork. Nice work! Since you have obvious experience with the transition process, do you have a rough idea of how much effort it would take to port this code base?
Could we have the codebase work on both versions, to avoid handling a python3 branch?

That being said, I'd be happy to take this on with you.

Alex

[phaethon]

By quickly looking at the code, I think it is mostly replacing some cases of str with bytes and some 'asdf' with b'asdf'. Most likely doable within a day of work or so.
Common codebase for both versions would be quite inconvenient for cases when there are string vs. bytes constants, which you have. For each such case you would have to write conditional statements and loose readability of the code.
One more option would be to have ssl_tls python3 code directly integrated into scapy3k as a module instead of being separate package. My experience shows that having to install additional packages frequently prevents people from using it.

[alexmgr]

Thanks for the information. Six package seems to allow some abstraction types for this use case. I think we don't want to maintain multiple branches.
I'll give it some more thought.

[phaethon]

scapy3k in the future might also divert from internal API of scapy 2.x, and adding library like six seems quite an overhead for scapy-ssl_tls, which does not have too many LOC at present. My guess is that maintaining compatibility layer (six, anything checking actual version and doing different things based on that, etc.) will be more laborious than the effort of separate branch.

[calebmadrigal]

This would be awesome, scapy3k is awesome, and so is scapy-ssl_tls :)

[MichaelLiuzzi]

Hey guys,
Did the port ever happen?

[alexmgr]

Hi @MichaelLiuzzi,
Sorry for the late answer. Nothing really happened on this. Anyone here interested in picking this up?

[phaethon]

If anybody is doing it, I will provide scapy3k support.

[cniemira]

There's a semi-functional py3 fork here, which I've submitted some tweaks to. I might be able to find some cycles to get a better port happening in the near future.

[alexmgr]

Thanks for the heads up @cniemira,

If you have some time to move this forward, could you work on top of the tls1.3 branch?

Appreciate the help,
Alex

[tintinweb]

Hi,

would be great to come up with a list of issues that prevent scapy-ssl_tls from being py3 compatible. Afaik, some of the issues fixed in the branch mentioned by @cniemira are already fixed. Other things like working with bytestr and ord could be tackled with a generic py3compat layer. What I'd like to find out however is if there's something in scapy-ssl_tls that would require substantial changes in order to be py3 compatible.

tin

[cniemira]

FWIW, I have a functional py3 (only) compatible fork here. I've got a whole py3/jupyter/scapy/x509 stack built up in a docker image that's currently very much hacked together, but which I'd like to clean up.

[cniemira]

BTW - I'd be happy to clean up my fork and submit a PR back here when I get a few minutes. The reason I haven't done so yet is that I dropped py2 support, which I'd need to put back, since I gather it's still desirable by some.

[tintinweb]

👍 for the docker image!

I had a cursory look at your diff and it kind of boils down to the following changes:

• print_function - already been fixed for releases/1.2.3
• exceptions 'as' - could be changed, may break py26 support
• no ord - can be tackled with a py3compat layer
• dict.items instead of iteritems - could be changed (we'd loose the iterator in py27) or tackled with py3compat layer (always an iterator)
• import StringIO - py3compat providing the right import
• next() instead of .next() - py3compat
• b'str' - py3compat b()

//Note: py26 support might be required in order to get scapy to work on windows - not sure if thats still the case

Would you mind submitting a PR targeting release/1.2.3 (thats the next release)? Just make sure it works for py3 and I'll try to add a compat layer fixing py2.

appreciate your help!

cheers,
tin

[cniemira]

Submitted! Happy to help.

[phaethon]

Good news, that this is moving. I have not checked out the code yet, but any specific help needed on scapy3k side?

[tintinweb]

@phaethon we're in the process of experimenting with a py3compat layer for release/1.2.3. I do not have any experience with scapy3k and therefore would appreciate your help in testing our code once we're ready.

[tintinweb]

Branched experimental pycompat off master and did some initial work based on the diff @cniemira provided. The basic idea is that incompatibilities are being handled by the pycompat layer which provides a stable interface for both versions.

[phaethon]

@tintinweb, any news on this?

[tintinweb]

@phaethon we'd appreciate feedback to the pycompat approach in pycompat. Would be great if you could give it a try.

[ghost]

@tintinweb I just setup scapy_ssl_tls in python3 and to let it work i had to change two things:
First had to remove the line
scapy>=2.2.0,<2.3.3
to
scapy-python3
in the file "requirements.txt" (Ln. 2)

and i had to change the line
except AttributeError, ex:
to
except AttributeError as ex:
in the file "setup.py" (Ln. 31)

Then i tried the code:
from scapy_ssl_tls import *

and it imported without errors, i didn't test it so i am not sure
but i think it works

[59e5aaf4]

Hi all, and thank you very much for this nice piece of code.

I had all the errors in the world at parsing TLS traffic with some python2 code, so I decided to switch to python3. For reference, and to spare the blood price to future users :

• python3 versions of scapy and scapy-ssl_tls can totally be used
• don't forget not to use the very old pip packages or your debian-based os
• don't use the bleeding edge git repos on the master branch

The steps are described at the end of this message, here are some error messages to attract search results.

  File "/usr/lib/python2.7/dist-packages/scapy/packet.py", line 172, in getfield_and_val
    return self.payload.getfield_and_val(attr)
  File "/usr/lib/python2.7/dist-packages/scapy/packet.py", line 1057, in getfield_and_val
    raise AttributeError(attr)
AttributeError: dns
(caused when trying to print the hexdump of a packet via 'str' (which yields back bytes) via do_build on packets containing x509 certificates, should not be related to improper packets)

Traceback (most recent call last):
  File "parser.py", line 9, in <module>
    from scapy_ssl_tls.ssl_tls import TLS
  File "/usr/local/lib/python3.4/dist-packages/scapy_ssl_tls/ssl_tls.py", line 11, in <module>
    import ssl_tls_registry as registry
ImportError: No module named 'ssl_tls_registry'
(caused by some reason, tried to hack the files in place, then realised I ran setup.py with the master branch, different from the py3compat branch)

It works now, on a somewhat fresh devuan :

user@devuan:~$ lsb_release -crdi
Distributor ID:	Devuan
Description:	Devuan GNU/Linux 1.0 (jessie)
Release:	1.0
Codename:	jessie
user@devuan:~$ dpkg -l | grep -E '(ii +python[0-9\.]+ |scapy)'
ii  python2.7                             2.7.9-2+deb8u1                             amd64        Interactive high-level object-oriented language (version 2.7)
ii  python3                               3.4.2-2                                    amd64        interactive high-level object-oriented language (default python3 version)
ii  python3.4                             3.4.2-1                                    amd64        Interactive high-level object-oriented language (version 3.4)
user@devuan:~$ python -m pip freeze|grep scapy
user@devuan:~$ python3 -m pip freeze|grep scapy
scapy-python3==0.22
scapy-ssl-tls==1.3.0
user@devuan:~$ python3 -c 'from scapy import *;print(TLS)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'TLS' is not defined
user@devuan:~$ python3 -c 'from scapy.all import *;print(TLS)'
WARNING: Failed to execute tcpdump. Check it is installed and in the PATH
WARNING: No route found for IPv6 destination :: (no default route?). This affects only IPv6
<class 'scapy.layers.ssl_tls.SSL'>

Installation steps

# python3 -m easy_install pip
# python3 -m pip install scapy
# git clone https://github.com/tintinweb/scapy-ssl_tls.git
# git fetch
# git branch -v -a
* master                                0144aaf Fixed Attribute error TLSCertificateRequest issue #106
  remotes/origin/HEAD                   -> origin/master
  remotes/origin/master                 0144aaf Fixed Attribute error TLSCertificateRequest issue #106
  remotes/origin/master-secret-proposal 3b7d120 add proposal for TLSKeyStore   (RSA, MasterSecret, PreMasterSecret, ...)
  remotes/origin/py3compat              e0f3be0 added py3compat layer
  remotes/origin/release/1.2.3          a523c1c prepare v1.2.3.2
  remotes/origin/renegotiation          c8709a0 Half working secure re-negotiation
# git checkout remotes/origin/py3compat
# python3 setup.py build
# python3 setup.py install

There you go, I hope this helps :)

[Te-k]

Hi,
It seems that the py3compat branch is not compatible with scapy 2.4.*, is there a big different in the code ? Would that take a lot of work to make the actual code python 3 compatible ?

[kalidasya]

for anyone who is interested, I created a py3 compatible branch from master: https://github.com/kalidasya/scapy-ssl_tls/tree/py3_update it is using scapy 2.4.*
I tested it by running all the unit tests (have not checked the examples at all, also the doc is not updated), I might do that later if someone is interested, PRs are welcome.

[jurgonaut]

I cant get it to work with python3, I tried with the @kalidasya update but got error when running an example.
My setup: python3.7 scapy 2.4.3, scapy-ssl-tls 3.0.0.
I tried running the example: full_rsa_connection_with_application_data.py
Traceback:

Traceback (most recent call last):
File "full_rsa_connection_with_application_data.py", line 50, in <module>
tls_client(server)
File "full_rsa_connection_with_application_data.py", line 32, in tls_client
server_hello, server_kex = tls_socket.do_handshake(tls_version, ciphers, extensions)
File "/usr/local/lib/python3.7/dist-packages/scapy_ssl_tls/ssl_tls.py", line 1313, in do_handshake
return tls_do_handshake(self, version, ciphers, extensions)
File "/usr/local/lib/python3.7/dist-packages/scapy_ssl_tls/ssl_tls.py", line 1527, in tls_do_handshake
resp1 = tls_do_round_trip(tls_socket, client_hello)
File "/usr/local/lib/python3.7/dist-packages/scapy_ssl_tls/ssl_tls.py", line 1506, in tls_do_round_trip
tls_socket.sendall(pkt)
File "/usr/local/lib/python3.7/dist-packages/scapy_ssl_tls/ssl_tls.py", line 1282, in sendall
self._s.sendall(pkt)
TypeError: a bytes-like object is required, not 'TLSRecord'

[kalidasya]

yeah I did not check the tests. probably some part what I did not use was not converted. might try to take a look into it this week, cannot promise much

[kalidasya]

I have a dis-functioning memory it seems, the tests were working then...

[jurgonaut]

Sorry my bad I meant example not test. I did not try to run the tests, I only tried with "full_rsa_connection_with_application_data.py" example and it failed.

[kalidasya]

@jurgonaut I updated my branch, fixed that as far as I could (was running fine for me) and a couple of other examples. some are failing but probably because I was using a self signed test server.

[jurgonaut]

Now it works, that you a lot for fixing it and for your fast response.

[tkoch96]

I cloned @kalidasya update, but when running:

from scapy.layers.ssl_tls import *

I get the following error:

venv/lib/python3.6/site-packages/scapy/layers/ssl_tls.py", line 227, in init

self.type_ = type_

AttributeError: 'TypedPacketListField' object has no attribute 'type_'

I don't see anyone else above having this problem. Did I install something incorrectly? Using scapy 2.4.3, Python 3.6.9.

[kalidasya]

@tkoch96 scapy has to be >=2.2.0,<2.3.3 (maybe just 2.3.*) at least thats what I see in the Pipfile