Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

winreg parser bails out on single corrupt key - please make it more resilient #3571

Closed
MikeHofmann opened this issue Apr 29, 2021 · 29 comments
Closed

winreg parser bails out on single corrupt key - please make it more resilient #3571

MikeHofmann opened this issue Apr 29, 2021 · 29 comments
Assignees
@joachimmetz
Labels
enhancement New or improved functionality parsers Issues related to parsers and parser plug-ins
Milestone
2021 June release

Comments

@MikeHofmann
Copy link

Description of problem:

Followup from #3238

log2timeline.py parses only the Service from the SYSTEM hive CurrentControlSet but ignores other ControlSets from past working configurations.

Command line and arguments:

log2timeline.py --debug --workers 1 system.plaso system
psort.py -o l2tcsv -w system.csv system.plaso

has been used to generate the timeline.

log2timeline.py sees the ControlSet002:

zcat log2timeline-20210429T074209.log.gz | grep ControlSet002 | wc -l
1031

but no Services are parsed:

zcat log2timeline-20210429T074209.log.gz | grep 'ControlSet002\\Services' | wc -l
0

See below for more details.

Source data:

I exported a SYSTEM Hive from a Windows XP (sigh) System. I extracted the corresponding Software Keys using xways by hand:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion]
"ProductName"="Microsoft Windows XP"
"CurrentVersion"="5.1"

The hive seems healthy:

ls -al system
-rwxrwxrwx 1 2143290499 2143289857 16515072 Feb  8 10:26 system

file system
system: MS Windows registry file, NT/2000 or above

and i can export from a past ControlSet002 a malicious service using:

reged -x system 'HKEY_LOCAL_MACHINE\SYSTEM' '\ControlSet002\Services\corspdeft' export.reg
cat export.reg

giving:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Services\corspdeft]
"Type"=dword:00000010
"Start"=dword:00000002
"ErrorControl"=dword:00000000
"ImagePath"=hex(2):22,00,43,00,3a,00,5c,00,57,00,49,00,4e,00,44,00,4f,00,57,00,\
  53,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,63,00,6f,\
  00,72,00,73,00,70,00,64,00,65,00,66,00,74,00,2e,00,65,00,78,00,65,00,22,00,\
  00,00
"DisplayName"="corspdeft"
"ObjectName"="LocalSystem"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Services\corspdeft\Security]
"Security"=hex:01,00,14,80,90,00,00,00,9c,00,00,00,14,00,00,00,30,00,00,00,02,\
  00,1c,00,01,00,00,00,02,80,14,00,ff,01,0f,00,01,01,00,00,00,00,00,01,00,00,\
  00,00,02,00,60,00,04,00,00,00,00,00,14,00,fd,01,02,00,01,01,00,00,00,00,00,\
  05,12,00,00,00,00,00,18,00,ff,01,0f,00,01,02,00,00,00,00,00,05,20,00,00,00,\
  20,02,00,00,00,00,14,00,8d,01,02,00,01,01,00,00,00,00,00,05,0b,00,00,00,00,\
  00,18,00,fd,01,02,00,01,02,00,00,00,00,00,05,20,00,00,00,23,02,00,00,01,01,\
  00,00,00,00,00,05,12,00,00,00,01,01,00,00,00,00,00,05,12,00,00,00

Plaso version:

log2timeline.py --version
plaso - log2timeline version 20210412

Operating system Plaso is running on:

Installed using latest docker image

Installation method:

see above

If multiple methods were used please indicate.
@joachimmetz
Copy link
Member

I'm unable to reproduce this with the SYSTEM file in test data:

2010-11-10T18:18:32.5312500+00:00,Content Modification Time,REG,Registry Key - Service,[HKEY_LOCAL_MACHINE\System\ControlSet001\services\LSI_SAS2] Type: Kernel Device Driver (0x1) Start: Manual (3) Image path: \SystemRoot\system32\DRIVERS\lsi_sas2.sys Error control: Normal (1) DriverPackageId: [REG_SZ] lsi_sas2.inf_x86_neutral_e12a5c4cfbe49204 Group: [REG_SZ] SCSI Miniport,winreg/windows_services,OS:test_data/SYSTEM,-
2010-11-10T18:18:32.5312500+00:00,Content Modification Time,REG,Registry Key - Service,[HKEY_LOCAL_MACHINE\System\ControlSet002\services\LSI_SAS2] Type: Kernel Device Driver (0x1) Start: Manual (3) Image path: \SystemRoot\system32\DRIVERS\lsi_sas2.sys Error control: Normal (1) DriverPackageId: [REG_SZ] lsi_sas2.inf_x86_neutral_e12a5c4cfbe49204 Group: [REG_SZ] SCSI Miniport,winreg/windows_services,OS:test_data/SYSTEM,-
2012-04-04T11:47:08.7656250+00:00,Content Modification Time,REG,Registry Key - Service,[HKEY_LOCAL_MACHINE\System\ControlSet001\services\LSI_SAS] Type: Kernel Device Driver (0x1) Start: Boot (0) Image path: \SystemRoot\system32\drivers\lsi_sas.sys Error control: Normal (1) Group: [REG_SZ] SCSI Miniport Tag: [REG_DWORD_LE] 64,winreg/windows_services,OS:test_data/SYSTEM,-
2012-04-04T11:47:08.7656250+00:00,Content Modification Time,REG,Registry Key - Service,[HKEY_LOCAL_MACHINE\System\ControlSet002\services\LSI_SAS] Type: Kernel Device Driver (0x1) Start: Boot (0) Image path: \SystemRoot\system32\drivers\lsi_sas.sys Error control: Normal (1) Group: [REG_SZ] SCSI Miniport Tag: [REG_DWORD_LE] 64,winreg/windows_services,OS:test_data/SYSTEM,-

@joachimmetz
Copy link
Member

@MikeHofmann can you provide the output of log2timeline.py --troubles and debug logs

Also see: https://plaso.readthedocs.io/en/latest/sources/Troubleshooting.html

@joachimmetz joachimmetz added needs closer look Issue that requires further analysis by a maintainer pending reporter input Issue is pending input from the reporter labels Apr 29, 2021
@joachimmetz joachimmetz changed the title Services from older ControlSets not parsed Not seeing services under ControlSet002 from Windows XP SYSTEM hive in psort results? Apr 29, 2021
@joachimmetz
Copy link
Member

joachimmetz commented Apr 29, 2021 •
edited
Loading

@MikeHofmann log2timeline-20210429T074209.log.gz is the log file, what does you psort output tell you?

@joachimmetz joachimmetz changed the title Not seeing services under ControlSet002 from Windows XP SYSTEM hive in psort results? Not seeing services under ControlSet002 from Windows XP SYSTEM hive in log2timeline log file? Apr 29, 2021
@joachimmetz
Copy link
Member

joachimmetz commented Apr 29, 2021 •
edited
Loading

@MikeHofmann also can you try dynamic output instead of l2tcsv. The amount of information l2tcsv can present is limited also see #3570

@joachimmetz joachimmetz changed the title Not seeing services under ControlSet002 from Windows XP SYSTEM hive in log2timeline log file? Not seeing services under ControlSet002 from Windows XP SYSTEM hive in l2tcsv output/log2timeline log file? Apr 29, 2021
@joachimmetz joachimmetz removed the needs closer look Issue that requires further analysis by a maintainer label Apr 29, 2021
@joachimmetz joachimmetz self-assigned this Apr 29, 2021
@joachimmetz
Copy link
Member

Then again for me l2tcsv does show both control sets:

11/02/2006,12:49:55,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet001\services\LSI_SAS\Parameters\PnpInter...,[HKEY_LOCAL_MACHINE\System\ControlSet001\services\LSI_SAS\Parameters\PnpInterface] 5: [REG_DWORD_LE] 1,2,OS:test_data/SYSTEM,-,-,winreg/winreg_default,sha256_hash: 96dc1f1cc3c0b44ef9af72d1c18a8e6a4338c67988f303d05693ca4be6bf7eb9
12/16/2008,02:28:41,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet001\services\LSI_SAS\Parameters] BusType...,[HKEY_LOCAL_MACHINE\System\ControlSet001\services\LSI_SAS\Parameters] BusType: [REG_DWORD_LE] 10,2,OS:test_data/SYSTEM,-,-,winreg/winreg_default,sha256_hash: 96dc1f1cc3c0b44ef9af72d1c18a8e6a4338c67988f303d05693ca4be6bf7eb9
07/14/2009,04:37:09,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet001\services\LSI_SAS2\Parameters] BusTyp...,[HKEY_LOCAL_MACHINE\System\ControlSet001\services\LSI_SAS2\Parameters] BusType: [REG_DWORD_LE] 10,2,OS:test_data/SYSTEM,-,-,winreg/winreg_default,sha256_hash: 96dc1f1cc3c0b44ef9af72d1c18a8e6a4338c67988f303d05693ca4be6bf7eb9
07/14/2009,04:37:09,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet001\services\LSI_SAS2\Parameters\PnpInte...,[HKEY_LOCAL_MACHINE\System\ControlSet001\services\LSI_SAS2\Parameters\PnpInterface] 5: [REG_DWORD_LE] 1,2,OS:test_data/SYSTEM,-,-,winreg/winreg_default,sha256_hash: 96dc1f1cc3c0b44ef9af72d1c18a8e6a4338c67988f303d05693ca4be6bf7eb9
07/14/2009,04:37:09,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet002\services\LSI_SAS2\Parameters] BusTyp...,[HKEY_LOCAL_MACHINE\System\ControlSet002\services\LSI_SAS2\Parameters] BusType: [REG_DWORD_LE] 10,2,OS:test_data/SYSTEM,-,-,winreg/winreg_default,sha256_hash: 96dc1f1cc3c0b44ef9af72d1c18a8e6a4338c67988f303d05693ca4be6bf7eb9
07/14/2009,04:37:09,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet002\services\LSI_SAS2\Parameters\PnpInte...,[HKEY_LOCAL_MACHINE\System\ControlSet002\services\LSI_SAS2\Parameters\PnpInterface] 5: [REG_DWORD_LE] 1,2,OS:test_data/SYSTEM,-,-,winreg/winreg_default,sha256_hash: 96dc1f1cc3c0b44ef9af72d1c18a8e6a4338c67988f303d05693ca4be6bf7eb9
07/14/2009,04:37:09,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet002\services\LSI_SAS\Parameters] BusType...,[HKEY_LOCAL_MACHINE\System\ControlSet002\services\LSI_SAS\Parameters] BusType: [REG_DWORD_LE] 10,2,OS:test_data/SYSTEM,-,-,winreg/winreg_default,sha256_hash: 96dc1f1cc3c0b44ef9af72d1c18a8e6a4338c67988f303d05693ca4be6bf7eb9

@MikeHofmann
Copy link
Author

@MikeHofmann log2timeline-20210429T074209.log.gz is the log file, what does you psort output tell you?

Attaching both logs (had to redact some path informations)

log2timeline-20210429T074209.log.redacted.gz
psort-20210429T074256.log.gz

@MikeHofmann can you provide the output of log2timeline.py --troubles

2021-04-29 09:00:57,708 [INFO] (MainProcess) PID:472 <data_location> Determined data location: /usr/share/plaso
Using Python version 3.8.5 (default, Jan 27 2021, 15:41:15)
[GCC 9.3.0]

Path: /usr/bin/log2timeline.py

plaso - log2timeline version 20210412

Checking availability and versions of dependencies.
[OK]            artifacts version: 20210404
[OK]            bencode
[OK]            certifi version: 2020.06.20
[OK]            chardet version: 3.0.4
[OK]            cryptography version: 3.3.1
[OK]            dateutil version: 2.8.1
[OK]            defusedxml version: 0.7.1
[OK]            dfdatetime version: 20200824
[OK]            dfvfs version: 20210213
[OK]            dfwinreg version: 20201006
[OK]            dtfabric version: 20200621
[OK]            elasticsearch version: 7.9.1
[OK]            future version: 0.18.2
[OK]            idna version: 2.9
[OK]            lz4 version: 3.1.3
[OK]            pefile version: 2019.4.18
[OK]            psutil version: 5.8.0
[OK]            pybde version: 20210327
[OK]            pycreg version: 20200725
[OK]            pyesedb version: 20200418
[OK]            pyevt version: 20200926
[OK]            pyevtx version: 20200709
[OK]            pyewf version: 20140811
[OK]            pyfsapfs version: 20201107
[OK]            pyfsext version: 20210129
[OK]            pyfshfs version: 20201104
[OK]            pyfsntfs version: 20201115
[OK]            pyfsxfs version: 20210403
[OK]            pyfvde version: 20191221
[OK]            pyfwnt version: 20200723
[OK]            pyfwsi version: 20201204
[OK]            pylnk version: 20200810
[OK]            pyluksde version: 20200205
[OK]            pymsiecf version: 20200710
[OK]            pyolecf version: 20201004
[OK]            pyparsing version: 2.4.7
[OK]            pyqcow version: 20201213
[OK]            pyregf version: 20201007
[OK]            pyscca version: 20200717
[OK]            pysigscan version: 20201117
[OK]            pysmdev version: 20201204
[OK]            pysmraw version: 20201210
[OK]            pytsk3 version: 20210130
[OK]            pytz
[OK]            pyvhdi version: 20201204
[OK]            pyvmdk version: 20200926
[OK]            pyvsgpt version: 20210207
[OK]            pyvshadow version: 20201222
[OK]            pyvslvm version: 20200817
[OK]            redis version: 3.5.3
[OK]            requests version: 2.24.0
[OK]            six version: 1.15.0
[OK]            urllib3 version: 1.25.9
[OK]            xlsxwriter version: 1.3.8
[OK]            yaml version: 5.3.1
[OK]            yara version: 4.0.5
[OK]            zmq version: 22.0.3

Also see: https://plaso.readthedocs.io/en/latest/sources/user/Troubleshooting.html

@MikeHofmann also can you try dynamic output instead of l2tcsv. The amount of information l2tcsv can present is limited also

noted. We use this for analyzing in Erics TimeLineExplorer and it's easier to quickly grep some information. But if we upload the .plaso file to timesketch the services is also not visible.

@joachimmetz
Copy link
Member

At first glance I do not see anything in Plaso that hints at an error.

  • Does regfinfo -H SYSTEM (part of libregf) show you ControlSet002\services key paths ?
  • What about regfexport -K '\ControlSet002\services' SYSTEM ?

@MikeHofmann
Copy link
Author

Hmmh, these tools seems to be older when i install them inside the log2timeline-container. They don't feature the -H or -K options.

apt-get install libregf-utils

regfinfo -V
regfinfo 20190303

Copyright (C) 2009-2019, Joachim Metz.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Report bugs to <[email protected]>.


regfexport -V
regfexport 20190303

Copyright (C) 2009-2019, Joachim Metz.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Report bugs to <[email protected]>.

When used without any cli-options on the Hive, i do get some errors on stderr:

regfinfo system

Unable to print file information.
libregf_key_item_read_named_key: unable to retrieve hive bin cell at offset: 16215544 (0x00f76df8).
libregf_key_item_read_node_data: unable to read named key at offset: 16215544 (0x00f76df8).
libfdata_tree_get_node_value: unable to read node at offset: 0x00f76df8.
libfdata_tree_node_get_node_value: unable to retrieve node value.
libregf_key_get_utf8_name_size: unable to retrieve key item.
info_handle_key_fprint: unable to retrieve key name size.
info_handle_key_fprint: unable to print sub key: 16 info.
info_handle_key_fprint: unable to print sub key: 42 info.
info_handle_key_fprint: unable to print sub key: 5 info.
info_handle_key_fprint: unable to print sub key: 0 info.
info_handle_key_fprint: unable to print sub key: 1 info.
info_handle_file_fprint: unable to print root key info.

I do see ControlSet002\Control but not ControlSet002\Services (ControlSet001\Services exists)

@joachimmetz
Copy link
Member

joachimmetz commented Apr 29, 2021 •
edited
Loading

Hmmh, these tools seems to be older when i install them inside the log2timeline-container.

Can you try libregf-tools (instead of any older versions libregf-utils from Ubuntu/Debian) from the GIFT PPA that should be enabled inside the container

@joachimmetz
Copy link
Member

This would indicate libregf is not able to find all the data it needs.

libregf_key_item_read_named_key: unable to retrieve hive bin cell at offset: 16215544 (0x00f76df8).
libregf_key_item_read_node_data: unable to read named key at offset: 16215544 (0x00f76df8).
libfdata_tree_get_node_value: unable to read node at offset: 0x00f76df8.
libfdata_tree_node_get_node_value: unable to retrieve node value.
libregf_key_get_utf8_name_size: unable to retrieve key item.
  • did log2timeline.py indicate any parsing warnings (pinfo.py should be able to give you that information)
  • is the entire SYSTEM file there? not that it did got truncated on copy or export?

@MikeHofmann
Copy link
Author

Can you try libregf-tools (instead of any older versions libregf-utils from Ubuntu/Debian) from the GIFT PPA that should be enabled inside the container

Nope, the PPA wasnt enabled, had to:

add-apt-repository ppa:gift/stable
apt-get install libregf-tools

Tools are newer, but still regfinfo missing the -H option. However regfexport now has -K.

regfinfo -h
regfinfo 20201007

Use regfinfo to determine information about a Windows NT
Registry File (REGF).

Usage: regfinfo [ -c codepage ] [ -hvV ] source

        source: the source file

        -c:     codepage of ASCII strings, options: ascii, windows-874,
                windows-932, windows-936, windows-949, windows-950,
                windows-1250, windows-1251, windows-1252 (default),
                windows-1253, windows-1254, windows-1255, windows-1256
                windows-1257 or windows-1258
        -h:     shows this help
        -v:     verbose output to stderr
        -V:     print version

regfexport -h
regfexport 20201007

Use regfexport to export information from a Windows NT
Registry File (REGF).

Usage: regfexport [ -c codepage ] [ -K key_path ] [ -l logfile ]
                  [ -hvV ] source

        source: the source file

        -c:     codepage of ASCII strings, options: ascii, windows-874,
                windows-932, windows-936, windows-949, windows-950,
                windows-1250, windows-1251, windows-1252 (default),
                windows-1253, windows-1254, windows-1255, windows-1256
                windows-1257 or windows-1258
        -h:     shows this help
        -K:     show information about a specific key path.
        -l:     logs information about the exported items
        -v:     verbose output to stderr
        -V:     print version

When using regfinfo i still get errors on stderr:

nable to print file information.
libregf_hive_bins_list_get_cell_at_offset: unable to retrieve hive bin cell at offset: 16215544 (0x00f76df8).
libregf_key_item_read_named_key: unable to retrieve hive bin cell at offset: 16215544 (0x00f76df8).
libregf_key_item_read: unable to read named key at offset: 16215544 (0x00f76df8).
libregf_key_initialize: unable to read key item at offset: 16215544 (0x00f76df8).
libregf_key_get_sub_key: unable to initialize sub key: 16.
info_handle_key_fprint: unable to retrieve sub key: 16.
info_handle_key_fprint: unable to print sub key: 42 info.
info_handle_key_fprint: unable to print sub key: 5 info.
info_handle_key_fprint: unable to print sub key: 0 info.
info_handle_key_fprint: unable to print sub key: 1 info.
info_handle_file_fprint: unable to print root key info.

but regfexport yields results:

regfexport -K '\ControlSet002\services' system | wc -l
44688

regfexport -K '\ControlSet002\services' system | grep corspdeft
Key path: \ControlSet002\services\Services\corspdeft
Key: corspdeft
Data: "C:\WINDOWS\system32\corspdeft.exe"
Data: corspdeft
Key path: \ControlSet002\services\Services\corspdeft\Security

While i do find the services\Services output troubling. The lowercase services shouldn't be there.

is the entire SYSTEM file there? not that it did got truncated on copy or export?

Yes, same problem exists when i process the entire EWF Image. I just exported the SYSTEM hive for simplicity. Also note that reged found the Keys as well as xways own registry viewer.

did log2timeline.py indicate any parsing warnings (pinfo.py should be able to give you that information)

Yes, one warning present:

           Message : pyregf_key_get_sub_key_by_index: unable to retrieve sub
                     key: 16. libregf_hive_bins_list_get_cell_at_offset: unable
                     to retrieve hive bin cell at offset: 16215544
                     (0x00f76df8). libregf_key_item_read_named_key: unable to
                     retrieve hive bin cell at offset: 16215544 (0x00f76df8).
                     libregf_key_item_read: unable to read named key at offset:
                     16215544 (0x00f76df8). libregf_key_initialize: unable to
                     read key item at offset: 16215544 (0x00f76df8).
                     libregf_key_get_sub_key: unable to initialize sub key: 16.
      Parser chain : winreg

@joachimmetz
Copy link
Member

Tools are newer, but still regfinfo missing the -H option.

Maybe something I added more recent than 20201007

While i do find the services\Services output troubling.

To determine if this is corruption or some format edge case

I assume you cannot share the file? If not could you compile the latest from source with verbose and debug output https://github.com/libyal/libregf/wiki/Troubleshooting#verbose-and-debug-output

regfexport -v -K '\ControlSet002\services' system 2>debug.log would still provide a lot of details, so double check it, but should likely be sufficient to pin point a possible root cause

The lowercase services shouldn't be there.

Don't know I would need to see the actual data

@MikeHofmann
Copy link
Author

MikeHofmann commented Apr 29, 2021 •
edited
Loading

I assume you cannot share the file?

No, this agency is not known for sharing. 😄

If not could you compile the latest from source with verbose and debug output

I quickly slapped a docker-container together and compiled (CPPCFLAGS=-g ./configure --enable-shared=no --enable-verbose-output --enable-debug-output) and used this on the hive. The debug.log is huge, probably more than github.com can handle. And i would need to filter out a lot of information (probably some grep -vE "^0" ?). But i think this wouldn't help, as regfexport is able to parse the hive without any problems:

Key path: \ControlSet002\services\Services\corspdeft
Name: corspdeft
Last written time: Jan 26, 2021 08:42:04.942000000 UTC

Value: 0 Type
Type: 32-bit integer little-endian (REG_DWORD_LITTLE_ENDIAN)
Data size: 4
Data: 16

Value: 1 Start
Type: 32-bit integer little-endian (REG_DWORD_LITTLE_ENDIAN)
Data size: 4
Data: 2

Value: 2 ErrorControl
Type: 32-bit integer little-endian (REG_DWORD_LITTLE_ENDIAN)
Data size: 4
Data: 0

Value: 3 ImagePath
Type: expandable string (REG_EXPAND_SZ)
Data size: 72
Data: "C:\WINDOWS\system32\corspdeft.exe"

Value: 4 DisplayName
Type: string (REG_SZ)
Data size: 20
Data: corspdeft

Value: 5 ObjectName
Type: string (REG_SZ)
Data size: 24
Data: LocalSystem

@joachimmetz
Copy link
Member

The debug.log is huge, probably more than github.com can handle.

Can you compress it with gzip and mail it to me ?

And i would need to filter out a lot of information (probably some grep -vE "^0"

any special chars in the key path ?

@joachimmetz
Copy link
Member

But i think this wouldn't help, as regfexport is able to parse the hive without any problems:

One thing that could be happening is that winreg stops parsing because the exception is raised. I'll have a look if I can make this more error resilient.

@MikeHofmann
Copy link
Author

Can you compress it with gzip and mail it to me ?
i just mailed you something.

Whats seems odd, that my compiled version doesn't complain nor does the exporter from ppa:gift/stable, it's only from the docker-container. Could be the build process for the container using some old libraries?

@joachimmetz
Copy link
Member

joachimmetz commented Apr 29, 2021 •
edited
Loading

The version detected in the --troubles output is:

[OK]            pyregf version: 20201007

which was the latest version when the container was released

@joachimmetz joachimmetz removed the pending reporter input Issue is pending input from the reporter label Apr 29, 2021
@joachimmetz
Copy link
Member

i just mailed you something.

thx, received will have a look later

Also made some changes to make winreg more resilient to corrupt files https://github.com/log2timeline/plaso/pull/3572/files

@MikeHofmann
Copy link
Author

Also made some changes to make winreg more resilient to corrupt files https://github.com/log2timeline/plaso/pull/3572/files

Tested with this change, this fixes the issue. Thx. Any idea when this will reach the container on hub.docker.com?

@MikeHofmann
Copy link
Author

Was a little too hasty here. CurrentControlSet should be a link to the ControlSet as indicated in HKLM\System\Select\Current. The timeline includes some services under the CurrentControlSet Keys, but i can't figure out from which ControlSet.

The SYSTEM hive has three ControlSets (ControlSet001, ControlSet002 and ControlSet004). Each having a different services configuration. For example ControlSet001 has a unique service tifsfilter that the other two don't have. ControlSet002 has corspdeft and ControlSet004 has 157786025 which the respective other ControlSets don't have:

grep -E "(tifsfilter|corspdeft|157786025)" system.csv 
05/14/2009,06:57:13,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet001\Services\tifsfilter\Security] Securi...,[HKEY_LOCAL_MACHINE\System\ControlSet001\Services\tifsfilter\Security] Security: [REG_BINARY] (168 bytes),2,OS:/export/system,-,-,winreg/winreg_default,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
02/26/2010,15:44:28,UTC,M...,REG,Registry Key - Service,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet001\Services\tifsfilter] Type: File Syst...,[HKEY_LOCAL_MACHINE\System\ControlSet001\Services\tifsfilter] Type: File System Driver (0x2) Start: Auto Start (2) Image path: system32\DRIVERS\tifsfilt.sys Error control: Normal (1) DisplayName: [REG_SZ] Acronis True Image FS Filter Group: [REG_SZ] Filter Tag: [REG_DWORD_LE] 7,2,OS:/export/system,-,-,winreg/windows_services,name: tifsfilter; sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
02/26/2010,15:44:43,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet001\Enum\Root\LEGACY_TIFSFILTER\0000] Cl...,[HKEY_LOCAL_MACHINE\System\ControlSet001\Enum\Root\LEGACY_TIFSFILTER\0000] Class: [REG_SZ] LegacyDriver ClassGUID: [REG_SZ] {8ECC055D-047F-11D1-A537-0000F8753ED1} ConfigFlags: [REG_DWORD_LE] 0 DeviceDesc: [REG_SZ] Acronis True Image FS Filter Legacy: [REG_DWORD_LE] 1 Service: [REG_SZ] tifsfilter,2,OS:/export/system,-,-,winreg/winreg_default,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
03/02/2011,14:03:42,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet002\Enum\Root\LEGACY_TIFSFILTER\0000] Cl...,[HKEY_LOCAL_MACHINE\System\ControlSet002\Enum\Root\LEGACY_TIFSFILTER\0000] Class: [REG_SZ] LegacyDriver ClassGUID: [REG_SZ] {8ECC055D-047F-11D1-A537-0000F8753ED1} ConfigFlags: [REG_DWORD_LE] 0 DeviceDesc: [REG_SZ] Acronis True Image FS Filter Legacy: [REG_DWORD_LE] 1 Service: [REG_SZ] tifsfilter,2,OS:/export/system,-,-,winreg/winreg_default,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
03/02/2011,14:03:42,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet004\Enum\Root\LEGACY_TIFSFILTER\0000] Cl...,[HKEY_LOCAL_MACHINE\System\ControlSet004\Enum\Root\LEGACY_TIFSFILTER\0000] Class: [REG_SZ] LegacyDriver ClassGUID: [REG_SZ] {8ECC055D-047F-11D1-A537-0000F8753ED1} ConfigFlags: [REG_DWORD_LE] 0 DeviceDesc: [REG_SZ] Acronis True Image FS Filter Legacy: [REG_DWORD_LE] 1 Service: [REG_SZ] tifsfilter,2,OS:/export/system,-,-,winreg/winreg_default,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
12/02/2019,10:12:00,UTC,....,REG,AppCompatCache Registry Entry,File Last Modification Time,-,-,Path: \??\C:\WINDOWS\157786025.exe,[HKEY_LOCAL_MACHINE\System\ControlSet004\Control\Session Manager\AppCompatibility] Cached entry: 76 Path: \??\C:\WINDOWS\157786025.exe,2,OS:/export/system,-,-,winreg/appcompatcache,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
12/02/2019,11:29:14,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet004\Services\157786025\Security] Securit...,[HKEY_LOCAL_MACHINE\System\ControlSet004\Services\157786025\Security] Security: [REG_BINARY] (168 bytes),2,OS:/export/system,-,-,winreg/winreg_default,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
12/02/2019,11:29:15,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet004\Enum\Root\LEGACY_157786025] NextInst...,[HKEY_LOCAL_MACHINE\System\ControlSet004\Enum\Root\LEGACY_157786025] NextInstance: [REG_DWORD_LE] 1,2,OS:/export/system,-,-,winreg/winreg_default,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
12/02/2019,11:29:15,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet004\Enum\Root\LEGACY_157786025\0000] Cla...,[HKEY_LOCAL_MACHINE\System\ControlSet004\Enum\Root\LEGACY_157786025\0000] Class: [REG_SZ] LegacyDriver ClassGUID: [REG_SZ] {8ECC055D-047F-11D1-A537-0000F8753ED1} ConfigFlags: [REG_DWORD_LE] 0 DeviceDesc: [REG_SZ] 157786025 Legacy: [REG_DWORD_LE] 1 Service: [REG_SZ] 157786025,2,OS:/export/system,-,-,winreg/winreg_default,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
04/22/2020,09:27:38,UTC,.A..,REG,AppCompatCache Registry Entry,Last Time Executed,-,-,Path: \??\C:\WINDOWS\157786025.exe,[HKEY_LOCAL_MACHINE\System\ControlSet004\Control\Session Manager\AppCompatibility] Cached entry: 76 Path: \??\C:\WINDOWS\157786025.exe,2,OS:/export/system,-,-,winreg/appcompatcache,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
06/06/2020,16:00:33,UTC,M...,REG,Registry Key - Service,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet004\Services\157786025] Type: Service - ...,[HKEY_LOCAL_MACHINE\System\ControlSet004\Services\157786025] Type: Service - Own Process (0x10) Start: Auto Start (2) Image path: %SystemRoot%\157786025.exe Error control: Ignore (0),2,OS:/export/system,-,-,winreg/windows_services,name: 157786025; object_name: LocalSystem; sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
01/26/2021,08:41:56,UTC,....,REG,AppCompatCache Registry Entry,File Last Modification Time,-,-,Path: \??\C:\WINDOWS\system32\corspdeft.exe,[HKEY_LOCAL_MACHINE\System\ControlSet002\Control\Session Manager\AppCompatibility] Cached entry: 46 Path: \??\C:\WINDOWS\system32\corspdeft.exe,2,OS:/export/system,-,-,winreg/appcompatcache,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
01/26/2021,08:42:04,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet002\Services\corspdeft\Security] Securit...,[HKEY_LOCAL_MACHINE\System\ControlSet002\Services\corspdeft\Security] Security: [REG_BINARY] (168 bytes),2,OS:/export/system,-,-,winreg/winreg_default,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
01/26/2021,08:42:04,UTC,M...,REG,Registry Key,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet002\Enum\Root\LEGACY_CORSPDEFT\0000] Cla...,[HKEY_LOCAL_MACHINE\System\ControlSet002\Enum\Root\LEGACY_CORSPDEFT\0000] Class: [REG_SZ] LegacyDriver ClassGUID: [REG_SZ] {8ECC055D-047F-11D1-A537-0000F8753ED1} ConfigFlags: [REG_DWORD_LE] 0 DeviceDesc: [REG_SZ] corspdeft Legacy: [REG_DWORD_LE] 1 Service: [REG_SZ] corspdeft,2,OS:/export/system,-,-,winreg/winreg_default,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
01/26/2021,08:42:04,UTC,M...,REG,Registry Key - Service,Content Modification Time,-,-,[HKEY_LOCAL_MACHINE\System\ControlSet002\Services\corspdeft] Type: Service - ...,[HKEY_LOCAL_MACHINE\System\ControlSet002\Services\corspdeft] Type: Service - Own Process (0x10) Start: Auto Start (2) Image path: "C:\WINDOWS\system32\corspdeft.exe" Error control: Ignore (0) DisplayName: [REG_SZ] corspdeft,2,OS:/export/system,-,-,winreg/windows_services,name: corspdeft; object_name: LocalSystem; sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae
01/26/2021,08:42:04,UTC,.A..,REG,AppCompatCache Registry Entry,Last Time Executed,-,-,Path: \??\C:\WINDOWS\system32\corspdeft.exe,[HKEY_LOCAL_MACHINE\System\ControlSet002\Control\Session Manager\AppCompatibility] Cached entry: 46 Path: \??\C:\WINDOWS\system32\corspdeft.exe,2,OS:/export/system,-,-,winreg/appcompatcache,sha256_hash: 9b0a0974e0c269b543290efef292cda700bc96b203b8309f24afd07d0cda42ae

But CurrentControlSet doesn't have any of these services. It has another set of services, which doesn't match any ControlSet.

@MikeHofmann
Copy link
Author

MikeHofmann commented Apr 30, 2021 •
edited
Loading

Oh and pinfo.py now gives multiple warnings when having the fix applied:

********************************** Warning: 0 **********************************
           Message : unable to process path specification with error: name
                     'subkey_index' is not defined
      Parser chain : 
Path specification : type: OS, location: /export/system
--------------------------------------------------------------------------------

********************************** Warning: 1 **********************************
           Message : unable to process path specification with error: name
                     'subkey_index' is not defined
      Parser chain : 
Path specification : type: OS, location: /export/system
--------------------------------------------------------------------------------

********************************** Warning: 2 **********************************
           Message : unable to process path specification with error: name
                     'subkey_index' is not defined
      Parser chain : 
Path specification : type: OS, location: /export/system
--------------------------------------------------------------------------------

********************************** Warning: 3 **********************************
           Message : in key:
                     HKEY_LOCAL_MACHINE\System\ControlSet002\Control\Class\{71A27CDD-812A-11D0-BEC7-08002BE2092F}
                     error: pyregf_key_get_sub_key_by_index: unable to retrieve
                     sub key: 16. libregf_hive_bins_list_get_cell_at_offset:
                     unable to retrieve hive bin cell at offset: 16215544
                     (0x00f76df8). libregf_key_item_read_named_key: unable to
                     retrieve hive bin cell at offset: 16215544 (0x00f76df8).
                     libregf_key_item_read: unable to read named key at offset:
                     16215544 (0x00f76df8). libregf_key_initialize: unable to
                     read key item at offset: 16215544 (0x00f76df8).
                     libregf_key_get_sub_key: unable to initialize sub key: 16.
      Parser chain : winreg
Path specification : type: OS, location: /export/system
--------------------------------------------------------------------------------

@joachimmetz
Copy link
Member

Any idea when this will reach the container on hub.docker.com?

In the next release (https://github.com/log2timeline/plaso/milestone/33), end of June likely

The timeline includes some services under the CurrentControlSet Keys, but i can't figure out from which ControlSet.

Can you provide an example entry?

But CurrentControlSet doesn't have any of these services. It has another set of services, which doesn't match any ControlSet.

CurrentControlSet of what? How are you looking at CurrentControlSet?

@joachimmetz joachimmetz changed the title Not seeing services under ControlSet002 from Windows XP SYSTEM hive in l2tcsv output/log2timeline log file? winreg bails out on single corrupt key - please make it more resilient Apr 30, 2021
@joachimmetz joachimmetz added enhancement New or improved functionality parsers Issues related to parsers and parser plug-ins labels Apr 30, 2021
@MikeHofmann
Copy link
Author

CurrentControlSet of what? How are you looking at CurrentControlSet?

Uff, i think i need a break. I was operating under the assumption, that log2timeline includes the virtual CurrentControlSet into the timeline as well. And i couldn't find the services i was looking for by grepping in the psorted .csv, while at the same time looking at the file using less and because of that my mind mixed a few columns on screen.

TLDR: ControlSet[0-9]{3} is included in the timeline, CurrentControlSet is not. As it should be.

I think this can be closed now. Do you keep this open until everything is merged or can i close it?

@joachimmetz
Copy link
Member

Keep it open for now, is a reminder for me to add test coverage for a corrupted Windows NT Registry file.

@joachimmetz joachimmetz added this to the 2021 June release milestone May 1, 2021
@joachimmetz joachimmetz changed the title winreg bails out on single corrupt key - please make it more resilient winreg parser bails out on single corrupt key - please make it more resilient May 1, 2021
@MikeHofmann
Copy link
Author

Just some statistical ramblings, no action needed:

I was wondering, on how often our images would run into a error: pyregf_key_get_sub_key_by_index: unable to retrieve. So i took the time and tested a few images. All from different sources (different organization/person, different examiner, different imager-tool, different hardware, etc.). I had the hopes in finding some common reason, why the registry is damaged. I'm afraid this looks totally random to me.

ImageID CurrentBuildNr ProductName regf Warnings present
35 17763 Windows 10 Pro no
45 18363 Windows 10 Enterprise no
46 14393 Windows Server 2016 Standard no
47 9600 Windows Server 2012 R2 Standard no
48 7601 Windows 7 Ultimate no
49 7601 Windows 7 Professional no
50 9600 Windows Server 2012 R2 Standard no
51 17763 Windows 10 Pro yes
52 2600 Microsoft Windows XP yes
53 6003 Windows Server 2008 Enterprise yes
54 14393 Windows Server 2016 Standard no
55 9600 Windows Server 2012 R2 Standard no
56 7601 Windows Server 2008 R2 Standard no
57 7601 Windows 7 Enterprise no
59 9600 Windows Server 2012 R2 Datacenter no
60 7601 Windows 7 Home Premium no
61 17134 Windows 10 Pro no

@joachimmetz
Copy link
Member

All from different sources (different organization/person, different examiner, different imager-tool, different hardware, etc.).

  • were these all offline acquisitions ? or all some live acquisitions ?
  • do you know if these systems were cleanly shutdown or some also pulled the plug ?

@MikeHofmann
Copy link
Author

were these all offline acquisitions ? or all some live acquisitions ?
do you know if these systems were cleanly shutdown or some also pulled the plug ?

Unfortunately these images are below average quality from a forensics standpoint. From the three images above: one was done with FTK Imager, another one with Logicube Falcon and the last just a .vmdk. No written aquisition reports, sometimes a log file from the aquisition/imager tool, no name of examiner, etc. Probably need to parse bootstat.dat (a very nice to have feature for l2t btw) to check for a clean shutdown.

@joachimmetz joachimmetz added the close after review Issue as outstanding pull request(s) and can be closed once these have been reviewed and merged label May 8, 2021
@joachimmetz
Copy link
Member

Probably need to parse bootstat.dat (a very nice to have feature for l2t btw) to check for a clean shutdown.

#3578

@joachimmetz
Copy link
Member

Test file added #3576, closing issue

@joachimmetz joachimmetz removed the close after review Issue as outstanding pull request(s) and can be closed once these have been reviewed and merged label May 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@joachimmetz joachimmetz

Labels
enhancement New or improved functionality parsers Issues related to parsers and parser plug-ins
Projects
None yet
Milestone
2021 June release
Development

No branches or pull requests
2 participants
@joachimmetz @MikeHofmann