Skip to content
This repository has been archived by the owner on Jul 25, 2024. It is now read-only.

Device scanner failure with ZFS 0.8.1 #87

Open
utopiabound opened this issue Sep 11, 2019 · 7 comments
Open

Device scanner failure with ZFS 0.8.1 #87

utopiabound opened this issue Sep 11, 2019 · 7 comments

Comments

@utopiabound
Copy link

Device scanner has errors when trying to create filesystem on existing pools with ZFS 0.8.1

Sep 11 14:39:26 ieel-mds01.ieel.utopiabound.net device-scanner-daemon[593]: [INFO  device_scanner_daemon::reducers::zed] Processing Pool command: SetZfsProp(Guid("0x22B783120AE5BED5"), Name("MGS/MGS"), Key("canmount"), Value("0"))
Sep 11 14:39:26 ieel-mds01.ieel.utopiabound.net device-scanner-daemon[593]: [ERROR device_scanner_daemon] Unhandled Error: LibZfsError(ZfsNotFound("MGS/MGS"))
Sep 11 14:39:26 ieel-mds01.ieel.utopiabound.net device-scanner-daemon[593]: [ERROR device_scanner_daemon] Unhandled Error: SendError(SendError("..."))
@jgrund
Copy link
Member

jgrund commented Sep 11, 2019

When we get a create event for a dataset, we look it up in libzfs

https://github.com/whamcloud/device-scanner/blob/acf2147f689c97c4bb54f427256b4ed6c3619c99/zed-enhancer/src/lib.rs#L80

If we’re not seeing the dataset for that next prop it means there is either an issue with libzfs for that call, or libzfs doesn’t know about that dataset until some point afterwards.

@utopiabound
Copy link
Author

sosreport-ieel-mds01-292-2019-09-11-seypwye.tar.gz

@utopiabound
Copy link
Author

utopiabound commented Sep 11, 2019 •
edited
Loading

Further up in same log file:

Sep 11 14:39:26 ieel-mds01.ieel.utopiabound.net zed-enhancer[941]: thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/libcore/option.rs:347:21
Sep 11 14:39:26 ieel-mds01.ieel.utopiabound.net zed-enhancer[941]: note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

@utopiabound
Copy link
Author

ZFS Events:
zfs-events-v.txt

@jgrund
Copy link
Member

jgrund commented Sep 11, 2019

This event is causing the panic:

Sep 11 2019 14:39:25.951875691 sysevent.fs.zfs.history_event
        version = 0x0
        class = "sysevent.fs.zfs.history_event"
        pool = "MGS"
        pool_guid = 0x22b783120ae5bed5
        pool_state = 0x0
        pool_context = 0x0
        history_hostname = "ieel-mds01.ieel.utopiabound.net"
        history_dsname = "MGS/MGS"
        history_internal_str = ""
        history_internal_name = "create"
        history_dsid = 0x76
        history_txg = 0x76f03
        history_time = 0x5d793f5d
        time = 0x5d793f5d 0x38bc786b
        eid = 0x9

@utopiabound
Copy link
Author

Hopefully useful traceback of zed-enhancer with RUST_BACKTRACE=1

Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed[3085]: eid=1 class=history_event pool_guid=0x22B783120AE5BED5
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed[3105]: eid=2 class=config_sync pool_guid=0x22B783120AE5BED5
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed[3111]: eid=3 class=pool_import pool_guid=0x22B783120AE5BED5
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed[3121]: eid=4 class=history_event pool_guid=0x22B783120AE5BED5
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed[3126]: eid=5 class=config_sync pool_guid=0x22B783120AE5BED5
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/libcore/option.rs:347:21
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: stack backtrace:
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 0: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 1: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 2: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 3: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 4: std::panicking::rust_panic_with_hook
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 5: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 6: rust_begin_unwind
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 7: core::panicking::panic_fmt
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 8: core::panicking::panic
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 9: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 10: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 11: libzfs::zfs::Zfs::props
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 12: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 13: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 14: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 15: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 16: libzfs::state::get_pool_by_name_and_guid
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 17: zed_enhancer::handle_zed_commands
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 18: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 19: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 20: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 21: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 22: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 23: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 24: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 25: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 26: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 27: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 28: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 29: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 30: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 31: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 32: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 33: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 34: __rust_maybe_catch_panic
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 35: std::rt::lang_start_internal
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 36: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 37: __libc_start_main
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[965]: 38: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net systemd[1]: zed-enhancer.service: main process exited, code=exited, status=101/n/a
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net systemd[1]: Unit zed-enhancer.service entered failed state.
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net systemd[1]: Triggering OnFailure= dependencies of zed-enhancer.service.
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net systemd[1]: zed-enhancer.service failed.
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net systemd[1]: Starting IML ZED Populator...
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net bash[3130]: P: /module/zfs
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net bash[3130]: E: DEVPATH=/module/zfs
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net bash[3130]: E: SUBSYSTEM=module
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net bash[3130]: E: SYSTEMD_WANTS=zed-enhancer.socket
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net bash[3130]: E: TAGS=:systemd:
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net bash[3130]: E: USEC_INITIALIZED=48967
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net systemd[1]: zed-enhancer.service holdoff time over, scheduling restart.
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net systemd[1]: Stopped IML ZED Enhancer Daemon.
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net systemd[1]: Started IML ZED Enhancer Daemon.
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: [INFO  zed_enhancer] Server starting
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/libcore/option.rs:347:21
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: stack backtrace:
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 0: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 1: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 2: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 3: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 4: std::panicking::rust_panic_with_hook
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 5: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 6: rust_begin_unwind
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 7: core::panicking::panic_fmt
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 8: core::panicking::panic
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 9: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 10: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 11: libzfs::zfs::Zfs::props
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 12: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 13: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 14: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 15: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 16: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 17: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 18: libzfs::state::get_imported_pools
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 19: zed_enhancer::handle_zed_commands
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 20: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 21: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 22: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 23: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 24: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 25: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 26: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 27: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 28: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 29: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 30: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 31: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 32: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 33: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 34: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 35: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 36: __rust_maybe_catch_panic
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 37: std::rt::lang_start_internal
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 38: <unknown>
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 39: __libc_start_main
Sep 11 17:13:58 ieel-mds01.ieel.utopiabound.net zed-enhancer[3134]: 40: <unknown>

@jgrund jgrund transferred this issue from another repository Sep 12, 2019
@jgrund
Copy link
Member

jgrund commented Sep 12, 2019

This is due to new properties being introduced in ZFS 0.8. #86 Should fix this.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jgrund @utopiabound