forked from zeek/zeek
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathNEWS
3568 lines (2697 loc) · 146 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
This document summarizes the most important changes in the current Zeek
release. For an exhaustive list of changes, see the ``CHANGES`` file
(note that submodules, such as Broker, come with their own ``CHANGES``.)
Zeek 4.0.0
==========
New Functionality
-----------------
- Added support for EDNS0 Cookie and Keep-Alive options.
- Added new Packet Analysis plugin architecture for parsing packet headers
at layers below the existing Session analysis plugins. This allows
writing plugins to parse the various parts of a packet header separately,
chaining down into other plugins as needed.
- Add ``dce_rpc_request_stub`` and ``dce_rpc_response_stub`` events for
accessing the contents of DCE-RPC request/response stub data.
- Add support for log filter policy hooks, which supersede the current
log predicates. The hook signature is as follows:
hook(rec: any, id: Log::ID, filter: Log::Filter);
The logging manager invokes hooks on each log record. Hooks can veto
log records via a break, and modify them if necessary. Log filters
inherit the stream-level hook, but can override or remove the hook
as needed. The distribution's existing log streams now come with
pre-defined hooks that users can add handlers to. The existing
predicates are deprecated for removal in 4.1 but continue to work.
See https://docs.zeek.org/en/master/frameworks/logging.html#filter-log-records
for more details.
- Added a ``udp-state`` signature condition to enforce matching against
either "originator" or "responder" flow direction of UDP packets.
- Improvements to capture-loss.zeek:
- A new option, ``CaptureLoss::initial_watch_interval``. When restarting a
Zeek cluster, one usually wants some immediate feedback as to the health of
the monitoring via capture loss. However, you previously needed to wait a
full ``CaptureLoss::watch_interval``, which defaults to 15 minutes. The
new option specifies the interval for the first-time report. So the new
default behavior provides stats after 1 minute and then after
15 minutes afterward.
- A new notice type, ``CaptureLoss::Too_Little_Traffic``.
If a Zeek process sees less than ``CaptureLoss::minimum_acks`` ACKs in a
given interval, this notice gets raised. This can be a useful diagnostic
if, for whatever reason, a Zeek process stops seeing traffic, but
capture-loss.zeek would have previously only reported that "0 gaps and 0
ACKs is 0% loss".
- A new ``zeek_script_args`` variable contains a list of arguments passed
to a script. E.g. either when explicitly executing Zeek like
``zeek -- myscript.zeek -arg1 -arg2``, or when using Zeek to interpret
executable scripts that contain a hashbang line at the top like::
#!/usr/local/zeek/bin/zeek --
- Added a new ``generate_all_events`` bif, which can be used to always raise
events, even when they are not used by scripts. This can be used by the
``dump-events.zeek`` script to log all events that happen; the script
got a new option to enable this behavior.
Changed Functionality
---------------------
- ``NetControl::DROP`` had 3 conflicting definitions that could potentially
be used incorrectly without any warnings or type-checking errors.
Such enum redefinition conflicts are now caught and treated as errors,
so the ``NetControl::DROP`` enums had to be renamed:
- The use as enum of type ``Log::ID`` is renamed to ``NetControl::DROP_LOG``
- The use as enum of type ``NetControl::CatchReleaseInfo`` is renamed to
``NetControl::DROP_REQUESTED``
- The use as enum of type ``NetControl::RuleType`` is unchanged and still
named ``NetControl::DROP``
- The extract_email_addrs_vec() BIF now returns all occurrences of emails,
including duplicates, with preserved order of occurrence. This seems like
the original/documented intent of the function, but the previous
implementation did not preserve ordering or duplicates.
- The Dictionary implementation is replaced (no API changes). The new version
uses clustered hashing, a variation of Robinhood / Open Addressing hashing.
This implementation generally performs better and utilizes less memory
than the previous one. A detailed explanation of the implementation is here:
https://jasonlue.github.io/algo/2019/08/20/clustered-hashing.html
- The ``p`` fields of ``Cluster::Node`` records now use a
``&default=0/unknown`` attribute with ``0/unknown`` meaning that the node is
not pre-configured to listen for incoming connections from other cluster
nodes.
- The ``|x|`` operator, where ``x`` is an expression with an integral result,
no longer performs an implicit coercion of that result into a signed
``int`` type. This was actually the behavior before Zeek 3.0 as well, but
the attempt to prevent mistakes that easily result from integer literals in
Zeek being unsigned like ``|5 - 9|`` causing an overflow/wraparound and
yielding a very large number is not generally consistent since overflows
are still generally able to happen in other ways and also in other contexts
besides just the absolute-value operator. So the preference was to revert
to a behavior that favors consistency. For reference, see
https://github.com/zeek/zeek/pull/251#issuecomment-713956976
Removed Functionality
---------------------
- The counter type was removed. This type was never fully functional/used
anywhere.
- Removed the PRI_PTR_COMPAT_INT, PRI_PTR_COMPAT_UINT, and PRI_SOURCE_ID
macros. There are no deprecation warnings for these because they were C
macros. Use the PRIdPTR and PRIuPTR macros from the standard library
instead.
- The ``successful_connection_remove`` and ``connection_successful`` events
as well as ``connection$successful`` field that were added in Zeek v3.1.0 are
now removed. They were found to be unintuitive along with having unresolved
corner cases. The original goal/intent to avoid the overhead and scalability
issues with every new protocol-analysis adding a new
``connection_state_remove`` handler can now be resolved with a less-confusing
approach: see the ``Conn::register_removal_hook`` function.
Deprecated Functionality
------------------------
- Marked the Continuation.h and PacketDumper.h files as deprecated. The code
contained within them is unused by Zeek.
Zeek 3.2.0
==========
New Functionality
-----------------
- X509 Certificate caching:
Zeek now caches certificates if they have (by default) been encountered
more than 10 times in 62 seconds. Information for cached certificates is
retained; if the certificate is encountered again it does not have to
be re-parsed and already existing information is used to raise the events.
This should especially help with performance in environments where the
same certificates are seen very often.
Certificate caching is very configureable; it is possible to disable the
feature, change the time intervals or even suppress X509 events.
For details see ``scripts/base/files/x509/main.zeek``.
- Add parsing support for Remote Desktop Protocol UDP Transport Extension
(RDPEUDP versions 1 and 2). This primarily only adds "rdpeudp" to
connection record service fields when an RDPEUDP session handhake is
detected, but also provides a few other events related to the RDPEUDP
connection establishment.
- Add the ``udp_content_ports`` configuration option. Any port added to
this set will cause the ``udp_contents`` event to be raised.
- Add the ``udp_content_delivery_ports_use_resp`` option which can be used
to specify how the destination port for the ``udp_content_delivery_ports_orig``
and ``udp_content_delivery_ports_orig`` options is determined. The current value
keeps behavior as it was in previous versions of Zeek.
- Add a file signature to identify ISO9660 disk images (application/x-iso9660-image)
- Add file signature to identify Python bytecode (application/x-python-bytecode)
- Events and hooks are now allowed to have multiple, alternate prototype
declarations. This allows for extending event/hook parameters in a way that
won't break an existing user's handlers and also allows users to define their
own custom event/hook prototypes that consume a subset of the parameters
(convenience of typing/memory/etc). This feature is documented in detail
here: https://docs.zeek.org/en/current/script-reference/types.html#type-event
- Add ``flags`` parameters to ``rdp_connect_request``,
``rdp_negotiation_response``, and ``rdp_negotiation_failure`` events.
- ``Reporter::conn_weird`` now correctly handles weirds for expired connections,
for which no connection state information is available in the core anymore. These
cases will raise the new ``expired_conn_weird`` event.
- Broker Store table synchronization (experimental).
Zeek now supports synchronizing tables/sets across clusters using a backing Broker
store. The same feature also allows persistent storage of data in tables/sets
over Zeek restarts. This feature is implemented using the new ``&backend`` attribute.
To synchronize a table over a cluster, you can, e.g., use:
global t: table[string] of count &backend=Broker::MEMORY;
This feature is documented in detail here:
https://docs.zeek.org/en/current/frameworks/broker.html#broker-store-backed-zeek-tables-for-data-synchronization-and-persistence
Note: this feature is experimental and the syntax/specifics can change in the future.
Changed Functionality
---------------------
- Several C++ functions have been changed to pass smart pointers
(``class IntrusivePtr<>``) instead of raw pointers. This makes the
code more robust. External plugins may need to be updated to this
API change.
- BIFs that use @ARG@, @ARGS@, or @ARGC@ may break since their type has
changed: BIF arguments are now passed as a ``std::vector<IntrusivePtr<Val>>``
rather than a ``val_list`` (i.e. ``List<Val*>``).
- RocksDB support for Broker data stores is now opt-in instead of automatically
detected and used at configuration-time. Use the ``--enable-rocksdb`` and
``--with-rocksdb=`` flags to opt-in.
- At the C++ level, a large number of functions had their return
values and/or arguments changed to use ``bool`` types instead of
``int``. This includes some virtual methods, which may cause build
failures in plugins that were overriding those methods. Those
plugins will need to be updated to match these API changes. To make
sure to catch changes you need to make to your plugin, double-check
that all virtual method implementations use "override".
- Many C++ classes were marked "final" which also has some performance benefits
due to devirtualization optimizations.
- Data members of many C++ classes/structs were reordered to achieve better
packing and smaller memory footprint.
- "using namespace std" was removed from the Zeek header files; Zeek now always
explicitly specifies std when using STL functionality in headers. This may
necessitate small changes in external plugins, if they relied on the using
statement in Zeek headers.
- The ``connection_external`` event was removed. This functionality that could
raise this event (injecting connections via broccoli) was removed a while ago;
the event handler served no purpose anymore.
- Reorganize the file signatures to break them out into more groups. This may
break scripts that had been explicitly loading any signature files that moved.
- The DCE/RPC operation string of "NetrLogonSamLogonWithFlags" has been
corrected from "NetrLogonSameLogonWithFlags".
- ``AsRecord()`` and ``AsNonConstRecord()`` have changed to return
``std::vector<IntrusivePtr<Val>>*``.
- ``AsVector()`` has changed to return ``std::vector<IntrusivePtr<Val>>*``.
- Moved a large number of classes from the global namespace into either the
``zeek`` or ``zeek::detail`` namespace. See https://github.com/zeek/zeek/issues/266
for the rationale behind these changes. Most types that were moved and functions
that used them have been marked as deprecated and will generate compiler
warnings if used (a few exceptions will not generate compiler warnings,
but the Deprecated Functionality section below will mention those
ones specifically).
This includes a number of renames to classes, removing 'Bro' from their
names. Deprecation warnings should notify plugin developers of these
name changes.
Removed Functionality
---------------------
- The fmt() function which takes a va_list argument is replaced, use
the new vfmt() function for equivalent functionality. The former is
deprecated because overloading it with the variadic fmt() function
can cause the unintended overload to be chosen depending on how the
platform implements va_list.
Deprecated Functionality
------------------------
- The ``plugin::Plugin::HookCallFunction()`` method is deprecated. Note
that compilers will not emit a deprecation warning, but the replacement
method to now use is called ``HookFunctionCall`` and uses ``IntrusivePtr``
arguments and return value.
- The ``Func::Call(val_list*, ...)`` method is now deprecated. Use ``Invoke()``
instead which takes a ``zeek::Args`` (``std::vector<IntrusivePtr<Val>>``).
There's also a variadic template for ``Invoke()`` that forwards all arguments
into a ``zeek::Args`` for you.
- The ``EventMgr::QueueEvent()`` and EventMgr::QueueEventFast()`` methods
are now deprecated, use ``EventMgr::Enqueue()`` instead.
- The ``Connection::ConnectionEvent()``, ``Connection::Event()``, and
``Connection::ConnectionEventFast()`` methods are now deprecated, use
``Connection::EnqueueEvent()`` instead.
- The ``file_analysis::File::FileEvent()`` methods taking ``val_list``
arguments are now deprecated, use the overload that takes a ``zeek::Args``
instead.
- The ``analyzer::Analyzer::ConnectionEvent()``, ``analyzer::Analyzer::Event``,
and ``analyzer::Analyzer::ConectionEventFast()`` methods are deprecated, use
``analyzer::Analyzer::EnqueueConnEvent()`` instead.
- All ``val_mgr`` methods starting with "Get" are deprecated, use the new
``val_mgr`` methods that return ``IntrusivePtr``.
- ``Connection::BuildConnVal()`` is deprecated, use ``Connection::ConnVal()``.
- ``Analyzer::BuildConnVal()`` is deprecated, use ``Analyzer::ConnVal()``.
- ``BifEvent::generate_`` functions are deprecated, use ``zeek::BifEvent::enqueue_``.
- ``binpac::bytestring_to_val()`` is deprecated, use ``binpac::to_stringval()``.
- ``binpac::string_to_val()`` is deprecated, use ``StringVal`` constructor.
- Returning ``Val*`` from BIFs is deprecated, return ``IntrusivePtr`` instead.
- Various methods of converting protocol structures, like IP or packet headers,
to associated ``Val`` type are now deprecated, the deprecation warning
message will advise what new method to use instead.
- Various methods of ``Tag`` classes are deprecated with the warning
message advising what new method to use instead.
- The ``utf16_bytestring_to_utf8_val()`` function is deprecated, use
``utf16_to_utf8_val()`` instead.
- ``RecordType::FieldType()`` is deprecated, use ``RecordType::GetFieldType()``
- ``BroType::HasField()`` and ``BroType::FieldType()`` are deprecated, use
the methods of ``RecordType`` directly.
- ``BroType::YieldType()`` is deprecated, use ``BroType::Yield()``.
- ``ID::AsType()`` is deprecated, use ``ID::IsType()`` and ``ID::GetType()``.
- ``ID::Type()`` is deprecated, use ``ID::GetType()``.
- ``ID::ID_Val()`` is deprecated, use ``ID::GetVal()``.
- ``internal_type()`` is deprecated, use ``zeek::id::find_type()``.
- ``internal_val()`` and ``internal_const_val()`` are deprecated, use
``zeek::id::find_val()`` or ``zeek::id::find_const()``.
- ``internal_func()`` is deprecated, use ``zeek::id::find_func()``.
- ``opt_internal_val()`` is deprecated, use ``lookup_ID()`` or
``zeek::id::find_val()``.
- ``Val::Type()`` is deprecated, use ``Val::GetType``.
- Most global type/value pointers in NetVar.h are deprecated, but one can
still always perform the lookup themselves.
- ``Scope::Lookup()`` is deprecated, use ``Scope::Find()``.
- All generated ``BroType*`` names in the ``BifType::`` namespaces are
deprecated, but there's an equivalent name in ``zeek::BifType::`` of
``IntrusivePtr`` type to use instead.
- All generated ``BifConst::`` names are deprecated, but there's an
equivalent name now in ``zeek::BifCont::``, and changed to ``IntrusivePtr``
if the old name was some ``Val*`` type.
- Constructors for ``Val`` types that take a ``BroType*`` are all generally
deprecated, with alternatives that instead take an ``IntrusivePtr`` argument.
- ``FuncType::Args()`` is deprecated, use ``FuncType::Params()``.
- ``FuncType::ArgTypes()`` is deprecated, use ``FuncType::ParamList()``.
- ``RecordVal::Assign(int, Val*)`` is deprecated, use the overload taking
``IntrusivePtr``.
- ``RecordVal::Lookup(int)`` is deprecated, use ``RecordVal::GetField(int)``.
- ``RecordVal::LookupWithDefault(int)`` is deprecated, use
``RecordVal::GetFieldOrDefault(int)``.
- ``RecordVal::Lookup(const char*, bool)`` is deprecated, use either
``RecordVal::GetField()`` or ``RecordVal::GetFieldOrDefault()``.
- ``TableVal::Assign`` methods taking raw ``Val*`` are deprecated, use the
overloads taking ``IntrusivePtr``.
- ``TableVal::Lookup()`` is deprecated, use ``TableVal::Find()`` or
``TableVal::FindOrDefault()``.
- ``VectorVal::Assign`` and ``Insert`` methods taking raw ``Val*`` are
deprecated, use the methods that take ``IntrusivePtr``.
- ``VectorVal::Lookup()`` is deprecated, use ``VectorVal::At()``.
- The file analysis/analyzer API has deprecated methods taking raw
``RecordVal*`` for analyzer arguments and replaced those with methods
taking ``IntrusivePtr``.
- The ``Val(double, TypeTag)`` constructor is deprecated, use either
``IntervalVal()``, ``TimeVal()`` or ``DoubleVal()`` constructors.
- The "BroString.h" file is deprecated, use "ZeekString.h"
- The str_split() BIF is deprecated, use str_split_indices(). Note
that the former returns a vector with indices starting at 1 while the
later returns a vector with indices starting at 0.
- The ``icmp_conn`` parameter of ICMP events is deprecated, there's an
alternate version with an ``icmp_info`` parameter to use instead.
The ``icmp_conn`` record passed to ICMP events has always been re-used
amongst all events within an ICMP "connection", so the
``itype``, ``icode``, ``len``, and ``hlim`` fields as inspected in
handlers never appears to change even if the underlying packet data
has different values for those fields. However, it's not known if
anyone relied on that behavior, so the new ``icmp_info`` record is
introduced with the more-expected behavior of being created and
populated for each new event. It also removes the orig_h/resp_h
fields since those are redundant with what's already available in
the connection parameter.
- External plugins should include Zeek header files like
``#include <zeek/Foo.h>`` instead of ``#include <Foo.h>``. The later
style is considered deprecated. Reliance on ``zeek-config --include_dir``
to contain ``$prefix/include/zeek`` is also deprecated: its replacement
output is expected to be just ``$prefix/include``, with it currently
outputting both paths, delimited by a colon, during the deprecation period.
Zeek 3.1.0
==========
New Functionality
-----------------
- Add a new supervisor framework that enables Zeek to operate clusters
of processes itself without any external help.
The Supervisor framework provides an entirely new deployment mode
for Zeek, one that supervises a set of Zeek processes that are meant
to be persistent. A Supervisor automatically revives any process
that dies or exits prematurely and also arranges for an ordered
shutdown of the entire process tree upon its own termination. This
Supervisor mode for Zeek provides the basic foundation for process
configuration/management that could be used to deploy a Zeek cluster
similar to what ZeekControl does, but is also simpler to integrate
as a standard system service.
This mode is still experimental and will evolve over time. The
command-line argument of ``-j`` toggles Zeek to run in "Supervisor
mode" to allow for creation and management of child processes. If
you're going to test this, please note that you will need some
custom script code to configure the processes you want Zeek to run.
See the documentation for more information:
https://docs.zeek.org/en/stable/frameworks/supervisor.html
- Add a new option, ``dpd_late_match_stop``, which can be used in conjuction
with the option ``dpd_match_only_beginning`` and the new event
``protocol_late_match`` to help annotate the conn.log with a field
to speculate on the protocol/service in cases where the DPD buffer
was already exhausted and can't analyze the full connection anymore,
but where there was still a late signature match. A new script,
``policy/protocols/conn/speculative-service.zeek``, was added as an
example of how to perform this tuning and add a "speculative_service"
field to conn.log, but it's not loaded by default.
- There is now a new ``tcp_options`` event that is raised for each TCP header
that contains options.
- Added a new option, ``Log::print_to_log`` that can be set to automatically
redirect the output from "print" statements to a real log stream (e.g.
instead of writing to stdout).
- There is now a new ``&on_change`` attribute that can be used to be notified
of changes to tables and sets.
Changed Functionality
---------------------
- A C++17-capable compiler and CMake 3.0+ are now required to compile Zeek
- The backwards-compability wrappers & work-arounds introduced in 3.0
for the "Bro to Zeek rename" have either changed their operation, or in some
cases been removed. Generally, anything that reported a
naming-related warning in 3.0 now aborts with a corresponding error
message. In cases where 3.0 silently continued to accept old names,
3.1 now reports warnings. Most importantly, that's loading of
scripts with ".bro" endings, which are now flagged and should be
renamed.
- Broker has switched versions for the underlying CAF communication
library from 0.16 to 0.17. CAF has changed its wireformat between
those versions, which means that previous Broker and Zeek versions
won't be able to connect to the new ones. In other words, all Zeek
instances, as well as other Broker clients, need to upgrade at the
same time. In case of version mismatches, Broker now reports better
error messages to point out the problem.
- The key type of ``Known::service_store`` has changed to
``Known::AddrPortServTriplet`` and ``Known::services`` is now a table
instead of just a set.
- The DNS class name for Hesiod in the ``DNS::classes`` table is now spelled
correctly as "C_HESIOD" instead of "C_HESOD". For reference, this
class name string may appear in the dns.log file or in any custom
script logic that inspects the ``qclass`` field of ``DNS::Info``
after a ``dns_request`` event.
- The configuration input reader now ignores trailing spaces at the end of
configuration lines.
- The tcp_option event is now correctly raised.
- The base scripts shipped with Zeek now use the new
``successful_connection_remove`` event instead of
``connection_state_remove`` where possible (when the logic doesn't
pertain to unestablished TCP connections). There's a performance
benefit to this switch, but it may potentially break custom scripts
that depended on accessing or modifying state via specific
``&priority`` ordering of ``connection_state_remove`` event
handlers. However, since most of Zeek's base scripts previously just
used that event with the intention of writing a finalized log as the
last thing executed for a given connection, and the new
``successful_connection_remove`` event handlers all run after
``connection_state_remove``, it's not likely this change to the base
scripts causes any incompatibility with user scripts.
There's also a new event called ``connection_successful`` and a new
``connection`` record field named "successful" to help indicate this
new property of connections.
- The JSON output formatters now use the RapidJSON library. This
improves their performance considerably over the library that was
previously used. Output from the formatters remains nearly
identical.
- The ``decompose_uri`` function no longer raises an error when parsing
URIs with an empty port number (e.g. ``http://example.org:/``). Instead,
the ``portnum`` component of the returned ``URI`` value is left
uninitialized.
- Replace old ``select``-based IO loop with a new architecture that doesn't
spin checking for active IO sources. The new architecture now waits for the
the sources to actively notify it when activity occurs and only processes
data once it's ready. This helps heavily reduce the CPU usage on idle
network connections. This includes a couple of breaking changes:
- Only a single packet source is allowed to be specified from the
command-line now. If you pass combinations of multiple ``-r`` and/or
``-i`` flags, Zeek will return an error at startup.
- The IOSource API changed fairly wildly. The ``GetFds()`` and
``NextTimestamp`` methods no longer exist. If you had previously
implemented a custom IOSource, you will need to look at the new API
and make changes to your code to accomodate it. This does not include
packet sources, which should remain functional with little to no
changes, since the entirety of the changes should be in ``PktSrc``.
- Remove a large number of headers from being included by various files across
the entire code base, which leads to a sizeable build time improvement. This
set of changes has the potential to cause plugins to not build anymore. The
fixes for this potential breakage should just be a matter of including the
necessary headers in the plugin code.
Removed Functionality
---------------------
- Removed the ``current_conns_extern`` field from the ConnStats record
type. Zeek only maintains a single timer manager now, and without the
manager tags that came with multiple tiemr managers, we don't track
whether a connection is external anymore.
Deprecated Functionality
------------------------
- The C++ API typedefs for int{8,16,32,64} and uint{8,16,32,64} are deprecated
in favor of the real <cstdint> types they alias. E.g. use int8_t instead of
int8.
- The C++ API functions "safe_snprintf" and "safe_vsnprintf" are deprecated.
Use "snprintf" and "vsnprintf" instead.
Zeek 3.0.0
==========
New Functionality
-----------------
- Added support for DNSSEC resource records RRSIG, DNSKEY, DS, NSEC, and NSEC3.
The associated events are:
- dns_RRSIG
- dns_DNSKEY
- dns_DS
- dns_NSEC
- dns_NSEC3
- Added support for parsing and logging DNS SPF resource records.
A new ``dns_SPF_reply`` event is also available.
- Zeek's Plugin framework now allows a patch version. If a patch version is not
provided, it will default to 0. To specify this, modify the plugin
Configuration class in your ``src/Plugin.cc`` and set
``config.version.patch``. Note that the default plugin skeleton
includes a unit test whose Baseline has the plugin version number in
it and that will now fail due to the version number now including a
patch number. For those that want to keep the unit test, simply adapt
the unit test/baseline to include the new plugin patch number.
- The default http.log not includes a field for the HTTP request Origin header.
- Support for decapsulating VXLAN tunnels.
- The for-loop syntax now allows for iterating over key-value pairs of tables.
Previously, a separate lookup within the loop was required to obtain the
value at a given index/key, but now this works::
local t: table[count] of string = table();
t[1] = "hello";
t[55] = "goodbye";
for ( key, value in t )
print key, value;
- Added options for controlling the source path/prefix for Input and
Intel framework files:
- InputAscii::path_prefix
- InputBinary::path_prefix
- Intel::path_prefix
- Support for NFLOG link-layer type.
- Support for some SMB 3.x features
- An ``smb2_transform_header`` event is raised after parsing
TRANSFORM_HEADER structures associated with encrypted messages.
- The ``SMB2::NegotiateResponse`` record now contains
``negotiate_context_count`` and ``negotiate_context_values`` fields
containing capability information found in an SMB 3.1.1 dialect's
negotiation message.
- Added a new hook, ``Intel::filter_item``, to assist in filtering and
removal of intelligence items that are about to be inserted.
- Add support for SMB filenames in the intel framework.
- Added a new event for weirdness found via file analysis: ``file_weird``.
- The conn.log "history" field supports a new character 'G' or 'g'
(capital for originator, lowercase responder) to indicate a content
gap in the TCP stream. These are recorded logarithmically.
- The ``ZEEK_DNS_RESOLVER`` environment variable now controls
the DNS resolver to use by setting it to an IPv4 or IPv6 address. If
not set, then the first IPv4 address from /etc/resolv.conf gets used.
- The ``/<re>/i`` convenience syntax for case-insensitive patterns is now
also allowed when specifying patterns used in signature files.
- New RDP functionality.
- New events:
- rdp_client_network_data
- rdp_client_security_data
- rdp_client_cluster_data
- rdp_native_encrypted_data
- Add a new "client_channels" field to rdp.log based on data parsed from
the Client Network Data (TS_UD_CS_NET) packet. The channel list is also
available in the new ``rdp_client_network_data`` event.
- Add parsing support for TLS 1.3 pre-shared key extension. This info
is available in the events: ``ssl_extension_pre_shared_key_client_hello``
and ``ssl_extension_pre_shared_key_server_hello``.
- Added/re-wrote support for NTP.
- Parsing support for modes 1-7, with parsed structures available in
the ``ntp_message`` event.
- An ntp.log is produced by default, containing data extracted from
NTP messages with modes 1-5.
- Add support for vector slicing operations. For example::
local v = vector(1, 2, 3, 4, 5);
v[2:4] = vector(6, 7, 8); # v is now [1, 2, 6, 7, 8, 5]
print v[:4]; # prints [1, 2, 6, 7]
- Add support for paraglob, a fairly quick data structure for matching a string
against a large list of patterns. For example::
local v1 = vector("*", "d?g", "*og", "d?", "d[!wl]g");
local p1 = paraglob_init(v1);
print paraglob_match(p1, "dog");
- An ``expire_func`` for a table with a multi-value-index will now unroll
the index and take one argument for each index value. For example, for a
``table[string,string] of count`` the expire function signature is:
function(t: table[string, string] of count, s: string, s2: string): interval
- Zeek's anonymous functions now capture their closures by reference.
This means that they can use and modify variables from the scope
that they were generated in. For example:
local n = 3;
local f = function() { n += 1; };
f();
print n; # prints 4
These anonymous functions can also be serialized over Broker with
their closures. In order to be serialzed over Broker the receiving
script needs to have an identical version of the function declared.
For the above example, a receiving script would need to have
declared a function
local name = function() { n += 1; };
to be able to receive the senders function over Broker.
Functions with closures can still use the variables they have
captured even after they have left the scope that they were declared
in. For example, a simple generator function like the one below
works as expected.
local make_adder = function(n: count): function(m: count): count
{
return function (m: count): count
{
return n + m;
};
};
print make_adder(3)(5); # prints 8
local three = make_adder(3);
print three(5); # prints 8
- Add ``LogAscii::enable_utf_8`` option to allow valid utf8 sequences
to be written directly into the ASCII logs without any escaping.
- Add parsing, analysis, and logging support for MQTT protocol v3.1/v3.1.1.
This is not enabled by default, use ``@load policy/protocols/mqtt`` to
use the new MQTT analysis support.
- Zeek now supports duration thresholding on connections, similarly to how it supports
byte and packet thresholds.
- New events:
- ConnThreshold::duration_threshold_crossed
- conn_duration_threshold_crossed
- New functions:
- ConnThreshold::set_duration_threshold
- ConnThreshold::delete_duration_threshold
- set_current_conn_duration_threshold
- get_current_conn_duration_threshold
Changed Functionality
---------------------
- The following executable names have changed (the old names will
continue to work, but emit a deprecation warning):
- ``bro`` is now ``zeek``
- ``bro-config`` is now ``zeek-config``
- ``broctl`` is now ``zeekctl``
- ``bro-cut`` is now ``zeek-cut``
- BroControl has been completely renamed to ZeekControl. Many installation
directories and files with "broctl" in their name have been changed
to use "zeekctl" instead. It's expected this has been done in a way
that's backwards compatible with previous Bro installations. E.g.
if you made customizations to the ``broctl.cfg`` file of a previous
installation, installing the newer Zeek version over it will retain that
file and even symlink the new ``zeekctl.cfg`` to it.
- The default install prefix is now ``/usr/local/zeek`` instead of
``/usr/local/bro``. If you have an existing installation that used
the previous default and are still using the new default when upgrading,
we'll crate ``/usr/local/zeek`` as a symlink to ``/usr/local/bro``.
Certain subdirectories will also get similar treatment: ``share/bro``,
``include/bro``, and ``lib/bro``.
- ``$prefix/share/bro/site/local.bro`` has been renamed to
``local.zeek``. If you have a ``local.bro`` file from a previous
installation, possibly with customizations made to it, the new
version of Zeek will install a ``local.zeek`` file that is a symlink
to the pre-existing ``local.bro``. In that case, you may want to
just copy ``local.bro`` into the new ``local.zeek`` location to
avoid confusion, but things are otherwise meant to work properly
without intervention.
- All scripts ending in ``.bro`` that ship with the Zeek source tree have
been renamed to ``.zeek``.
- The search logic for the ``@load`` script directive now prefers files
ending in ``.zeek``, but will fallback to loading a ``.bro`` file if
it exists. E.g. ``@load foo`` will first check for a ``foo.zeek``
file to load and then otherwise ``foo.bro``. Note that
``@load foo.bro`` (with the explicit ``.bro`` file suffix) prefers
in the opposite order: it first checks for ``foo.bro`` and then
falls back to a ``foo.zeek``, if it exists.
- The for-loop index variable for vectors has been changed from
'int' to 'count' type. It's unlikely this would alter/break any
script behavior unless they were explicitly inspecting the variable's
type (and there's typically no reason to do that).
- The startup/initialization behavior has changed such that any errors
encountered while processing the ``bro_init()`` event will cause the
process to terminate rather than continue on the main run loop.
- The ``dns_state`` field within ``connection`` records has changed: the
``pending_queries`` and ``pending_replies`` fields are now ``&optional``,
and there is a new field ``pending_query`` that is populated before
``pending_queries``. If you have scripts that access the ``pending_queries``
or ``pending_replies`` fields, they will need to be updated.
This change was made to improve performance.
- The ternary operator ("<expr> ? <alt1> : <alt2>") now enforces that
if "<alt1>" and "<alt2>" are both records, they are of the same
type. It was always assumed that they were, but code might have
still worked even if not.
- The "orig_fuids", "orig_filenames", "orig_mime_types" http.log fields
as well as their "resp" counterparts are now limited to having
"HTTP::max_files_orig" or "HTTP::max_files_resp" entries, which are 15
by default. The limit can also be ignored case-by-case via the
"HTTP::max_files_policy" hook.
- The binpac library is now only compiled as a shared library by default.
To revert back to compiling only a static library, there's the
``--enable-static-binpac`` configure option.
- The Broker C++ API has some breaking changes, see it's own NEWS file for
details on how to migrate old code.
- Some Weirds associated with generic binpac parsing exceptions in analyzers
that didn't otherwise handle them (like syslog, modbus, dnp3) are now
a ProtocolViolation instead
- An "addl" parameter was added to the ``flow_weird`` and ``net_weird`` events
for describing additional information about the weird. The ``conn_weird``
event already had such a parameter.
- Weird names that contained variable content and may result in an unbounded
number of weird names have been renamed to remove the variable content
(which has been made available in the "addl" field of ``conn_weird``,
``flow_weird``, or ``net_weird`` events):
- "unknown_dce_rpc_auth_type_%d" -> unknown_dce_rpc_auth_type
- "gtp_invalid_info_element_%d" -> gtp_invalid_info_element
- "unknown_netbios_type:" 0x%x -> unknown_netbios_type
- "excess_netbios_hdr_len" (%d > %d) -> excess_netbios_hdr_len
- "deficit_netbios_hdr_len" (%d > %d) -> deficit_netbios_hdr_len
- "bad_RPC_program (%d)" -> bad_RPC_program
- "unknown_MOUNT_request(%u)" -> unknown_MOUNT_request
- "unknown_NFS_request(%u)" -> unknown_NFS_request
- "RPC resync: discard %d bytes\n" -> RPC_resync
- "RPC_message_too_long (%d64)" -> RPC_message_too_long
- "socks5_unsupported_authentication_method_%d" -> socks5_unsupported_authentication_method
- "socks5_unsupported_authentication_%d_%d" -> socks5_unsupported_authentication
- "ssh_unknown_kex_algorithm=%s" -> ssh_unknown_kex_algorithm
- "Encountered unknown type in server name ssl extension: %d" -> ssl_ext_unknown_server_name_type
- "UDP_datagram_length_mismatch(%d!=%d)" -> UDP_datagram_length_mismatch
- "OPENSSL Could not parse OCSP request (fuid %s)" -> openssl_ocsp_request_parse_error
- "OPENSSL Could not parse OCSP response (fuid %s)" -> openssl_ocsp_response_parse_error
- "Could not parse X509 certificate (fuid %s)" -> x509_cert_parse_error
- "Certificate with invalid BasicConstraint. fuid %s" -> x509_invalid_basic_constraint
- "Could not parse subject alternative names. fuid %s" -> x509_san_parse_error
- "DNS-field does not contain an IA5String. fuid %s" -> x509_san_non_string
- "Weird IP address length %d in subject alternative name. fuid %s" -> x509_san_ip_length
- "Could not parse time in X509 certificate (fuid %s) -- UTCTime has wrong length" -> x509_utc_length
- "Could not parse UTC time in non-YY-format in X509 certificate (x509 %s)" -> x509_utc_format
- "Could not parse time in X509 certificate (fuid %s) -- Generalized time has wrong length" -> x509_gen_time_length
- "Invalid time type in X509 certificate (fuid %s)" -> x509_invalid_time_type
- "Could not parse time in X509 certificate (fuid %s) -- additional char after time" -> x509_time_add_char
- "Could not parse time in X509 certificate (fuid %s) -- not enough bytes remaining for offset" -> x509_time_offset_underflow
- "Could not parse time in X509 certificate (fuid %s) -- unknown offset type" -> x509_time_offset_type
- "X509::GetExtensionFromBIO: %s" -> x509_get_ext_from_bio
- "unknown_mobility_type_%d" -> unknown_mobility_type
- "unknown_routing_type_%d" -> unknown_routing_type
- "unknown_protocol_%d" -> unknown_protocol
- "unknown_gre_version_%d" -> unknown_gre_version
- "unknown_gre_protocol_%u16" -> unknown_gre_protocol
- The "missed_bytes" field of conn.log can be calculated slightly differently
in some cases: ACKs that reveal a content gap, but also come at
the end of a connection (in a FIN or RST) are considered unreliable
and aren't counted as true gaps.
- The Broxygen component, which is used to generate our Doxygen-like
scripting API documentation has been renamed to Zeekygen. This likely has
no breaking or visible changes for most users, except in the case one
used it to generate their own documentation via the ``--broxygen`` flag,
which is now named ``--zeekygen``. Besides that, the various documentation
in scripts has also been updated to replace Sphinx cross-referencing roles
and directives like ":bro:see:" with ":zeek:see:".
- The catch-and-release and unified2 scripts are no longer loaded by
default. Because there was a performance impact simply from loading
them and it's unlikely a majority of user make use of their features,
they've been moved from the scripts/base/ directory into
scripts/policy/ and must be manually loaded to use their
functionality. The "drop" action for the notice framework is likewise
moved since it was implemented via catch-and-release. As a result,
the default notice.log no longer contains a "dropped" field.
If you previously used the catch-and-release functionality add this:
@load policy/frameworks/netcontrol/catch-and-release
If you previously used Notice::ACTION_DROP add:
@load policy/frameworks/notice/actions/drop
If you previously used the Unified2 file analysis support add:
@load policy/files/unified2
- The default value of ``peer_description`` has changed from "bro"
to "zeek". This won't effect most users, except for the fact that
this value may appear in several log files, so any external plugins
that have written unit tests that compare baselines of such log
files may need to be updated.
- The "remote_ip" field of "addr" type was removed from radius.log and
replaced with a field named "tunnel_client" of "string" type. The
reason for this is that the Tunnel-Client-Endpoint RADIUS attribute
this data is derived from may also be a FQDN, not just an IP address.
- The ``ssl_server_hello`` event's ``server_random`` parameter has been
changed to always include the full 32-byte field from the
ServerHello. Previously a 4-byte timestamp and 28-byte random data
were parsed separately as some TLS protocol versions specified a
separate timestamp field as part of the full 32-byte random sequence.
- The namespace used by all the builtin plugins that ship with Zeek have
changed to use "Zeek::" instead of "Bro::".
- Any Broker topic names used in scripts shipped with Zeek that
previously were prefixed with "bro/" are now prefixed with "zeek/"
instead.
In the case where external applications were using a "bro/" topic
to send data into a Bro process, a Zeek process still subscribes
to those topics in addition to the equivalently named "zeek/" topic.
In the case where external applications were using a "bro/" topic
to subscribe to remote messages or query data stores, there's no
backwards compatibility and external applications must be changed
to use the new "zeek/" topic. The thought is this change will have
low impact since most data published under "bro/" topic names is
intended for use only as a detail of implementing cluster-enabled
versions of various scripts.
A list of the most relevant/common topic names that could potentially
be used in external applications to consume/query remote data that
one may need to change:
- store names
- bro/known/services
- bro/known/hosts
- bro/known/certs
- cluster nodes
- bro/cluster/<node type>
- bro/cluster/node/<name>
- bro/cluster/nodeid/<id>
- logging
- bro/logs/<stream>
- The ``resp_ref`` argument was removed from the ``ocsp_response_bytes``
event. ``resp_ref`` was not used by anything in the codebase and could not be
passed to any other functions for further processing. The remainder of the
``ocsp_response_bytes`` is unchanged.
- For performance reasons, processing of notices is now always
performed by the node on which the notice is raised rather than
the centralized Manager node. This has potential incompatibilities
for those that relied on global state for notice policy processing.
It also introduces an expected race condition that may cause multiple
notices of the same kind that are generated within a short timespan
of each other on separate cluster nodes to all be logged rather
than suppressed and de-duplicated into a single notice.
- to_json is now a bif, no longer a script. Loading base/utils/json.zeek is no
longer necessary and has been deprecated. to_json should yield much better, always
valid json. There are some small differences in output; unnecessary spaces are removed
and port values are rendered differently, now including the port and the protocol.
- The output of the JSON logger now uses an external library to generate json. There
are small changes to the output; most visibly double numbers are now rounded slightly