Add / (search) commands to newshell + rewrite of rz_search #4742

wargio · 2024-11-25T14:40:51Z

Your checklist for this pull request

I've read the guidelines for contributing to this repository
I made sure to follow the project's coding style
I've documented or updated the documentation of every function and struct this PR changes. If not so I've explained why.
I've added tests that prove my fix is effective or that my feature works (if possible)
I've updated the rizin book with the relevant information (if needed)

Detailed description

This is a complete rewrite of the legacy code which is too complicated to import into new-shell.

The goal is to refactor rz_search and implement the search command properly using pre-existing code like search of strings via rz_utils and more.

closes #1909

Rot127 · 2024-11-25T16:01:46Z

librz/include/rz_search.h

+RZ_API void rz_search_collection_free(RZ_NULLABLE RzSearchCollection *sc);
+RZ_API void rz_search_hit_free(RZ_NULLABLE RzSearchHit *hit);
+
+RZ_API RZ_OWN RzList /*<RzSearchHit *>*/ *rz_search_run(RZ_NONNULL RzSearchOpt *opt, RZ_NONNULL RzSearchCollection *col, RZ_NONNULL RzIO *io, RZ_NONNULL RzList /*<RzIOMap *>*/ *search_in);


Maybe changing the search_in parameter to void * and infer the type based on some flag in opt. This way we can use this function later to search other things then IO.

Not so sure about this, but maybe.

Rot127 · 2024-11-25T16:05:54Z

librz/search/search_internal.h

+
+struct rz_search_collection_t {
+	RzPVector /*<void *>*/ *collection; ///< Collection of elements to search in a buffer
+	RzSearchOverCallback search_over; ///< Collection search over the collection callback


Docs are a little confusing. Is the search_over callback called for every element in the collection above?

No, the function takes the vector and uses it

Rot127 · 2024-11-25T16:11:00Z

librz/include/rz_search.h


 typedef struct rz_search_hit_t {
-	RzSearchKeyword *kw;
-	ut64 addr;
+	char *metadata; ///< Metadata for extra details (can be NULL)


This seems to save mostly the type of the search hit or the things to search for (I saw aes and regex). Maybe changing it to an enum (RzSearchMetaDataType type) and a add a void *meta_data to save more complex stuff. This way we can save allocation of simple strings.

I added it for the flags so we know it's hit.regex or hit.rsa etc...

Ah I see. What about hit_desc then? Meta data seems kinda broad IMHO.

sure for me.

Rot127 · 2024-11-25T16:11:28Z

librz/core/cmd_descs/cmd_search.yaml

@@ -3,8 +3,174 @@
 ---
 name: cmd_search
 commands:
+  - name: "//"
+    summary: Repeat the last seach


Suggested change

summary: Repeat the last seach

summary: Repeat the last search

Rot127 · 2024-11-25T16:14:21Z

librz/core/cconfig.c

@@ -3745,16 +3792,18 @@ RZ_API int rz_core_config_init(RzCore *core) {
 	SETOPTIONS(n, "auto", "rosections", "raw", NULL);

 	/* search */
-	SETCB("search.contiguous", "true", &cb_contiguous, "Accept contiguous/adjacent search hits");
-	SETICB("search.align", 0, &cb_searchalign, "Only catch aligned search hits");
+	SETB("search.progress", false, "Shows search progress (true: enable, false: disable)");


librz/core/cmd/cmd_print.c

Rot127 · 2024-11-25T16:27:43Z

librz/search/collection.c

+ *
+ * \return     On success returns a valid pointer, otherwise NULL.
+ */
+RZ_IPI RZ_OWN RzSearchCollection *rz_search_collection_new(RZ_NONNULL RzSearchOverCallback search_over, RZ_NULLABLE RzPVectorFree free) {


Suggested change

RZ_IPI RZ_OWN RzSearchCollection *rz_search_collection_new(RZ_NONNULL RzSearchOverCallback search_over, RZ_NULLABLE RzPVectorFree free) {

RZ_IPI RZ_OWN RzSearchCollection *rz_search_collection_new(RZ_NONNULL RzSearchOverCallback search_over, RZ_NULLABLE RzPVectorFree element_free) {

Or something similar.

XVilka

This is great work!

XVilka · 2024-11-30T15:56:26Z

librz/core/cmd/cmd_print.c

@@ -4371,6 +4371,36 @@ RZ_IPI RzCmdStatus rz_print_pattern8_handler(RzCore *core, int argc, const char
 	return RZ_CMD_STATUS_OK;
 }

+static void incAlphaBuffer(ut8 *buf, int bufsz) {


These are very weird functions

XVilka · 2024-11-30T15:58:11Z

librz/core/cmd/cmd_search.c

-struct search_parameters {
-	RzCore *core;
+typedef struct search_parameters {
+	RzCore *core; ///< RzCore instance to use


I think the search parameters should not store RzCore and RzCmdStateOutput, or maybe rename the structure in that case.

XVilka · 2024-11-30T15:58:52Z

librz/core/cmd/cmd_search.c

+typedef struct search_prelude_cb_ctx {
+	int counter; ///< Number of preludes found
+	RzCore *core; ///< RzCore to use
+	int depth; ///< analysis.depth value


Should it be size_t or ut32 for both depth and counter?

XVilka · 2024-11-30T16:01:55Z

librz/core/cmd_descs/cmd_search.yaml

+      - RZ_OUTPUT_MODE_TABLE
+    args: []
+
+  - name: "/p"


For every search subcommand we need more detailed description and few examples.

XVilka · 2024-11-30T16:03:15Z

librz/main/rz-find.c

@@ -15,6 +15,18 @@
 #include <rz_io.h>
 #include <rz_bin.h>

+typedef enum {
+	SEARCH_MODE_BYTES = 0,


I think these should be in librz/include/rz_search.h and also documented.

XVilka · 2024-11-30T16:07:56Z

librz/search/pkey_search.c

+// SPDX-FileCopyrightText: 2008 J. Alex Halderman
+// SPDX-License-Identifier: BSD-3-Clause
+
+// Originally from RSAKeyFinder 1.0 (2008-07-18) By Nadia Heninger and J. Alex Halderman


Same, please convert into \file

XVilka · 2024-11-30T16:08:35Z

librz/search/pkey_search.c

+	{ 0x02, 0x01, 0x00, 0x30 },
+};
+
+/*Baby BER parser, just good enough for private keys.


Since there is comment already for this function , maybe convert to Doxygen?

XVilka · 2024-11-30T16:08:41Z

librz/search/pkey_search.c

+	return start + 2 + lensize;
+}
+
+/* Check if `start` points to an ensemble of BER fields


XVilka · 2024-11-30T16:08:49Z

librz/search/pkey_search.c

+	return true;
+}
+
+// Finds and return index of a private key based on the asn1 bytes


XVilka · 2024-11-30T16:10:23Z

librz/search/string_search.c

+
+#include <rz_search.h>
+#include <rz_util.h>
+


Please add \file to note that it searches not only ANSI/ASCII but also Unicode and some other encodings

github-actions bot added RzAnalysis rz-find API RzCore ARM MIPS PPC X86 RzUtil RzSearch labels Nov 25, 2024

wargio and others added 10 commits November 25, 2024 22:44

Refactor search.c

9fcb104

Use search_parameters_t and implement /

1d6aae5

Use new apis for rz_config

ade02af

Move incAlphaBuffer and incDigitBuffer under cmd_print.c

2f4607e

Fix declaration of RzSearchCallback

731aef4

Remove rz_core_magic_reset

c52bc92

rewrite cmd_search.c

a4da156

Refactor rz_core_search_prelude to remove global variable

11d30ce

I give up. requires total refactor.

abd0c6c

Total rewrite of rz_search

fd68bba

Rot127 reviewed Nov 25, 2024

View reviewed changes

librz/core/cmd/cmd_print.c Show resolved Hide resolved

Rot127 reviewed Nov 25, 2024

View reviewed changes

Add rz_search_collection_strings

9b1253f

wargio force-pushed the fuzz-dist-cmd-search branch from 2123d77 to 9b1253f Compare November 26, 2024 14:00

XVilka mentioned this pull request Nov 29, 2024

Tracking porting the commands parser to the tree-sitter-based one #1342

Open

23 tasks

XVilka mentioned this pull request Nov 29, 2024

Arbitrarily high hit counts / high allocation counts hurts search performance, refactor search hits to use rz_vector? #1909

Open

XVilka requested changes Nov 30, 2024

View reviewed changes

XVilka added this to the 0.8.0 milestone Nov 30, 2024

XVilka added the blocker label Nov 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add / (search) commands to newshell + rewrite of rz_search #4742

Add / (search) commands to newshell + rewrite of rz_search #4742

wargio commented Nov 25, 2024 •

edited by Rot127

Loading

Rot127 Nov 25, 2024

wargio Nov 25, 2024

Rot127 Nov 25, 2024 •

edited

Loading

wargio Nov 25, 2024

Rot127 Nov 25, 2024

wargio Nov 25, 2024

Rot127 Nov 25, 2024

wargio Nov 26, 2024

Rot127 Nov 25, 2024

Rot127 Nov 25, 2024

Rot127 Nov 25, 2024

XVilka left a comment

XVilka Nov 30, 2024

XVilka Nov 30, 2024

XVilka Nov 30, 2024

XVilka Nov 30, 2024

XVilka Nov 30, 2024

XVilka Nov 30, 2024

XVilka Nov 30, 2024

XVilka Nov 30, 2024

XVilka Nov 30, 2024

XVilka Nov 30, 2024

	summary: Repeat the last seach
	summary: Repeat the last search

	RZ_IPI RZ_OWN RzSearchCollection *rz_search_collection_new(RZ_NONNULL RzSearchOverCallback search_over, RZ_NULLABLE RzPVectorFree free) {
	RZ_IPI RZ_OWN RzSearchCollection *rz_search_collection_new(RZ_NONNULL RzSearchOverCallback search_over, RZ_NULLABLE RzPVectorFree element_free) {

Add / (search) commands to newshell + rewrite of rz_search #4742

Are you sure you want to change the base?

Add / (search) commands to newshell + rewrite of rz_search #4742

Conversation

wargio commented Nov 25, 2024 • edited by Rot127 Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rot127 Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XVilka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wargio commented Nov 25, 2024 •

edited by Rot127

Loading

Rot127 Nov 25, 2024 •

edited

Loading