ws: allow filtering notification by parameters #3689

carpawell · 2024-11-18T08:49:04Z

Closes #3624.

carpawell · 2024-11-18T08:55:21Z

Some questions:

Is there any full test for client-to-server communication to check that filtering works? Not just unmarshalling but also creating notifications and receiving only the important ones. Have not found it.
Are type limitations correct?
What limit should be used for parameter filters? Why should we have it (the issue says)? Just not to have some OOMs on a server side or it is possible to abuse a node somehow?

codecov · 2024-11-18T08:57:00Z

Codecov Report

Attention: Patch coverage is 60.86957% with 18 lines in your changes missing coverage. Please review.

Project coverage is 82.97%. Comparing base (176593b) to head (b4fc362).
Report is 16 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/neorpc/filters.go	48.27%	11 Missing and 4 partials ⚠️
pkg/neorpc/rpcevent/filter.go	82.35%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3689      +/-   ##
==========================================
- Coverage   83.05%   82.97%   -0.08%     
==========================================
  Files         334      335       +1     
  Lines       46604    46753     +149     
==========================================
+ Hits        38705    38793      +88     
- Misses       6320     6368      +48     
- Partials     1579     1592      +13

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests

AnnaShaleva · 2024-11-18T09:31:44Z

Is there any full test for client-to-server communication to check that filtering works?

Use and extend the following tests:

neo-go/pkg/services/rpcsrv/subscription_test.go

Line 130 in 990634a

func TestSubscriptions(t *testing.T) {

neo-go/pkg/services/rpcsrv/client_test.go

Line 2131 in 990634a

func TestWSClient_SubscriptionsCompat(t *testing.T) {

That's the way how we test subscriptions. If it's not enough, then create your own test based on

neo-go/pkg/services/rpcsrv/client_test.go

Line 1895 in 990634a

func TestSubClientWait(t *testing.T) {

Are type limitations correct?

Will be answered in review.

What limit should be used for parameter filters?

Let's limit the number of parameters to 16 for now, it's pretty enough for notifications used by NeoFS and at the same time it won't allow to DoS the node with useless filtering process for large notifications/filters. Also, parameter types should be limited by non-compound types (simple Integer, String, Hash160 and etc.; excluding Arrays, Structs and Maps), we don't need compounds for now and NeoFS contracts don't use them in notifications; in future the set of supported types may be extended.

Why should we have it (the issue says)?

Avoid filters misuse and unwanted load for RPC server. This extension will be available on public RPC nodes.

it is possible to abuse a node somehow?

Deploy contract that emits thousands of notifications (it's possible, hi, #3490), then subscribe to RPC server with matching filters.

AnnaShaleva · 2024-11-18T09:32:18Z

pkg/neorpc/filters.go

+	//  - [smartcontract.AnyType]
+	//	- [smartcontract.BoolType]


AnnaShaleva · 2024-11-18T09:32:40Z

pkg/neorpc/filters.go

@@ -29,11 +31,24 @@ type (
 	}
 	// NotificationFilter is a wrapper structure representing a filter used for
 	// notifications generated during transaction execution. Notifications can
-	// be filtered by contract hash and/or by name. nil value treated as missing
-	// filter.
+	// be filtered by contract hash, by event name or by notification parameters.


s/or/and/or

not sure if it is right to have 3 things with comma and and/or but ok

AnnaShaleva · 2024-11-18T09:33:38Z

pkg/neorpc/filters.go

+	// be filtered by contract hash, by event name or by notification parameters.
+	// Notification parameter filters will be applied in the order corresponding
+	// to a produced notification's parameters. Any-typed parameter with zero
+	// value allows any notification parameter. Supported parameter types:


Any-typed parameter with zero value allows any notification parameter.

Add a note that filter with all parameters of Any type is a no-op.

isnt "Any-typed parameter with zero value allows any notification parameter" a general rule for what you are saying? can you, please, suggest exact wording if it is hard to understand what i wrote?

Filter with [Any, Any, Any] is invalid. User must be enforced to filter only by name in this case, without additional filters. Documentaion doesn't say anything about it.

pkg/neorpc/filters.go

AnnaShaleva · 2024-11-18T09:40:46Z

pkg/neorpc/filters.go

@@ -134,6 +152,16 @@ func (f NotificationFilter) IsValid() error {
 	if f.Name != nil && len(*f.Name) > runtime.MaxEventNameLen {
 		return fmt.Errorf("%w: NotificationFilter name parameter must be less than %d", ErrInvalidSubscriptionFilter, runtime.MaxEventNameLen)
 	}
+	if len(f.Parameters) != 0 { // todo: limit max size? what number?


AnnaShaleva · 2024-11-18T09:45:13Z

pkg/neorpc/filters.go

+	if len(f.Parameters) != 0 { // todo: limit max size? what number?
+		for i, parameter := range f.Parameters {
+			if parameter.Type < smartcontract.AnyType || parameter.Type > smartcontract.SignatureType {
+				return fmt.Errorf("%w: NotificationFilter unsupported %d parameter type: %s", ErrInvalidSubscriptionFilter, i, parameter.Type)


NotificationFilter unsupported %d parameter type: %s

It's supposed to be a sentence, like other logs. Let's rephrase to NotificationFilter type parameter %d is unsupported: %s

pkg/neorpc/rpcevent/filter.go

AnnaShaleva · 2024-11-18T10:03:12Z

pkg/neorpc/rpcevent/filter.go

+					parametersOk = false
+					break
+				}
+				converted, err := p.ToStackItem()


It's too expensive to convert every filter's parameter every time you need to access it. Consider contract that emits 100500 notifications that match filter's requirements, every block. To improve it define an additional method (some (*neorpc.NotificationFilter) ParametersSI() []stackitem.Item), this method should convert filter parameters to stckitems and cache the resulting value inside filter's structure. Cache should be reused for subsequent invocations of this method. Cache should be cleaned on filter's Copy.

Also, prior to parameter's value comparison use parameter types comparison:

neo-go/pkg/smartcontract/param_type.go

Line 193 in 990634a

func (pt ParamType) Match(v stackitem.Item) bool {

Also please add a separate unit-test for various parameter types matching comparison.

not sure what is more scary: caching bugs and more complex logic or converting vars on stack, but don't mind. no mutexes since it should not be used concurrently, ping me if you don't agree

Also, prior to parameter's value comparison use parameter types comparison:

isn't stackitem.Item.Equal enough?

Also, prior to parameter's value comparison use parameter types comparison:

can, please, explain, why it is needed? there is a single smartcontract.Parameter -> stackitem.Item conversion rule and i use it (was not written by me). if it is not fixed, how does this work at all then? at least that is how i understand it

isn't stackitem.Item.Equal enough?

It's enough but type comparison allows to fail fast. Parameter to stackitem conversion is not cheap.

can, please, explain, why it is needed?

Fail fast in case of types mismatch.

Fail fast

got you

pkg/neorpc/rpcevent/filter.go

AnnaShaleva · 2024-11-18T10:20:48Z

pkg/neorpc/filters.go

@@ -134,6 +152,16 @@ func (f NotificationFilter) IsValid() error {
 	if f.Name != nil && len(*f.Name) > runtime.MaxEventNameLen {
 		return fmt.Errorf("%w: NotificationFilter name parameter must be less than %d", ErrInvalidSubscriptionFilter, runtime.MaxEventNameLen)
 	}
+	if len(f.Parameters) != 0 { // todo: limit max size? what number?
+		for i, parameter := range f.Parameters {
+			if parameter.Type < smartcontract.AnyType || parameter.Type > smartcontract.SignatureType {


The case when all parameter types are Any is also a no-op.

carpawell · 2024-11-20T17:33:23Z

@AnnaShaleva thanks for the review! It was kinda draft with the main questions but tried to answer and fix all the threads you left, check one more time, please.

AnnaShaleva

Tests are not yet checked, will review them after PR finalisation.

docs/notifications.md

pkg/neorpc/filters.go

AnnaShaleva · 2024-11-22T08:27:00Z

pkg/neorpc/filters.go

+	if len(f.Parameters) != 0 {
+		res.Parameters = slices.Clone(f.Parameters)
+	}
+	f.parametersCache = nil


But why? It's res.Parameters cache that should be cleaned (but it's not set anyway). So just remove f.parametersCache = nil and adjust method documentation a bit.

i tried to follow your suggestion, i believe:

Cache should be cleaned on filter's Copy.

i treat it like a new life for the struct (otherwise i do not know why somebody needs to copy something). if you copy smth, you can try to reuse the original struct and then it will be unexpected when you have copied the struct, changed f.Parameters, and then called ParametersSI with the old return values

AnnaShaleva · 2024-11-22T08:28:21Z

pkg/neorpc/filters.go

+// to update parameters, use [NotificationFilter.Copy].
+// It mainly should be used by server code. Must not be used concurrently.


Preserve the overall line width.

it was like a "new paragraph" and i divided it on purpose, cause it is a new logical block, IMO. dropped new line

AnnaShaleva · 2024-11-22T08:29:12Z

pkg/neorpc/filters.go

+// MaxNotificationFilterParametersCount is a reasonable filter's parameter limit
+// that does not allow attackers to increase node resources usage but that
+// also should be enough for real applications.
+const MaxNotificationFilterParametersCount = 16


Usually all constants are places at the top of the document.

i dont mind but it is a const that is used exactly for a single purpose and defined right before the func that uses it. i would even do it as an internal const but thought that it would be useful to have some exported ref on what the requirements are

AnnaShaleva · 2024-11-22T08:31:41Z

pkg/neorpc/filters.go

+				return fmt.Errorf("%w: NotificationFilter type parameter %d is unsupported: %s", ErrInvalidSubscriptionFilter, i, parameter.Type)
+			}
+			if _, err := parameter.ToStackItem(); err != nil {
+				return fmt.Errorf("%w: NotificationFilter filter parameter does not correspond to any stack item: %w", ErrInvalidSubscriptionFilter, err)


Would be nice to add parameter index to the error details.

AnnaShaleva · 2024-11-22T08:32:06Z

pkg/neorpc/filters.go

+		}
+	}
+	if noopFilter {
+		return fmt.Errorf("%w: NotificationFilter cannot have all parameters of %s", ErrInvalidSubscriptionFilter, smartcontract.AnyType)


all parameters of %s

all parameters of type %s

AnnaShaleva · 2024-11-22T09:19:32Z

pkg/neorpc/filters.go

+// method it will not change even if you change any structure fields. If you need
+// to update parameters, use [NotificationFilter.Copy].
+// It mainly should be used by server code. Must not be used concurrently.
+func (f *NotificationFilter) ParametersSI() ([]stackitem.Item, error) {


BTW, this name was just an example, you may use any other name if you'd like to.

ok, i feel it more like ParametersAsStackItems

AnnaShaleva · 2024-11-22T09:23:06Z

pkg/services/rpcsrv/subscription_test.go

@@ -261,6 +268,39 @@ func TestFilteredSubscriptions(t *testing.T) {
 				require.Equal(t, "my_pretty_notification", n)
 			},
 		},
+		"notification matching contract hash and parameter": {
+			params: `["notification_from_execution", {"contract":"` + testContractHash + `", "parameters":[{"type":"Any","value":null},{"type":"Hash160","value":"449fe8fbd4523072f5e3a4dfa17a494c119d4c08"}]}]`,


449fe8fbd4523072f5e3a4dfa17a494c119d4c08 is a priv0 hash, right? Rubles contract transfers 1000 rubles to priv0 at block 5. Use goodSender.StringLE() instead of the raw value instead, otherwise it's painless to maintain this test.

oh, my bad, it is testContractHash, it calls Init and sends all the tokens to itself, a perfect unique notification IMO. started to use const

AnnaShaleva · 2024-11-22T09:27:53Z

pkg/services/rpcsrv/subscription_test.go

+				hashExp, err := hex.DecodeString(testContractHash)
+				require.NoError(t, err)
+				slices.Reverse(hashExp)
+				hashGot, err := base64.StdEncoding.DecodeString(transferReceiver)
+				require.NoError(t, err)
+				require.Equal(t, hashExp, hashGot)


In a separate commit make: s/testContractHash/testContractHashLE

Declare var testContractHash, _ = util.Uint160DecodeStringLE(testContractHashLE) similar to nfsoHash declaration.

Replace selected code with require.Equal(t, base64.StdEncoding.EncodeToString(testContractHash.StringBE(), transferReceiver). The shorter the better.

AnnaShaleva · 2024-11-22T09:28:52Z

pkg/services/rpcsrv/subscription_test.go

+				transferReceiverType := parameters[1].(map[string]any)["type"].(string)
+				require.Equal(t, smartcontract.Hash160Type.ConvertToStackitemType().String(), transferReceiverType)
+				transferReceiver := parameters[1].(map[string]any)["value"].(string)


s/transferReceiverType/toType
s/transferReceiver/to

Because it's commonly used [from, to, amount, args] cortege.

AnnaShaleva · 2024-11-22T09:29:49Z

pkg/services/rpcsrv/subscription_test.go

+				amountType := parameters[2].(map[string]any)["type"].(string)
+				require.Equal(t, smartcontract.IntegerType.ConvertToStackitemType().String(), amountType)
+				amount := parameters[2].(map[string]any)["value"].(string)
+				require.Equal(t, amount, "1000000")


Expected/actual misplaced.

`Any` type with nil/null value is treated as a parameter filter that allows any notification value. Not more than 16 filter parameters are allowed. Closes #3624. Signed-off-by: Pavel Karpy <[email protected]>

Signed-off-by: Pavel Karpy <[email protected]>

carpawell self-assigned this Nov 18, 2024

carpawell force-pushed the feat/filter-events-by-parameters branch from d4643f2 to fd5ad7b Compare November 18, 2024 08:57

AnnaShaleva reviewed Nov 18, 2024

View reviewed changes

carpawell force-pushed the feat/filter-events-by-parameters branch 2 times, most recently from 9191c15 to b4fc362 Compare November 20, 2024 17:31

carpawell marked this pull request as ready for review November 20, 2024 17:37

carpawell requested a review from roman-khimov as a code owner November 20, 2024 17:37

carpawell requested a review from AliceInHunterland November 20, 2024 17:38

AnnaShaleva reviewed Nov 21, 2024

View reviewed changes

carpawell force-pushed the feat/filter-events-by-parameters branch from b4fc362 to a87d7cb Compare November 21, 2024 17:50

carpawell requested a review from AnnaShaleva November 21, 2024 23:12

AnnaShaleva reviewed Nov 22, 2024

View reviewed changes

carpawell force-pushed the feat/filter-events-by-parameters branch from a87d7cb to b7d0172 Compare November 22, 2024 15:45

carpawell requested a review from AnnaShaleva November 22, 2024 15:45

carpawell added 2 commits November 22, 2024 18:51

ws: allow filtering notification by parameters

efecabf

`Any` type with nil/null value is treated as a parameter filter that allows any notification value. Not more than 16 filter parameters are allowed. Closes #3624. Signed-off-by: Pavel Karpy <[email protected]>

rpcsrv/tests: simplify tests with parsed test contract hash

29d2a57

Signed-off-by: Pavel Karpy <[email protected]>

carpawell force-pushed the feat/filter-events-by-parameters branch from b7d0172 to 29d2a57 Compare November 22, 2024 16:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ws: allow filtering notification by parameters #3689

ws: allow filtering notification by parameters #3689

carpawell commented Nov 18, 2024

carpawell commented Nov 18, 2024

codecov bot commented Nov 18, 2024 •

edited

Loading

AnnaShaleva commented Nov 18, 2024

AnnaShaleva Nov 18, 2024

AnnaShaleva Nov 18, 2024

carpawell Nov 20, 2024

AnnaShaleva Nov 18, 2024

carpawell Nov 20, 2024

AnnaShaleva Nov 21, 2024

AnnaShaleva Nov 18, 2024

AnnaShaleva Nov 18, 2024

AnnaShaleva Nov 18, 2024 •

edited

Loading

AnnaShaleva Nov 18, 2024

carpawell Nov 20, 2024

AnnaShaleva Nov 21, 2024

carpawell Nov 21, 2024

AnnaShaleva Nov 18, 2024

carpawell commented Nov 20, 2024

AnnaShaleva left a comment

AnnaShaleva Nov 22, 2024

carpawell Nov 22, 2024

AnnaShaleva Nov 22, 2024

carpawell Nov 22, 2024

AnnaShaleva Nov 22, 2024

carpawell Nov 22, 2024

AnnaShaleva Nov 22, 2024

AnnaShaleva Nov 22, 2024

AnnaShaleva Nov 22, 2024

carpawell Nov 22, 2024

AnnaShaleva Nov 22, 2024

carpawell Nov 22, 2024

AnnaShaleva Nov 22, 2024

AnnaShaleva Nov 22, 2024

AnnaShaleva Nov 22, 2024

		// to update parameters, use [NotificationFilter.Copy].
		// It mainly should be used by server code. Must not be used concurrently.

ws: allow filtering notification by parameters #3689

Are you sure you want to change the base?

ws: allow filtering notification by parameters #3689

Conversation

carpawell commented Nov 18, 2024

carpawell commented Nov 18, 2024

codecov bot commented Nov 18, 2024 • edited Loading

Codecov Report

AnnaShaleva commented Nov 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnnaShaleva Nov 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carpawell commented Nov 20, 2024

AnnaShaleva left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Nov 18, 2024 •

edited

Loading

AnnaShaleva Nov 18, 2024 •

edited

Loading