Notification on failures?

[DanTulovsky]

Hi,

I am setting up: https://argoproj.github.io/argo-rollouts/features/notifications/

It comes with the following pre-defined triggers:

  trigger.on-rollout-completed: |
    - send: [rollout-completed]
  trigger.on-rollout-step-completed: |
    - send: [rollout-step-completed]
  trigger.on-rollout-updated: |
    - send: [rollout-updated]
  trigger.on-scaling-replica-set: |
    - send: [scaling-replicaset]

Does on-rollout-completed include the failed case as well?

Thank you
Dan

[harikrongali]

@DanTulovsky Currently, only success triggers are added and framework is available to add other remaining especially failures ones.
Users can add custom triggers to manifest files and use. Please contribute back if you have defined more custom triggers. We have it as a lower priority given other priority items.

[DanTulovsky]

Would it be possible to provide an example for a failure trigger?

…

On Wed, Oct 20, 2021, at 11:54 AM, harikrongali wrote: @DanTulovsky <https://github.com/DanTulovsky> Currently, only success triggers are added and framework is available to add other remaining especially failures ones. Users can add custom triggers to manifest files and use. Please contribute back if you have defined more custom triggers. We have it as a lower priority given other priority items. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1586 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAD4PPY6CSKWRGTMKRCFJB3UH3Q2RANCNFSM5GCT4M5Q>.

[jessesuen]

You are probably interested in on-rollout-aborted which translates to the RolloutAborted Kubernetes event. We don't have a notification template/trigger for it yet, but it should be easy to add one following the examples here:
https://github.com/argoproj/argo-rollouts/tree/master/manifests/notifications

This would be great to contribute upstream if you come up with a useful template that others can use.

[DanTulovsky]

Thanks. I am also interested in "Canary Analysis Failed', how would I get that?

…

On Wed, Oct 20, 2021, at 3:28 PM, Jesse Suen wrote: You are probably interested in `on-rollout-aborted` which translates to the `RolloutAborted` Kubernetes event. We don't have a notification template/trigger for it yet, but it should be easy to add one following the examples here: https://github.com/argoproj/argo-rollouts/tree/master/manifests/notifications This would be great to contribute upstream if you come up with a useful template that others can use. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1586 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAD4PP4255W5OTJZDFX24YLUH4J57ANCNFSM5GCT4M5Q>.

[DanTulovsky]

Also, I don't understand, how is a pre-defined trigger like "trigger.on-rollout-step-completed" actually defined? I understand I can add custom ones, but where do these built-in ones come from?

…

On Thu, Oct 21, 2021, at 7:11 AM, Dan Tulovsky wrote: Thanks. I am also interested in "Canary Analysis Failed', how would I get that? On Wed, Oct 20, 2021, at 3:28 PM, Jesse Suen wrote: > > > > > You are probably interested in `on-rollout-aborted` which translates to the `RolloutAborted` Kubernetes event. We don't have a notification template/trigger for it yet, but it should be easy to add one following the examples here: > https://github.com/argoproj/argo-rollouts/tree/master/manifests/notifications > > > This would be great to contribute upstream if you come up with a useful template that others can use. > > > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub <#1586 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAD4PP4255W5OTJZDFX24YLUH4J57ANCNFSM5GCT4M5Q>. > >

[DanTulovsky]

Oh, i see.. these are just EventReason from `kubectl get events`? with a "on-" and EventName changed to "event-name" ...

…

On Fri, Oct 22, 2021, at 9:28 AM, Dan Tulovsky wrote: Also, I don't understand, how is a pre-defined trigger like "trigger.on-rollout-step-completed" actually defined? I understand I can add custom ones, but where do these built-in ones come from? On Thu, Oct 21, 2021, at 7:11 AM, Dan Tulovsky wrote: > Thanks. I am also interested in "Canary Analysis Failed', how would I get that? > > On Wed, Oct 20, 2021, at 3:28 PM, Jesse Suen wrote: >> >> >> >> >> You are probably interested in `on-rollout-aborted` which translates to the `RolloutAborted` Kubernetes event. We don't have a notification template/trigger for it yet, but it should be easy to add one following the examples here: >> https://github.com/argoproj/argo-rollouts/tree/master/manifests/notifications >> >> >> This would be great to contribute upstream if you come up with a useful template that others can use. >> >> >> >> — >> You are receiving this because you were mentioned. >> Reply to this email directly, view it on GitHub <#1586 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAD4PP4255W5OTJZDFX24YLUH4J57ANCNFSM5GCT4M5Q>. >> >> >

[DanTulovsky]

For those following at home, this is for rollout-aborted:

  template.rollout-aborted: |
  {{`
    message: Rollout {{.rollout.metadata.namespace}}/{{.rollout.metadata.name}} has been ABORTED.
    email:
      subject: Rollout {{.rollout.metadata.namespace}}/{{.rollout.metadata.name}} has been ABORTED.
    slack:
      attachments: |
          [{
            "title": "{{.rollout.metadata.namespace}}/{{ .rollout.metadata.name}}",
            "color": "#d61317",
            "fields": [
            {
              "title": "Strategy",
              "value": "{{if .rollout.spec.strategy.blueGreen}}BlueGreen{{end}}{{if .rollout.spec.strategy.canary}}Canary{{end}}",
              "short": true
            }
            {{range $index, $c := .rollout.spec.template.spec.containers}}
              {{if not $index}},{{end}}
              {{if $index}},{{end}}
              {
                "title": "{{$c.name}}",
                "value": "{{$c.image}}",
                "short": true
              }
            {{end}}
            ]
          }]
    `}}

  trigger.on-rollout-aborted: |
    - send: [rollout-aborted

[ervinb]

The list of all built-in events can be found here:

argo-rollouts/utils/conditions/conditions.go

Lines 20 to 177 in 76cfc0e

	const (
	// InvalidSpecReason indicates that the spec is invalid
	InvalidSpecReason = "InvalidSpec"
	// MissingFieldMessage the message to indicate rollout is missing a field
	MissingFieldMessage = "Rollout has missing field '%s'"
	// RolloutSelectAllMessage the message to indicate that the rollout has an empty selector
	RolloutSelectAllMessage = "This rollout is selecting all pods. A non-empty selector is required."
	// AvailableReason the reason to indicate that the rollout is serving traffic from the active service
	AvailableReason = "AvailableReason"
	// NotAvailableMessage the message to indicate that the Rollout does not have min availability
	NotAvailableMessage = "Rollout does not have minimum availability"
	// AvailableMessage the message to indicate that the Rollout does have min availability
	AvailableMessage = "Rollout has minimum availability"
	// Reasons and Messages for rollout Progressing Condition
	// ReplicaSetUpdatedReason is added in a rollout when one of its replica sets is updated as part
	// of the rollout process.
	ReplicaSetUpdatedReason = "ReplicaSetUpdated"
	// RolloutProgressingMessage is added in a rollout when one of its replica sets is updated as part
	// of the rollout process.
	RolloutProgressingMessage = "Rollout %q is progressing."
	// ReplicaSetProgressingMessage is added in a rollout when one of its replica sets is updated as part
	// of the rollout process.
	ReplicaSetProgressingMessage = "ReplicaSet %q is progressing."
	// FailedRSCreateReason is added in a rollout when it cannot create a new replica set.
	FailedRSCreateReason = "ReplicaSetCreateError"
	// FailedRSCreateMessage is added in a rollout when it cannot create a new replica set.
	FailedRSCreateMessage = "Failed to create new replica set %q: %v"
	// NewReplicaSetReason is added in a rollout when it creates a new replica set.
	NewReplicaSetReason = "NewReplicaSetCreated"
	//NewReplicaSetMessage is added in a rollout when it creates a new replicas set.
	NewReplicaSetMessage = "Created new replica set %q"
	// NewReplicaSetDetailedMessage is a more detailed format message
	NewReplicaSetDetailedMessage = "Created ReplicaSet %s (revision %d)"
	// FoundNewRSReason is added in a rollout when it adopts an existing replica set.
	FoundNewRSReason = "FoundNewReplicaSet"
	// FoundNewRSMessage is added in a rollout when it adopts an existing replica set.
	FoundNewRSMessage = "Found new replica set %q"
	// RolloutUpdatedReason indicates the rollout spec was updated.
	RolloutUpdatedReason = "RolloutUpdated"
	// RolloutUpdatedMessage indicates the rollout spec was updated.
	RolloutUpdatedMessage = "Rollout updated to revision %s"
	// RolloutCompletedReason is added in a rollout when it is completed.
	RolloutCompletedReason = "RolloutCompleted"
	// RolloutCompletedMessage is added when the rollout is completed
	RolloutCompletedMessage = "Rollout completed update to revision %d (%s): %s"
	// RolloutNotCompletedReason is added in a rollout when it is completed.
	RolloutNotCompletedReason = "RolloutNotCompleted"
	// RolloutNotCompletedMessage is added when the rollout is completed
	RolloutNotCompletedMessage = "Rollout not completed, started update to revision %d (%s)"
	// RolloutHealthyReason is added in a rollout when it is healthy.
	RolloutHealthyReason = "RolloutHealthy"
	// RolloutHealthyMessage is added when the rollout is completed and is healthy or not.
	RolloutHealthyMessage = "Rollout is healthy"
	// RolloutNotHealthyMessage is added when the rollout is completed and is healthy or not.
	RolloutNotHealthyMessage = "Rollout is not healthy"
	// RolloutAbortedReason indicates that the rollout was aborted
	RolloutAbortedReason = "RolloutAborted"
	// RolloutAbortedMessage indicates that the rollout was aborted
	RolloutAbortedMessage = "Rollout aborted update to revision %d"
	// RolloutRetryReason indicates that the rollout is retrying after being aborted
	RolloutRetryReason = "RolloutRetry"
	// RolloutRetryMessage indicates that the rollout is retrying after being aborted
	RolloutRetryMessage = "Retrying Rollout after abort"
	// RolloutPausedReason is added in a rollout when it is paused. Lack of progress shouldn't be
	// estimated once a rollout is paused.
	RolloutPausedReason = "RolloutPaused"
	// RolloutPausedMessage is added in a rollout when it is paused. Lack of progress shouldn't be
	// estimated once a rollout is paused.
	RolloutPausedMessage = "Rollout is paused"
	// ReplicaSetNotAvailableReason is added when the replicaset of an rollout is not available.
	// This could happen when a fully promoted rollout becomes incomplete, e.g.,
	// due to pod restarts, evicted -> recreated. In this case, we'll need to reset the rollout's
	// condition to `PROGRESSING` to avoid any timeouts.
	ReplicaSetNotAvailableReason = "ReplicaSetNotAvailable"
	// RolloutResumedReason is added in a rollout when it is resumed. Useful for not failing accidentally
	// rollout that paused amidst a rollout and are bounded by a deadline.
	RolloutResumedReason = "RolloutResumed"
	// RolloutResumedMessage is added in a rollout when it is resumed. Useful for not failing accidentally
	// rollout that paused amidst a rollout and are bounded by a deadline.
	RolloutResumedMessage = "Rollout is resumed"
	// RolloutStepCompleted indicates when a canary step has completed
	RolloutStepCompletedReason = "RolloutStepCompleted"
	RolloutStepCompletedMessage = "Rollout step %d/%d completed (%s)"
	// TrafficWeightUpdated is emitted any time traffic weight is modified
	TrafficWeightUpdatedReason = "TrafficWeightUpdated"
	TrafficWeightUpdatedMessage = "Traffic weight updated %s"
	// NewRSAvailableReason is added in a rollout when its newest replica set is made available
	// ie. the number of new pods that have passed readiness checks and run for at least minReadySeconds
	// is at least the minimum available pods that need to run for the rollout.
	NewRSAvailableReason = "NewReplicaSetAvailable"
	// RolloutAnalysisRunFailedReason is added in a rollout when the analysisRun owned by a rollout fails or errors out
	RolloutAnalysisRunFailedReason = "AnalysisRunFailed"
	// RolloutAnalysisRunFailedMessage is added in a rollout when the analysisRun owned by a rollout fails or errors out
	RolloutAnalysisRunFailedMessage = "AnalysisRun '%s' owned by the Rollout '%q' failed."
	// RolloutExperimentFailedReason is added in a rollout when the analysisRun owned by a rollout fails to show any progress
	RolloutExperimentFailedReason = "ExperimentFailed"
	// RolloutExperimentFailedMessage is added in a rollout when the experiment owned by a rollout fails to show any progress
	RolloutExperimentFailedMessage = "Experiment '%s' owned by the Rollout '%q' has timed out."
	// TimedOutReason is added in a rollout when its newest replica set fails to show any progress
	// within the given deadline (progressDeadlineSeconds).
	TimedOutReason = "ProgressDeadlineExceeded"
	// RolloutTimeOutMessage is added in a rollout when the rollout fails to show any progress
	// within the given deadline (progressDeadlineSeconds).
	RolloutTimeOutMessage = "Rollout %q has timed out progressing."
	RolloutDeletedReason = "RolloutDeleted"
	RolloutDeletedMessage = "Rollout %s/%s is deleted."
	ScalingReplicaSetReason = "ScalingReplicaSet"
	ScalingReplicaSetMessage = "Scaled %s ReplicaSet %s (revision %d) from %d to %d"
	// ReplicaSetTimeOutMessage is added in a rollout when its newest replica set fails to show any progress
	// within the given deadline (progressDeadlineSeconds).
	ReplicaSetTimeOutMessage = "ReplicaSet %q has timed out progressing."
	// ReplicaSetCompletedMessage is added when the rollout is completed
	ReplicaSetCompletedMessage = "ReplicaSet %q has successfully progressed."
	// ServiceReferenceReason is added to a Rollout when there is an error with a Service reference
	ServiceReferenceReason = "ServiceReferenceError"
	// ServiceReferencingManagedService is added in a rollout when the multiple rollouts reference a Rollout
	ServiceReferencingManagedService = "Service %q is managed by another Rollout"
	// TargetGroupHealthyReason is emitted when target group has been verified
	TargetGroupVerifiedReason = "TargetGroupVerified"
	TargetGroupVerifiedRegistrationMessage = "Service %s (TargetGroup %s) verified: %d endpoints registered"
	TargetGroupVerifiedWeightsMessage = "Service %s (TargetGroup %s) verified: canary weight %d set"
	// TargetGroupHealthyReason is emitted when target group has not been verified
	TargetGroupUnverifiedReason = "TargetGroupUnverified"
	TargetGroupUnverifiedRegistrationMessage = "Service %s (TargetGroup %s) not verified: %d/%d endpoints registered"
	TargetGroupUnverifiedWeightsMessage = "Service %s (TargetGroup %s) not verified: canary weight %d not yet set (current: %d)"
	// TargetGroupVerifyErrorReason is emitted when we fail to verify the health of a target group due to error
	TargetGroupVerifyErrorReason = "TargetGroupVerifyError"
	TargetGroupVerifyErrorMessage = "Failed to verify Service %s (TargetGroup %s): %s"
	// WeightVerifyErrorReason is emitted when there is an error verifying the set weight
	WeightVerifyErrorReason = "WeightVerifyError"
	WeightVerifyErrorMessage = "Failed to verify weight: %s"
	// LoadBalancerNotFoundReason is emitted when load balancer can not be found
	LoadBalancerNotFoundReason = "LoadBalancerNotFound"
	LoadBalancerNotFoundMessage = "Failed to find load balancer: %s"
	)

[martinimartinimartini]

Hey guys!
I'm configuring notifications for rollouts as well but it doesn't work in "vanilla" way.

I guess I'm suppose to create a when statement for each trigger right?

  triggers:
    trigger.on-rollout-completed: |
      - when: rollout.status.conditions[0].reason == 'RolloutCompleted'
        send: 
          - rollout-completed

[ervinb]

Yes, it's best to configure them explicitly.

The log from your argo-rollouts might look something like this:

Trigger on-rollout-step-completed result: [{[0].2jmj7l5rSw0yVb_vlWAYkK_YBwk  [rollout-step-completed] false}]

• 2jmj7l5rSw0yVb_vlWAYkK_YBwk is the hash of the provided condition (the when); in this case the condition was an empty string (default). You can test it here in this Go Playground
• [rollout-step-completed] the event which was recorded
• false, the state of the notification: triggered or not

A notification is sent in 2 cases:

1. Condition is not empty AND the trigger state is true
2. Condition was empty (like in the log above), regardless of the state
   • there's a caveat to this tough in notifications-engine: only the false => true state transition sends an actual notification. This can be bypassed/reconfigured by the oncePer field

[IstvanCsVarga]

This was a tremendous help. Thanks a lot!

[martinimartinimartini]

Are you guys been able to create a trigger like this?

  trigger.on-rollout-aborted: |
    - send: [rollout-aborted]

When I configure it and try to get the trigger list:

kubectl exec -it -n argocd argocd-notifications-controller-xxxxxxxxxxx -- /app/argocd-notifications trigger get
failed to parse config: unexpected token EOF (1:1)

How to solve it ?