Skip to content

Latest commit

 

History

History
237 lines (187 loc) · 13.8 KB

5.6. Messaging Services.md

File metadata and controls

237 lines (187 loc) · 13.8 KB
Raw

Messaging Services

  • Includes Azure Storage Queues, Service Bus Queues, Service Bus Relay, IoT Hubs, Event Hubs, and Notification Hubs
  • Integration services include Azure Functions and Logic Apps.

Event Messaging

Storage Queues

Storage Components

  • Storage account
  • URL: E.g. http://[account].queue.core.windows.net/queue
  • Queue: A queue contains a all of messages.
  • Message: A message, in any format, of up to 64KB.

Storage Queue Message Handling

  • 💡 Processors/consumers must be designed to not have side effects from processing the same message multiple times.
    • E.g. if it inserts to SQL it should check first if record exists.
    • Service Bus Queues on the other hand guarantees that a message is handled at least and most by single customer.
  • Operations:
    • Create/Delete Queue: Client SDKs, PowerShell or the REST API
    • Measure Queue Length
      • You can get an estimate of the number of messages in the queue
      • ❗ This count is not guaranteed and should be treated in your application as an approximate queue length.
    • Insert Message into Queue: Either a string (in UTF-8 format) or a byte array.
    • Retrieve the Next Message
      • A copy is retrieved
      • Message is made invisible for a specific duration
        • After duration the message is available to other queue consumers.
    • Extend Message Lease: You can return to the queue and update the invisibility duration for the queue message
    • Peek at the Next Message: Does not make message invisible
    • Update a Message
      • The contents of a retrieved message can be updated in the queue
      • Used e.g. with checkpoints or state for queue messages.
    • Delete a Message: Otherwise invisible will timeout and message will be reprocessed.

Service Bus

  • Infrastructure for communication, event distribution, naming and service publishing.
  • Provides connectivity options for Windows Communication Foundation (WCF) and other service endpoints e.g. REST
    • Endpoints can be located behind network address translation (NAT) boundaries, or bound to frequently-changing, dynamically-assigned IP addresses, or both.
  • Partitioned into namespaces as each namespace provide both a service and security boundary.
  • Uses pull-model
    • 💡 Can be integrated with Event grid to have push model as event grid uses push model

Messaging capabilities

Relayed messaging pattern (relays)
  • Direct one-way messaging
  • Request/response messaging
  • Peer-to-peer messaging
Brokered messaging pattern
  • E.g. topics (one-to-many), queues (one-to-many), Event Hub.
  • Available through APIs and SDKs over HTTP.
  • Asynchronous messaging with Queues, Topics and Subscriptions
  • Publish-subscribe and temperal decoupling
    • Senders and receivers do not have to be online at the same time.
      • Messages are stored until they are received.

Service Bus Queues

  • Extends Storage queues.
  • Implements FIFO.
  • Guarantees that message is received and processed both at least and at most once by the message consumers
  • Flow: Message sender (e.g. web app, mobile app and service) => Namespace (contains queue) => Message receiver (service or application)

Service Bus Relay

  • Connects existing services to new client applications without exposing the true location or address of the service.
  • Supports direct peer-to-peer communication
  • Advantages / Use-cases:
    • On-prem resources can be revealed without them having a public IP address but using outbound connection.
    • Abstraction helps with having same URI pointing out different resources.
  • Flow: Application <=> Service Bus Relay <=> Client1, Client2

Event Grid

  • Single service designed to manage and route systemic events from any source service in your Azure subscription.
  • Uses push model
    • Removes need to polling
      • Triggers applications when needed
      • Pay per event: Consume compute only when necessary.
    • Can be integrated with e.g. Service Hub to pull the events and push it to e.g. functions
  • Can use custom workloads or pre-determined events with each Azure service.
  • Reliability through the use of a 24-hour retry with exponential back-off.
  • Example ways to use
    • Integration: Integrate various components of your workloads together using a publish-subscribe mode.
    • B2B: Enable your solution to listen to events from third-party B2B services of publish events for the third-party services to consume.
    • Compute: Create serverless compute that is triggered by a specific Azure service event such as the creation of a database or VM.
    • Ops automation work: Automate the deployment of resources by subscribing to ARM events.
      • E.g. a user can request Event Grid to notify Azure Automation when a virtual machine starts or a SQL Database is spun up.
        • Events can be then used to e.g.
          • Tag virtual machines
          • File work items
          • Put metadata into operations tools
          • Automatically check to ensure services configurations are correct.
    • Router: You can use filters to route specific events to different or even multiple endpoints.
      • You can route to a webhook endpoint or event handler.

Integration

Serverless Integration

  • Serverless applications: Azure Logic Apps, Azure Functions.
  • Development: IDE support, visual debug history, local development, verbose debugging.
  • Platform:
    • Functions: Developer Tooling, Bindings and triggers, open source
    • Logic Apps: Visual designer, 200+ connectors, functions orchestration
    • Even more such as data/storage, messaging, gateway connectors, intelligence, bots

Notification Hubs

  • Service infrastructure and a set of client libraries that allows you to publish push notifications.
  • Can send to specific user, all users or segmented users.
  • Abstracts the implementation of the various Platform Notification Systems (PNS) for each mobile platform with a single method call.
    • Devices are only responsible for registering PNS handles.
    • Backend is responsible for sending platform-independent messages to users or interest groups.
Notification Hubs Advantages
  • Multiple platforms
  • Works with any backend
  • Scale
  • Rich set of delivery patterns
  • Personalization: Each device can have one or more templates to achieve per-device localization and personalization without effecting the back-end code
Platform notification system flow
  1. Client application contacts the PNS to retrieve its handle.
  2. Application stores the handle for later usage.
  3. App back-end contacts the PNS by using handle to target a client application.
  4. PNS forwards notification to the device specified by the handle.
Notification hub flow
  1. Client reaches out to the PNS through Notification Hubs SDK and registers PNS handle.
  2. Alternatively, client sends PNS handle to application backend to have the application register the service.
  3. Application back-end sends message to Notification hubs service, service forwards it to appropriate target client by using their PNS handles.

Internet of Things (IoT)

Event Hubs

  • Event ingestion service
  • Uses pull-based model
    • Can be integrated with Event Grid to allow a push model
  • Can be used as a "front line" for an event pipeline of a solution architecture, sometimes known as an event investor
    • Event investor
      • a service or component
      • sits between the event consumers and publishers
      • separates the production of an event stream that obtained said events.
  • Capable of publishing via AMQP 1.0 or HTTPS
  • It can capture Event Hubs streaming data to store in an Azure Blob storage account.
    • Allows e.g. :
      • to each consumer to read a specific subset of the event stream
      • to consumer to act independently
  • Uses SAS (Shared Access Signature) tokens
    • Can identify and authenticate the event publisher though SAS tokens.
  • Flow: Event Producers --HTTP,AMQP--> Partitions -> Consumer Groups -> Event Recievers.
Event hubs pricing
  • Throughput units or Pre-purchased units of capacity
  • Throughput units can include the capacity of either 1 MB per second or 100 events per second
  • Up to to 2 MB per second if its egress
  • Units are billed hourly for a minimum of one hour, and up to a maximum of 20 throughput units per Event Hubs namespace.
Event Hubs scenarios
  • Process streams
  • Save in long/term storage e.g. service bus, Azure DBs, HDInsight, Azure Storage, Azure Data Lake
  • Present take actions through Power BI Dashboards, Search and query, Cortana analytics, Devices to take action

IoT Hubs

  • Provides reliable and secure bi-directional communication between a multitude of IoT devices with a solution back end.
  • Message routing option built in that sends the message to other Azure services
  • More advanced & device specific than Event Hubs
    • Can provide additional features such as Device twins, which can be used to store and research device state information.
    • Event Hubs handles only event ingress communication (device-to-cloud) while IoT provides device-to-cloud and cloud-to-device communication.
  • You can use an IoT Hub to receive messages from a device and then push them to a Logic App for processing
    • ❗ Up to 10 endpoints are supported.

IoT Hubs connection and security

  • Uses outbound connection only from devices.
  • The path between device and service must be secured at application protocol layer.
  • Requires gateway that can be either:
    • Protocol gateway
      • Always deployed in the cloud
      • With a protocol gateway protocol, translations are carried out, such as MQTT to AMQP.
    • Field gateway
      • Deployed locally with a device.
      • You can make time-sensitive decisions, run analytics on edge, provide device management services or even enforce security and privacy constraints.

Event Hubs vs IoT Hubs

Area IoT Hub Event Hubs
Communication patterns • Device-to-cloud communications (messaging, file uploads, and reported properties)_ • Cloud-to-device communications (direct methods, desired properties, messaging). Only enables event ingress (usually considered for device-to-cloud scenarios).
Device state information Device twins can store and query device state information. No device state information can be stored.
Device state protocol • Supports MQTT, MQTT over WebSockets, AMQP, AMQP over WebSockets, and HTTPS. • Works with the Azure IoT protocol gateway, a customizable protocol gateway implementation to support custom protocols, authentication, message transformations etc. Supports AMQP, AMQP over WebSockets, and HTTPS.
Security Provides per-device identity and revocable access control. See the Security section of the IoT Hub developer guide. Provides Event Hubs-wide shared access policies, with limited revocation support through publisher's policies. IoT solutions are often required to implement a custom solution to support per-device credentials and anti-spoofing measures.
Operations monitoring Enables IoT solutions to subscribe to a rich set of device identity management and connectivity events such as individual device authentication errors, throttling, and bad format exceptions. These events enable you to quickly identify connectivity problems at the individual device level. Exposes only aggregate metrics.
Scale Is optimized to support millions of simultaneously connected devices. Meters the connections as per Azure Event Hubs quotas. On the other hand, Event Hubs enables you to specify the partition for each message sent.
Device SDKs Provides device SDKs for a large variety of platforms and languages, in addition to direct MQTT, AMQP, and HTTPS APIs. Is supported on .NET, Java, and C, in addition to AMQP and HTTPS send interfaces.
File upload Enables IoT solutions to upload files from devices to the cloud. Includes a file notification endpoint for workflow integration and an operations monitoring category for debugging support. Not supported.
Route messages to multiple endpoints Rules determine how messages are routed to custom endpoints. ❗ Up to 10 custom endpoints are supported. Requires additional code to be written and hosted for message dispatching.

Time Series Insights

  • Built for visualizing, storing, and querying vast amounts of time series information, such as information generated by IoT devices.
  • Requires no upfront data preparation.
  • It can obtain and store millions of sensor events per day with a one-minute latency, giving it the ability to provide near real-time insights.

Time series data

  • Shows how an asset or process changes over time.
    • It has a timestamp that is unique to its kind which is most meaningful as an axis.
  • Arrives in time order and can usually be an insert rather than an update to your database.
  • Because it obtains and stores every new event as a row, the changes can be measured over a time frame, enabling one to not only look back into the past but also predict the future with the data.

Use cases for Time Series Insights

  • Storing and maintaining time series data in a scalable format.
  • Near real-time data visualization
    • When you connect an event source, the event data can be viewed, queried, and explored.
  • Producing customer application with REST API.
  • Global view of time series data streaming from different locations for multiple sites or assets
    • Allows you to connect multiple event sources to the environment