Skip to content

Commit

Permalink
feat: introduce SDF tree placeholder and sidebar support (#26)
Browse files Browse the repository at this point in the history
  • Loading branch information
LeoBorai authored Jun 3, 2024
1 parent ad004ef commit ca803ac
Show file tree
Hide file tree
Showing 106 changed files with 105 additions and 13 deletions.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
2 changes: 2 additions & 0 deletions docs/quickstart.md → docs/fluvio/quickstart.mdx
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
---
sidebar_position: 1
title: "Quickstart"
description: "Getting started with Fluvio in a few minutes."
slug: /
---

Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
sidebar_position: 11
title: "Amplitude Analytics"
description: "Data pipeline that collects events from multiple services and sends them to Amplitude."
slug: /tutorials/amplitude-analytics
---

With this guide, you can send events from InfinyOn Cloud to Amplitude. Connecting your services to this pipeline is simpler than microservices running around
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
sidebar_position: 11
title: "Cloudflare Workers"
description: "How to create a Cloudflare worker that sends events to a Webhook in InfinyOn Cloud."
slug: /tutorials/cloudflare-workers
---

# Integrate Cloudflare Workers with InfinyOn Cloud
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
sidebar_position: 9
title: "Custom Clients with Docker"
description: "A guide to run InfinyOn Cloud client in Docker environment"
slug: /tutorials/custom-clients-with-docker
---

In this guide, you'll learn how to package your custom app with a Fluvio client into a Docker image, and how to run a container with `docker` CLI.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
sidebar_position: 6
title: "GitHub to Discord"
description: "Data pipeline that detects changes in GitHub and publishes them as events to Discord."
slug: /tutorials/github-to-discord
---

Create an end-to-end event pipeline that detects changes in github stars & forks and publishes the result to Discord. This guide uses two connectors:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
sidebar_position: 5
title: "GitHub to Slack"
description: "Data pipeline that detects changes in GitHub and publishes them as events to Slack."
slug: /tutorials/github-to-slack
---

Create an end-to-end event pipeline that detects changes in github stars & forks and publishes the result to Slack. This guide uses two connectors:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
sidebar_position: 1
title: "Hackernews Reader"
description: "Data pipeline that periodically reads articles from Hackernews and publishes them on a topic."
slug: /tutorials/hackernews-reader
---

Hackernews Reader helps you build an XML reader that ingests hackernews articles, converts them to `json`, divides them into records, and publishes each record to a topic. This guide uses the following connector:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
sidebar_position: 3
title: "HTTP to SQL"
description: "Data pipeline that periodically reads from a websitea and sends the result to a SQL database."
slug: /tutorials/http-to-sql
---

This guide expects you to already have the Fluvio CLI installed, and InfinyOn Cloud set up. If neither of these is the case, please follow the instructions [here][docs]!
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
sidebar_position: 7
title: "IoT Mirroring - Cloud"
description: "Reliable IoT monitoring from movable Edges sensors with poor connections to Cloud."
slug: /tutorials/iot-mirroring-cloud
---

InfinyOn IoT edge is a **~14 Mb** binary that runs on ARMv7 chips on less than **256 MB** memory. We are working with teams building the future of monitoring dynamic assets to push the boundaries of edge data stream processing.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
sidebar_position: 8
title: "IoT Mirroring - Local"
description: "Reliable IoT monitoring from movable or static Edges sensors."
slug: /tutorials/iot-mirroring-local
---

This advanced tutorial reuqires a Raspberry Pi and a local installation of your collector cluster running on Kubernetes.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
sidebar_position: 4
title: "MQTT to SQL"
description: "Data pipeline that listens for mqtt events, converts them to json, and sends them to a SQL database."
slug: /tutorials/mqtt-to-sql
---

At the end of this tutorial, we will see data starting from an MQTT broker and ending in a PostgreSQL table.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
sidebar_position: 2
title: "Webhook to Slack"
description: "Event data pipeline that recevies events from an InfinyOn Cloud Webhook and sends an alert to Slack"
slug: /tutorials/webhook-to-slack
---

This guide shows an end-to-end event pipeline that reads an event from a webhook, generates a formatted string, and publishes the result to Slack. While this is a simple example, it has many event notification use cases, such as:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
sidebar_position: 10
title: "Zapier Triggers"
description: "This guide describes how to send events back and forth between InfinyOn Cloud and Zapier"
slug: /tutorials/zapier-triggers
---

# How to Link Zapier and InfinyOn Cloud
Expand Down
64 changes: 64 additions & 0 deletions docs/sdf/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
---
sidebar_position: 1
title: "Stateful Dataflows Overview"
description: "Stateful Dataflows are data pipelines that integrate fluvio event streaming with stateful processing to automate real-time data analytics at scale."
slug: /sdf
---

Stateful Dataflows, the embodiment of versatility in data pipelines, seamlessly integrate fluvio event streaming with stateful processing, sparking innovation among data engineers. These pipelines, built-in Rust and powered by WebAssembly, are small, fast, and incredibly versatile. They empower engineers to write custom logic snippets that compile to WebAssembly and inject them into the pipeline for in-line processing. This custom logic can perform various tasks, from data transformations and error correction to malicious payload detection and complex stateful processing.

The following example is a dataflow that performs various operations on data from sensors camera sensors at different locations.

To run this dataflow, follow the instructions in <a href="https://github.com/infinyon/stateful-dataflows-examples/tree/main/dataflows-inline/car-processing" target="_blank">github</a>.


#### Stateful Dataflows vs. Big Data Stream Processing

Unlike traditional big data frameworks built in Java, like Kafka, Flink, KStream, and Spark, where each component is independently managed, scaled, and built into dataflows through external Microservices, InfinyOn Stateful Dataflows introduces a unique paradigm. This paradigm empowers users to quickly develop and test individual services in their favorite programming language. It seamlessly integrates them into scalable end-to-end dataflows, streamlining the entire data processing workflow.


#### Stateful Dataflows vs. Legacy Solutions

Automating data operations within legacy technology stacks, spanning message brokers, databases, microservices, and batch jobs, typically demands months of setup and years of experimentation before yielding positive outcomes. InfinyOn Stateful Dataflows frees you from infrastructure intricacies and lets you focus on your core business logic instead.


#### Who is this for?

This platform is tailored for developers creating event-driven applications with continuous enrichment. The product streamlines the composition of dataflows with external sources such as databases, AI/ML models, and Redis caches, producing powerful results for analytics, application, and operational dashboards.


#### How are Stateful Dataflows different from Fluvio?

Stateful Dataflows are an extension of Fluvio, leveraging the Fluvio infrastructure for communication and persistence. Fluvio is responsible for connectors, data streaming, and persistence, whereas dataflows handle data routing, transformations, and stateful processing.


#### How do I integrate this into my existing data architecture?

Fluvio connectors serve as the interface to external ingress and egress services. Fluvio publishes a growing library of connectors, such as HTTP, Kafka, NATs, SQL, S3, etc. Connectors are easy to build, test, deploy, and share with the community.

#### How do I get started?

Provisioning and operating a Stateful Dataflow requires the following system components:

1. [Fluvio Cluster] to connect the dataflows with data streaming.

2. [Dataflow File] to define the schema, composition, services, and operations.

3. [SDF (Stateful Dataflows) CLI] to build, test, and deploy the dataflows.

The Stateful Dataflows can be built, tested, and run locally during preview releases. As we approach general availability, they can also be deployed in your InfinyOn Cloud cluster. In addition, the dataflows may be published to Hub and shared with others with one click and installation.


## Next Steps

In the [next section], we'll walk through the steps to get started with Stateful Dataflows.

[Let's Get Started].



[Dataflow File]:
[Fluvio Cluster]:
[SDF (Stateful Dataflows) CLI]:
[next section]:
[Let's Get Started]:
17 changes: 17 additions & 0 deletions docusaurus.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -84,13 +84,30 @@ const config: Config = {
themeConfig: {
// Replace with your project's social card
image: "img/docusaurus-social-card.jpg",
docs: {
sidebar: {
autoCollapseCategories: true,
}
},
navbar: {
logo: {
alt: "Fluvio Logo",
src: "img/fluvio-with-text.svg",
srcDark: "img/fluvio-with-text-white.svg",
},
items: [
{
type: 'doc',
position: 'left',
docId: 'fluvio/quickstart',
label: 'Docs',
},
{
type: 'doc',
position: 'left',
docId: 'sdf/index',
label: 'SDF',
},
{
href: FLUVIO_REPOSITORY_URL,
label: "GitHub",
Expand Down
23 changes: 10 additions & 13 deletions sidebars.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type {SidebarsConfig} from '@docusaurus/plugin-content-docs';
import type { SidebarsConfig } from "@docusaurus/plugin-content-docs";

/**
* Creating a sidebar enables you to:
Expand All @@ -11,21 +11,18 @@ import type {SidebarsConfig} from '@docusaurus/plugin-content-docs';
Create as many sidebars as you want.
*/
const sidebars: SidebarsConfig = {
// By default, Docusaurus generates a sidebar from the docs folder structure
tutorialSidebar: [{type: 'autogenerated', dirName: '.'}],

// But you can create a sidebar manually
/*
tutorialSidebar: [
'intro',
'hello',
fluvio: [
{
type: "autogenerated",
dirName: "fluvio",
},
],
sdf: [
{
type: 'category',
label: 'Tutorial',
items: ['tutorial-basics/create-a-document'],
type: "autogenerated",
dirName: "sdf",
},
],
*/
};

export default sidebars;

0 comments on commit ca803ac

Please sign in to comment.