forked from finos/morphir
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request finos#1 from mcleo-d/docs-build
Fix Docusaurus broken Netlify build
- Loading branch information
Showing
9 changed files
with
659 additions
and
845 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| --- | ||
| name: 🤝 Morphir Meeting Agenda | ||
| about: To track Morphir meeting agenda and attendance | ||
| --- | ||
|
|
||
| ## Date | ||
|
|
||
| Thursday DD MMM yyyy - 11am ET / 3pm UK | ||
|
|
||
| ## Untracked attendees | ||
|
|
||
| | Name | Firm | Comment | | ||
| | :--- | :--- | :------ | | ||
|
|
||
| ## Meeting notices | ||
|
|
||
| - FINOS **Project leads** are responsible for observing the FINOS guidelines for [running project meetings](https://community.finos.org/docs/governance/meeting-procedures/). Project maintainers can find additional resources in the [FINOS Maintainers Cheatsheet](https://community.finos.org/docs/finos-maintainers-cheatsheet). | ||
|
|
||
| - **All participants** in FINOS project meetings are subject to the [LF Antitrust Policy](https://www.linuxfoundation.org/antitrust-policy/), the [FINOS Community Code of Conduct](https://community.finos.org/docs/governance/code-of-conduct) and all other [FINOS policies](https://community.finos.org/docs/governance/#policies). | ||
|
|
||
| - FINOS meetings involve participation by industry competitors, and it is the intention of FINOS and the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. Please contact [email protected] with any questions. | ||
|
|
||
| - FINOS project meetings may be recorded for use solely by the FINOS team for administration purposes. In very limited instances, and with explicit approval, recordings may be made more widely available. | ||
|
|
||
| ## Agenda | ||
|
|
||
| - [ ] Convene, roll call, welcome new people (5mins) | ||
| - [ ] Display [FINOS Antitrust Policy summary slide](https://community.finos.org/Compliance-Slides/Antitrust-Compliance-Slide.pdf) | ||
| - [ ] Review Meeting Notices (see above) | ||
| - [ ] Approve previous meeting minutes - https://github.com/finos/morphir/issues?q=is%3Aissue+is%3Aclosed+label%3Ameeting | ||
| - [ ] _Add Items Here_ | ||
| - [ ] AOB, Q&A & Adjourn (5mins) | ||
|
|
||
| ## Decisions Made | ||
|
|
||
| - [ ] Decision 1 | ||
| - [ ] Decision 2 | ||
| - [ ] ... | ||
|
|
||
| ## Action Items | ||
|
|
||
| - [ ] Action 1 | ||
| - [ ] Action 2 | ||
| - [ ] ... | ||
|
|
||
| ### Zoom info | ||
|
|
||
| - https://zoom.us/j/95452540437?pwd=TEk0bitJcVFsK3lrWVZCYmxyTm5LZz09 | ||
| - **Meeting ID:** 954 5254 0437 | ||
| - **Passcode:** 778814 | ||
|
|
||
| ### Dial-in | ||
|
|
||
| - **Find your local number:** https://zoom.us/u/aFGSSGV9x | ||
|
|
||
| **Github Repo:** https://github.com/finos/morphir/ | ||
|
|
||
| **Mailing List:** Email [email protected] to subscribe to our mailing list |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
72 changes: 72 additions & 0 deletions
72
...s/0001-implement-the-spark-backend-as-a-special-case-of-a-relational-backend.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| # Implement the Spark backend as a special case of a Relational backend | ||
|
|
||
| ## Context and Problem Statement | ||
|
|
||
| Given that Apache Spark's data model is an extended relational data model and most of the complexity in the backend will likely be related to the differences between the FP and the relational model (like flattening data structures, truning lookups into joins, ...), should we consider implementing the Spark backend as a special case of a Relational backend? This way we would be able to reuse the solutions we come up with in this process across a wide variety of backends including various relational databases or technologies like Apache Kafka SQL (any technology that is fundamentally relational). | ||
|
|
||
| ## Decision Drivers | ||
|
|
||
| - Future potential/limitations | ||
| - Implementation effort | ||
|
|
||
| ## Considered Options | ||
|
|
||
| - Map to Relational IR first, then to Spark IR | ||
| - Map directly to Spark IR | ||
| - Map directly to Spark IR, but implement flatten and joins as reusable functions | ||
|
|
||
| ## Decision Outcome | ||
|
|
||
| Chosen option: "{option 1}", because {justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force {force} | … | comes out best (see below)}. | ||
|
|
||
| ### Positive Consequences <!-- optional --> | ||
|
|
||
| - {e.g., improvement of quality attribute satisfaction, follow-up decisions required, …} | ||
| - … | ||
|
|
||
| ### Negative Consequences <!-- optional --> | ||
|
|
||
| - {e.g., compromising quality attribute, follow-up decisions required, …} | ||
| - … | ||
|
|
||
| ## Pros and Cons of the Options <!-- optional --> | ||
|
|
||
| ### Map to Relational IR first, then to Spark IR | ||
|
|
||
| ```mermaid | ||
| graph LR | ||
| A[Morphir IR] --> A2B(Relational Backend) --> B[Relational IR] | ||
| B --> B2C(Spark Backend) --> C[Spark IR] | ||
| ``` | ||
|
|
||
| - Good, because it makes the relational mapping reusable | ||
| - Bad, because the Relational IR limits the scope of Spark operations we can map to | ||
|
|
||
| ### Map directly to Spark IR | ||
|
|
||
| ```mermaid | ||
| graph LR | ||
| A[Morphir IR] --> B2C | ||
| B2C(Spark Backend) --> C[Spark IR] | ||
| ``` | ||
|
|
||
| - Good, because we can fully utilize Spark's capabilities to implement all possible Morphir features | ||
| - Bad, because we cannot reuse the solutions we come up with for flattening and joins | ||
|
|
||
| ### Map directly to Spark IR, but implement flatten and joins as reusable functions | ||
|
|
||
| ```mermaid | ||
| graph LR | ||
| A[Morphir IR] --> B2C | ||
| B2C(Spark Backend) --> C[Spark IR] | ||
| B2C -.-> F[flatten] | ||
| B2C -.-> J[join] | ||
| ``` | ||
|
|
||
| - Good, because we can fully utilize Spark's capabilities to implement all possible Morphir features | ||
| - Good, because we can reuse the solutions we come up with for flattening and joins | ||
|
|
||
| ## Links <!-- optional --> | ||
|
|
||
| - {Link type} {Link to ADR} <!-- example: Refined by ADR-0005 in 0005-example.md --> | ||
| - … <!-- numbers of links can vary --> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -99,3 +99,4 @@ In Morphir's data format this would translate into something like this: | |
| # Security | ||
| # License | ||
| --> | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.