Is vineyard the only solution for data transfer?

[cfangplus]

Hi,

We know that vineyard is supposed to act as a role for data transfer among the three graph systems and those transferred data is distributed in memory. Also we know that there exists many, many products that could support similar function, such as Redis, Alluxio, or even graph database to some extent could be ok for data transfer among the graph systems.
So the question is, is vineyard the only solution for this? Could you help to explain what's the real difference between vineyard and the other similar product?

[sighingnow]

Hi @cfangplus,

Thanks for your interest on GraphScope!

(I have moved the issue to a Q&A discussion to make it visible for more people.)

We know that vineyard is supposed to act as a role for data transfer among the three graph systems and those transferred data is distributed in memory.

GraphScope leverages vineyard to storage large-scale distributed graphs in memory across a cluster. But we DON'T _transfer) data among the three graph systems. Rather, GAE, GIE and GLE in GraphScope shares the graph data in a zero-copy fashion by memory sharing. Those engines refer to the same graph object when performing specific graph computations.

many products that could support similar function, such as Redis, Alluxio, or even graph database to some extent could be ok for data transfer among the graph systems.

• Redis: redis is a KV storage as far as I know. Rather, vineyard address on in-memory sharing for both low level data structures as well
  as high level data structure abstractions.
• Alluixo: vineyard isn't a cache layer for filesystem-like storage.
• GraphScope is not a graph database. GraphScope aims to provides a one-stop graph computation platform for different kinds of workload on large-scale graph data.

is vineyard the only solution for this?

As described above, we actually don't do data transfer between those engines.

Even when considering scale in/out and the following data migration, vineyard is not the solution for data transfer, rather, the data transfer will be handled by a set of vineyard "drivers" (for more information of the vineyard drivers, please refer to vineyard documentation, and this discussion thread).

Hope the above could answer you question. Free feel to comment here if you have any other questions 😆

[Zarca]

Cool and awesome C++ project!
After checking the core data loading process of GAE in en_fragment_loader:
https://github.com/alibaba/libgrape-lite/blob/7903afded169ead2a7cd88cef71130d0d75f4b97/grape/fragment/ev_fragment_loader.h#L101

I assume that all GAE server instances will need a complete full vertex file, and a partition of the edge file(even split) . Then it can hold a subgraph of the full graph (all nodes, but part edges ) , and began the algorithm like SSSP,PageRank in a parallelized way , which is discussed in the paper:
https://dl.acm.org/doi/pdf/10.1145/3035918.3035942.
But, if this is the true, i still get struggled about how can the subgraph inside each of the GAE's instance be recognized by GLE? It looks like that the GLE must have the same ability to treat a distributed subgraphs like GAE.
(which i think can't be handled by just using the same set of C++ API of vineyard of ArrowFragment)

[sighingnow]

all nodes, but part edges

Not all nodes. Nodes are partitioned using hash.

GAE's instance be recognized by GLE

GLE does have the the ability to processing distributed graphs.

i think can't be handled by just using the same set of C++ API of vineyard of ArrowFragment

Why cannot ?

[Zarca]

Hi,
i do check again the core loading process, and now i believe it is indeed every node has been hashed and assigned to different fragments.
As for this puzzle:

i think can't be handled by just using the same set of C++ API of vineyard of ArrowFragment

Why cannot ?

What I'm trying to say is that, considered we have a GAE's distributed graph here(using MPI to send nodes and edges to the corresponding fragment's machine). But when it comes to GLE, unless the GLE itself has exact the same message system design as the GAE (and exact the same graph building way like GAE's ev_fragment_loader do), otherwise, I think it can't do computation on the Graph DataStructure created by GAE directly.

For instance, if the GLE is aim to launch a GCN , which will aggregate the node's neighbors from other machines,
must it call some API from GAE to get such message from remote subgraph(or the so called "fragment")?

Off course for me, I think this part is one of the key parts of the whole GraphScope's brilliant innovation, but i also think it is difficult to understand.

Cause I'm thinking how i can adapt some third party's GAE or GLE or GIE to shared the same distributed graph even the message system, like you did in GraphScope.

[Zarca]

In a word, i.e. how to adapt a distributed graph defined and split by GAE's logics to another engine (like GLE )？
For instance, the distributed graph may have nodes hashed to different machine, so if a GLE engine wanna to reuse this graph, it's partition logics must be the very same as GAE .
Is it that true? (GLE and GAE have exactly the same graph partition strategy ?, or the GLE has to repartition a new graph from the GAE's output?)

[sighingnow]

GLE and GAE have exactly the same graph partition strategy ?

They shares the same graph data, so, the partition is same as well.

[Zarca]

Do GLE also follows the same distributed graph model's communication mode (i.e the same partition strategy, the PEval, the IncEva and Assemble communication mode ) as GAE(GRAPE)'s ?
Or, the GAE can simulate another parallel model of other engines, without a reconstruction of graph?（like Grape's paper said）

[sighingnow]

So did the gremline engine (GIE) also can share and recognize the same graph partition (like GAE and GLE did)?

Definitely yes.

Otherwise, it has to rebuild the graph itself.

The answer is yes as well. But we promise our implementation for graph data structure is highly-optimized and yields pretty good performance.

[Zarca]

Thanks for all the kind and patient relies , now i got know the awesome things and the data flow outlines behind GraphScope :)

[Zarca]

B.T.W. , Finally, I suppose the GAE's graph partition strategy is a edge cut way by default,（through MPI）,
Is that true? can a vertex cut strategy be applied to GraphScope?

[sighingnow]

Currently the fragment is partitioned as edge cut. Vertex cut will be released later (may before Sep, 2021).

[Zarca]

Got it 🥇