Seeking efficient ways to load data for multiple modules onto single lyd_node datastore

[sensankum]

Hi everyone,

I am seeking advice and insights on the most efficient and correct method to load data for multiple modules into a single datastore lyd_node, namely cm_datastore. Currently, I'm utilizing a specific logic embedded in the cm_load_data() function, and I’m uncertain if this is the optimal approach. Below is the code snippet.

static void
cm_load_data(struct ly_ctx *ctx, struct lyd_node **cm_datastore)
{
    struct lyd_node *src_tree;
    char *data;
    LY_ERR ret;

    for (int i = 0; i < g_num_modules; i++) {
        data = cm_get_data(g_modules[i]);
        if (!data) {
            continue;
        }

        ret = lyd_parse_data_mem(ctx, data, LYD_JSON, LYD_PARSE_ONLY,
                        LYD_VALIDATE_NO_STATE, &src_tree);
        free(data);

        if (ret != LY_SUCCESS) {
            continue;
        }

        if (!*cm_datastore) {
            *cm_datastore = src_tree;
        }
        else {
            lyd_merge_tree(cm_datastore, src_tree, LYD_MERGE_DESTRUCT);
        }
    }
}

int main()
{
    struct ly_ctx *ctx;
    struct lyd_node *cm_datastore = NULL;
    LY_ERR ret;
    clock_t otic, otoc;
    int rc;

    cm_ctx_create(&ctx);
    cm_load_data(ctx, &cm_datastore);
    ret = lyd_validate_all(&cm_datastore, ctx, LYD_VALIDATE_PRESENT, NULL);
    fprintf(stderr, "Validation status: %d", ret);
}

cm_load_data() iteratively walks through all the modules of interest (denoted by g_num_modules), retrieves the JSON-formatted data from each module by invoking cm_get_data(), parses the data into src_tree, and eventually merges src_tree into cm_datastore.

My primary concern is the potential inefficiency of creating data nodes (dnodes) for every module data twice:

• Once, during the parsing of the JSON data with lyd_parse_data_mem().
• A second time, during the merging process with lyd_merge_tree(), where src_tree dnodes are merged into cm_datastore.

Am I mistaken in my understanding? If not, is there a more efficient way to achieve my goal?

Thank you for your assistance.

[michalvasko]

First question, what is cm? I do not know those functions.

A second time, during the merging process with lyd_merge_tree(), where src_tree dnodes are merged into cm_datastore.

The nodes are not created again in this step and if using the flag LYD_MERGE_DESTRUCT (which you are), this operation is really fast. So you are probably fine doing what you are doing.

[sensankum]

Thanks for the quick response. cm_*() are my own configuration mgmt (cm) functions.
If there is a case to call merge without LYD_MERGE_DESTRUCT, will the node still not be created again while merging ?

[michalvasko]

If you do not use the flag, the missing nodes need to be duplicated so technically created.

[sensankum]

Thank you for the clarifying. Really appreciate your time on this.