libyang 2.x

[rkrejci]

You might already notice that we are working on a new libyang major version in the libyang2 branch. I would like to clarify the reasons to rewrite libyang and how it is going to change.

Motivation

We are trying our best to fix the issues we find or which someone else reports via the issue tracker. YANG is really complex language - all the ways how one part of a YANG schema can be interconnected, extended, changed or removed by another YANG schema are quite challenging for implementation. By the time many of the corner cases are being clarified by the NETMOD WG community and we are striving to reflect such changes in libyang as well as to fix or add handling of overlooked corner cases.

Unfortunately, the maintenance and bug fixing became difficult mainly because of some specific features complicating many other parts of libyang. Another observation was that libyang could be faster and code could be more simple (better maintainable) if the design of some of the structures would be different. But such changes are so complex that they cannot be done by iterative changes. So we have decided to rewrite the libyang codebase. However, we are really satisfied with the overall design and various mechanisms in the current libyang. We would like to keep the most of the usage approaches, so besides changing some of the parameters or structure members, it shouldn't be necessary to redesign the whole code using libyang.

Changes

I have mentioned, that there are some features we want to get rid of. We actually believe that they were not used frequently and that they can be replaced by a sequence of other steps. This is the case of removing (or disabling) modules in the context. Due to the interconnection between YANG modules, the process of removing the YANG module is really complex and must be taken into consideration in many other parts of libyang. The common use case is to prepare context and then use it for work with YANG data, while no changes to the context are needed. And if it is necessary to remove some of the modules in the context, a new context with the required set of YANG modules can be created. Therefore, in libyang 2.x, it will not be possible to remove or disable a YANG module present in the context.

The second big change is about schema tree structures. Currently, libyang has just one type of schema tree structure used for all the purposes - for data validation and manipulation as well as for printing various schema formats. To speed up and simplify mainly the data validation and manipulation, we have decided to split schema structures into two types - parsed and compiled schema tree. The parsed tree preserves the original schema structure from the input data. There are groupings, typedefs, augments, and so on but the references such as uses statement are not resolved (not even validated). This type of schema tree is suitable for printing (transforming) schema tree formats and internally for resolving some of the references when the parsed schema tree is being compiled. In contrast, the compiled schema does not preserve the original structure of the schemas. There are no typedefs or groupings because they are just instantiated where they are referenced. Type chain is compiled into a single structure at the leaf or leaf-list where it is used with all the restrictions applicable to the data. So if you are using a schema tree structure somehow, the API will change. On the other hand, all the relevant information should be newly available much more easily.

libyang already provides API for external plugins for data types and extensions. Both these APIs will change to allow the implementation of complex cases. The same API will be actually used even for internal implementation of YANG built-in types and selected YANG extensions (NACM or Metadata). In contrast to the current libyang, these internal plugins will be compiled as an integral part of the binary instead of standalone dynamic object files. We believe that it should simplify libyang deployment.

In the end, we would like to provide a detailed description of the changes and how to migrate from libyang 1.x to 2.x. Despite we will do our best, it can be hardly complete as we don't know all the use cases you are aware of. Therefore, we will appreciate all your feedback.

[rkrejci]

Hi all,
I have just migrated the previous post from our issue tracker because I believe that the newly available GitHub Discussions is a much better place for this kind of information.

The original post is from September 2019, so it's time for an update, I guess.

We did a lot of work on libyang 2.0 and I believe that there will be an official release in January 2021. You can already check the documentation including the transition manual. You may notice that there are still some TODO/TBD in code as well as in the docs. The truth is that the initial release will miss some of the promised and still planned features. We will add them incrementally during 2021. In this case, I'm talking mainly about libyang plugins. Don't worry, YANG base data types or Metadata and NACM are already implemented. And as promised, they use, currently internal, API which is intended to become public. But before doing so, this API still needs some work and cleanup. We just didn't want to postpone the initial release just because of this.

What about the libyang 1.x? You might notice that, for already some time, we refuse various asks for new features or bigger changes in the current master/devel branches. But we still fix issues and help using libyang for various use cases. And we plan to continue with this even in 2021. We don't want to kick out the 1.x and forget. Both versions will be here with us for some time simultaneously. But please expect that answer for some of your issues with libyang 1.x will be just "Please move to libyang 2.x."

So, I'm looking forward to the libyang 2.0 release as well as to your feedback.

[choppsv1]

Is there any update on the release date. :)

[michalvasko]

The update is that it is still going to take a while :) It should be ready and fully working so you are free to use it, but there is still plugin support (both user types and extensions) missing. That should be implemented in the upcoming weeks and once it is done, there should be a release. But since there is no pressure on this, we are focusing on other projects.

[choppsv1]

There is a discussion in FRR (contrary to earlier decisions :) to move to libyang2 in the next major release, so we are seeking some clarity on API freezes and releases. It sounds like the API for finished features are frozen. Could that be made official in some way?

[michalvasko]

That is definitely good news but I do not think we can help you right now. API is really not frozen yet because what we still need to do is migrate all our other projects to use it (mainly libnetconf2 and netopeer2). There most likely be some changes needed for that. So, that migration is actually pressing and must be done well before the end of Q1. So the release will almost certainly happen right after that.

[rkrejci]

The plan is to avoid any changes of the finished features (the only in-progress features are YANG extensions and data types plugins support), but we still feel it is good to don't have it frozen and be able to make changes that make sense. You saw it when you came with the feedback to schema/data tree structures.

[iGovindY]

Hi @rkrejci , @michalvasko

do we have a documented procedure for git repo upgrade from 1.x to 2.x ?

Thanks
Govind.

[michalvasko]

Since simple git pull is not working, you need to clone the repository again, should not be a problem.

[iGovindY]

same for other repo? sysrepo, libnetconf2, netopeer2 also

git pull won't work? git clone needed?

we are seeing many conflicts during git pull even though we have not changed anything in those files.

[michalvasko]

Hm, I may have created some conflicts when merging even those other repositories, sorry about that. So yes, just clone it anew, that can never do any harm.

[iGovindY]

Actually we are maintaining all 4 repo as git submodules of our main repo.
so git clone might create issues for us.

Also, we have added some logging changes to all 4 repos, which we need to port, if we use clone

[michalvasko]

Then you are on your own just make sure you really have the latest correct code so that there are no false issues. Again, sorry, we do not plan any similar major release anytime soon, this was an exception.