IETF directories not updating

[samans]

https://github.com/YangModels/yang/tree/main/standard/ietf
The RFC directory has not been updated since 23 November 2023
121ac33 is the last commit.

The internet-draft extracted yang modules directory https://github.com/YangModels/yang/tree/main/experimental/ietf-extracted-YANG-modules hasn't been upaded since 20 December 2023 ead6cbf is the last commit.

The modules are available at the https://www.ietf.org/ietf-ftp/yang/, just not in git.

Can someone please look at this and advise?

[xorrkaz]

Yes, this has been broken since the move to the new AWS instance. I'll admit I do not know why as yet. Modules were being checked against that git repo, it's just for some reason the push isn't happening correctly. I should have some time at the IETF 120 meeting to take a deeper look at this. I might have to break out a big hammer to get it working again.

[rwestphal]

The modules are available at the https://www.ietf.org/ietf-ftp/yang/, just not in git.

Indeed, but apparently not all modules.

draft-ietf-bier-bier-yang-09 was released a few days ago, but I can't find ietf-bier.yang in https://www.ietf.org/ietf-ftp/yang/draftmod/

I realize this is a separate problem though.

[samans]

Thanks for all the efforts to get the I-D yang models automatically extracted and put in experimental/ietf-extracted-YANG-modules. However, there are still (at least one) that is missing. https://datatracker.ietf.org/doc/draft-ietf-netmod-rfc6991-bis/ was updated and contains two yang modules. But only one of them was extracted and added to experimental/ietf-extracted-YANG-modules. This one was added.. [email protected], the [email protected] was not extracted and added to the repository.

[xorrkaz]

The problem is this line in the ietf-inet-types module:

+ '\[[a-zA-Z0-9!"#$%&'+"'"+'()*+,./:;<=>?@\^_{|}~-]+])';`

That single " quote character there is causing an imbalance in xym's processing of this module, and thus it is not being extracted. If you adding a second " to the character class, it would extract, but then you have something weird in there.

@einarnn Since you maintain xym, I wonder if you could review this approach to solving the problem:

xym.py line 539:

-                if line.count("\"") % 2 == 0:
+                if line.count("\"") % 2 == 0 or re.search(r"'[^']+\"", line):

Essentially, ignore a double-quote if itself is quoted by a single quote on the same line. This feels naive, but does work in this case. The problem with this approach being that it only needs one double-quote to be single-quoted. Another could be outside the quote block, and this would pass.

[xorrkaz]

@einarnn Thoughts on the above? Do you want me to open an issue with xym?

[einarnn]

Joe, thanks for raising this again. I actually tried to find the referred content that causes the ***@***.*** to evaluate the suggested fix, but I couldn’t find it where the issue thread suggested. Can you provide a link to the problem text that I can review in conjunction with suggested fix?

…

On Sep 14, 2024 at 4:04 PM +0100, Joe Clarke ***@***.***>, wrote: @einarnn Thoughts on the above? Do you want me to open an issue with xym? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[xorrkaz]

Sure. The most unambiguous place to find this is by opening the XML and looking at line 1622. That is the offending line preventing the module [email protected] from being extracted.

[einarnn]

@xorrkaz, I agree that this is not seen to fail. But it leads me to wonder if we can just stop trying to do it this way. If we are extracting from the XML variant of the drafts, can't we depend more explicitly on extracting the `sourcecode element

<sourcecode><![CDATA[
<CODE BEGINS> file "[email protected]"
module ietf-inet-types {

...rest of the yang model text...

<CODE ENDS>
]]></sourcecode>

The current iteration of the tool was implemented to extract text from a text version of the draft, not the XML version. If we instead depend on matching this regex to the contents of sourcecode elements:

r"""<CODE BEGINS> file "(.+)"\n(.+)\n<CODE ENDS>"""

I may not have that completely right, but the intent is that group 1 will contain YANG file name and group 2 will contain the YANG module.

This style of parsing could be triggered with a new option, maybe --xml | -x, leaving the current text file targeted approach as it is.

Thoughts?

[xorrkaz]

I think that's a great idea given I believe XML is now mandatory. And if you throw lxml or maybe xmltodict in here, you could just literally grab the element.

[xorrkaz]

Heh. I was going to implement this, and it turns out xym already has a --rfcxml option to do this. yc.o doesn't use it, though. It only extracts from .txt files. I'm going to look into changing that.

[xorrkaz]

Okay. So I found xym to be a bit broken. I added the pure regex+XML approach and did a PR.

[einarnn]

@xorrkaz, I've reviewed and accepted the PR and released a new version of xym, 0.8.0.

[xorrkaz]

I have implemented this XML support in the yc.o extraction code. I'll be rolling it out shortly, so I'm hoping we see this extraction problem go away.

[xorrkaz]

@samans With the new XML parsing code in place, I see [email protected] properly extracted and in my local repo. I'm going to kick off a PR for @einarnn to merge, and this thing should then be resolved.