Hierarchical Schematics and Design Libraries

[curtisma]

Thanks for the great work!

How do I include schematic and symbol libraries?
Could the web viewer understand the directory hierarchy when providing a schematic from GitHub using the ?file selection in the URL? i.e. Add directories above the schematic to XSCHEM_LIBRARY_PATH for it.

Example: https://xschem-viewer.com/?file=https://github.com/cascode-labs/bandgaps/blob/main/bandgaps/bandgap_sky130_v1/bandgap_1v_v01/bandgap_1v_v01.sch

Problem

You can see in the screenshot below from the web viewer that I am missing the amplifier symbol that is shown in the second screenshot using the desktop executable version of xschem.

Desktop Implementation

In the desktop version I just have to make sure the XSCHEM_LIBRARY_PATH is set in the xschemrc according to how I organize my design library files as explained in the xschem docs.

In order to define this more portably across projects without having seperate xschemrc file for each project, I add the following Tcl code to the end of my xschemrc. This then allows me to define an environment variable using one of the file structures in the xschem docs. In my case option 3 makes the most sense.

# allow a user-specific path add-on (https://github.com/iic-jku/iic-osic-tools/issues/7)
if { [info exists ::env(XSCHEM_USER_LIBRARY_PATH) ] } {
    append XSCHEM_LIBRARY_PATH :$env(XSCHEM_USER_LIBRARY_PATH)
}

A Possible Solution

Could we have the web viewer understand the file structure from github so that it can capture the schematics and symbols up to 2 levels above the provided schematic. This would allow us to include the library from the same repo assuming you folllow a similar library structure.

Here's the library file structure in my example:

Issue in the web viewer

Here is the schematic from one of my github projects:
https://xschem-viewer.com/?file=https://github.com/cascode-labs/bandgaps/blob/main/bandgaps/bandgap_sky130_v1/bandgap_1v_v01/bandgap_1v_v01.sch

Expected Result

Here is the same schematic in the desktop app:

@urish, @StefanSchippers, @barakhoffer

[urish]

Thanks for the detailed report!

The current logic for looking up symbol files is defined here:

xschem-viewer/src/model/LibraryLoader.ts

Lines 43 to 80 in 57c026d

	private async fetchContent(path: string): Promise<Response> {
	if (path.toLowerCase().startsWith('https://')) {
	const url = githubURLToRaw(path);
	return await fetch(url);
	}
	for (const library of this.libraries) {
	if (path.startsWith(library.path)) {
	const url = library.url + path;
	return await fetch(url);
	}
	}
	if (this.baseURL != null) {
	const url = new URL(path, githubURLToRaw(this.baseURL));
	const result = await fetch(url);
	if (result.ok) {
	this.pathToUrl.set(path, url.toString());
	return result;
	}
	// Try to look under the root of the github repo
	const altPath = url.pathname.split('/').slice(0, 4).join('/') + '/' + path;
	if (url.hostname === 'raw.githubusercontent.com' && url.pathname !== altPath) {
	url.pathname = altPath;
	const result = await fetch(url);
	if (result.ok) {
	this.pathToUrl.set(path, url.toString());
	return result;
	}
	return await fetch(`https://raw.githubusercontent.com${url.pathname}`);
	}
	}
	if (!path.includes('/')) {
	// as a fallback, look under devices/
	return await this.fetchContent(`devices/${path}`);
	}
	throw new Error(`File not found: ${path}`);
	}
	}

It's pretty complex already - it first tries to look if the given path starts in one of the libraries listed here, then it loads the file from that library:
https://github.com/TinyTapeout/xschem-viewer/blob/57c026d2f91b143bdc3d37347a9d8ae39811c025/src/model/libraries.ts

Otherwise, it looks if the file exists at the same directory level as the current schematic pointed by ?file=. Finally, if it can't find it, and the URL looks like a github repo, it'll make one last attempt to fetch from the root of the GitHub repo.

Here are some things to take into account when finding a solution:

1. The viewer can't currently read the tree from the GitHub repo (and introducing this feature will most likely require signing in to GitHub to perform API queries). This only applies to the online version.
2. The viewer can make queries against specific files (if we can guess the path of the file, we can ask if it exists), but each query has a non trivial latency, so we'd like to reduce the number of queries (e.g. 2-5 queries per file are still acceptable, more than that can be an issue). Again, latency only applies to the online version.
3. One way to solve it would be to either add another query variable, or, alternatively, some JSON file to the root of the repo, that will tell the viewer which strategy to use for looking up files. The query variable is only relevant for the online version, and a JSON file could be used by either the online version of the VS Code extension, and perhaps in the future by xschem itself if that proves popular.