Streamer/Tape support (focus on tar-compatibility)

[bog-dan-ro]

Same request as PiSCSI/piscsi#480

I already have a non working WIP for piscsi (that was before I found scsi2pi).
I'll be happy to port it to scsi2pi. But I'll someones help, who knows the internals of this project, to fix the remaining stuff.

[uweseimet]

@bog-dan-ro Thank you for creating this ticket. Currently there are no plans to add streamer support to SCSI2Pi. In the past tapes were used for backups, because there was no better/cheaper solution. Nowadays, especially with tools like SCSI2Pi, you simply use a drive image for a backup. This is faster and more flexible than a streamer emulation. Using a streamer emulation instead would be rather inconvenient.

However, as soon as you have added fully functional tape support to PiSCSI please get back to me. I wil then run tests with my Atari (I have streamer software for it and also an HP streamer), and I will then re-evaluate this ticket based on the results of my tests.

[uweseimet]

@bog-dan-ro By the way, I would be interested to learn which methods from the Disk class you think should be moved into the StorageDevice class.

[bog-dan-ro]

Actually they are not too many, check my last commit from https://github.com/bog-dan-ro/piscsi

[bog-dan-ro]

See bog-dan-ro/piscsi@52a50c9#diff-ba201351e19eac2f5e902b9308a59f1e3bdc19ed324119dab011d8336c345410
and bog-dan-ro/piscsi@52a50c9#diff-76d682148e5aa7e6ce46bd9e40aae7de042a2a4e24242baf6f95b93612e6a16b

I didn't moved the caching as streamers do not need it as they are sequential so, once you read/write a block it will not seek back to read it again.

[uweseimet]

I will have a look at the code you moved and will get back to you, but this may take some time.

Is the streamer code in your repository already tested and working?

[bog-dan-ro]

@bog-dan-ro Thank you for creating this ticket. Currently there are no plans to add streamer support to SCSI2Pi. In the past tapes were used for backups, because there was no better/cheaper solution. Nowadays, especially with tools like SCSI2Pi, you simply use a drive image for a backup. This is faster and more flexible than a streamer emulation. Using a streamer emulation instead would be rather inconvenient.

There are other use-casese for than tape than backup. For instance, Altos 386 Series 1000 needs it to install the the system utilities.
I also want to use it to transfer large software on that machine as splitting big tar files and transferring them from floppy is not fun :).

I will have a look at the code you moved and will get back to you, but this may take some time.

No rush, I'll be in vacation for a couple of weeks.

Is the streamer code in your repository already tested and working?

I tested it a bit, but sadly is not working ... The mandatory implementation is there, but I don't know how to properly debug it. Most probably I'll have to add a modern build system (cmake?) so my IDE to know how to cross compile, deploy & debug it properly.
Also, I didn't have too much time to properly understand the API nor the workflow :). I'll have to cry for help on PiSCSI maybe someone who knows the code well, will have time to help me fix it.

I chose tandber as the reference device for two reasons:

1. Is one of the most used tape drives
2. it has proper scsi documentation ;-)

[uweseimet]

Indeed, I missed the use case of an installation. Anyway, I will get back to you (presumably in a couple of days) regarding moving code from Disk to StorageDevice.

There is no direct relationship between cmake and cross-compling, by the way. The nightly SCSI2Pi builds and the release builds are cross-compiled, but SCSI2Pi does not use cmake. I thought about using it, but it would not be worth the effort, considering that the Makefile is portable and straightforward.

[bog-dan-ro]

There is no direct relationship between cmake and cross-compling, by the way.

Nope is not, at least not direct. Current makefile works great to cross-compile it from command line. I created a docker file to help me cross compile it, and I have no issues to cross-compile it.
But ... Some IDEs (qtcreator) need to know what files are built, where they are on the host computer and where they should be deployed on the target. Using CMake, gives all that info to IDE.

Indeed, I missed the use case of an installation. Anyway, I will get back to you (presumably in a couple of days) regarding moving code from Disk to StorageDevice.

No hurry at all, starting next Friday I'll be in vacation for two weeks.

[uweseimet]

@bog-dan-ro In the develop branch I moved the code related to sector/block sizes from the Disk class to the StorageDevice class.
Changes related to shift counts (and others) are not required, because this code does not needed anymore in SCSI2Pi.

[bog-dan-ro]

@uweseimet I'll check it once I'm back.
BTW for piscsi I almost finished the testing. So far it works great. After I'll clean a bit the code I'll create a PR.

[bog-dan-ro]

@uweseimet Here PiSCSI/piscsi#1477 is the PR

[uweseimet]

@bog-dan-ro Can you please remove my name in the copyright notice of the new streamer class code (.cpp and .h)?

[bog-dan-ro]

@bog-dan-ro Can you please remove my name in the copyright notice of the new streamer class code (.cpp and .h)?

Done

[uweseimet]

@bog-dan-ro In the meantime I have added a branch issue_93 with initial tape drive support to SCSI2Pi. This implementation is completely new, i.e. it is not based on any existing code.

I have already successfully tested this branch with tar, both in the tar-file compatible mode (file extension .tar), and in the full-featured mode with filemarks support. When writing files to a real device instead of the filesystem tar actually makes use of filemarks.

[uweseimet]

@bog-dan-ro There has not been any recent feedback, neither in this ticket nor in PiSCSI/piscsi#480. As far as the SCSI2Pi ticket is concerned is is essentially done and tested. What about your PiSCSI ticket/PR?
I would like to ensure that both implementations are compatible from the user's perspective. For SCSI2Pi there are binaries of the development branch with all changes included, see https://www.scsi2pi.net/packages/develop/.
See the comments in the header of tape.cpp for details on the SCSI2Pi implementation.

@Pacjunk @sidick @ppuskari Quite long ago you were interested in tape support for the PiSCSI board. In case you are still interested, I would appreciate help with testing.

[ppuskari]

This is wonderful news. I am about ready to test this after building. I have been in the middle of a major job change so have been out of pocket a bit. Looking forward to seeing if the IIgs software can properly work with the new code and recognize it as the 3m drive and 40meg block formatted and sellable capacity. If this works I am going to learn a lot from your code and I am very appreciative! Petar Puskarich ***@***.***

…

Sent from my iPhone On Oct 5, 2024, at 5:01 AM, Uwe Seimet ***@***.***> wrote:  @bog-dan-ro<https://github.com/bog-dan-ro> There has not been any recent feedback, neither in this ticket nor in PiSCSI/piscsi#480<PiSCSI/piscsi#480>. As far as the SCSI2Pi ticket is concerned is is essentially done and tested. What about your PiSCSI ticket/PR? I would like to ensure that both implementations are compatible from the user's perspective. For SCSI2Pi there are binaries of the development branch with all changes included, see https://www.scsi2pi.net/packages/develop/. See the comments in the header of tape.cpp for details on the SCSI2Pi implementation. @Pacjunk<https://github.com/Pacjunk> @sidick<https://github.com/sidick> @ppuskari<https://github.com/ppuskari> Quite long ago you were interested in tape support for the PiSCSI board. In case you are still interested, I would appreciate help with testing. — Reply to this email directly, view it on GitHub<#93 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAIMVUB5TTRBYJCTSTB5FILZZ62GTAVCNFSM6AAAAABMWBHWR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJVGAYDGMZXHA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[bog-dan-ro]

Hi @uweseimet,

Sorry for slow reply, I was quite busy at work.
I'll check your implementation next weekend.
If is better than mine I'll probably back port it to piscsi if you're okay with it.

BogDan.

[uweseimet]

@bog-dan-ro Backporting is fine, but what about the existing PiSCSI PR? As far as I can tell the PiSCSI team has been waiting for your feedback for weeks.

[bog-dan-ro]

@uweseimet my patch works for me and from my POV it's done. There are a few small issues to fix on the UI side which I could not fix them myself.

[uweseimet]

When will these issues be fixed? What about the device type name? Please see my last comment in the PiSCSI ticket.

[uweseimet]

@Pacjunk You mentioned issues with the PiSCSI web UI and SCSI2Pi. While waiting for your feedback I verified whether the web UI is working with SCSI2Pi and did not notice any problems. I installed
2024-04-30-PiSCSI-v24.04.01-arm64-lite.zip, and then I installed SCSI2Pi 3.4.3 with apt. After starting the s2p system service (see Quickstart on https://www.scsi2pi.net/en/installation.html) I could properly access the UI, see the attached screenshot.
Note that the web UI does not work anymore after updating the Pi to the latest (bullseye) packages before (and maybe also after) launching the web UI for the first time. Looks as if the up to date bullseye setup of pip-managed python packages (at least pybabel) is not compatible with what the web UI requires, or at least not with the PiSCSI arm64-lite image. But this is a PiSCSI web UI issue and not caused by SCSI2Pi. I created PiSCSI/piscsi#1481 for that.

[Pacjunk]

Hi, got a couple of hours today to do some testing using 13d6667 (although same results with 5cb8b4f). Using a blank tap file I managed to initialise (ie write a label), then mount the tape with that label successfully. Trying to copy files to the image resulted in errors. I notice that every record written to the .tap file has a "SCTP" prefix. This is most likely the reason that I cannot mount .tap files produced by others from places such as archive.org. Of course they don't have records prefixed with SCTP!

I have attached two files. The .tap file (after it has been initialised) and the log file. The steps were:
1 - Initialise the tape with a label (TEST77)
2 - Mount the tape
3 - Try to copy a file to the tape (failed)
4 - Dismount the tape.

I probably won't get any time with this until next week as I will be away for a few days. Cheers,

2.log
2.tap.txt

[uweseimet]

@Pacjunk Thank you for the log. Yes, the non-tar format is proprietary because it also contains meta data (see the comment at the beginning of tape.cpp) that represent filemarks and end-of-data markers. Without these meta data spacing in order to search for a filemark, for example, would not be possible. This is why spacing cannot be supported with tar files: The required markers are missing in these files.

If I understand your correctly the log does not contain the SCSI operations executed when labeling a tape?

Edit: But the logs now reveal more information. Looks as if the labeling step actually IS included, but please confirm. I see that your workstation/driver uses reverse spacing:

[2024-11-06 18:30:10.820] [trace] (ID 3) - COMMAND phase
[2024-11-06 18:30:10.820] [debug] (ID 3) - Controller is executing SPACE(6), CDB $11:00:ff:ff:ff:00
[2024-11-06 18:30:10.820] [trace] (ID 3) - Going to transfer 0 byte(s) in chunks of 0 byte(s)
[2024-11-06 18:30:10.820] [debug] (ID:LUN 3:0) - Device is executing SPACE(6) ($11)
[2024-11-06 18:30:10.820] [error] (ID:LUN 3:0) - Reverse spacing is not supported

This is not supported by SCSI2Pi (and also not a mandatory SCSI requirement) and I don't know if I will support it. See the initial comment of tape.cpp for details.
What I would like to check is whether your physical drive really supports reverse spacing. This can be checked by running s2pexec against it. I will provide the required s2pexec commands later. They are essentially the same commands you can see in the logfile, e.g. line 3200.

[Pacjunk]

Yes the log includes labelling the tape.

[uweseimet]

This is most likely the reason that I cannot mount .tap files produced by others from places such as archive.org.

Is there any information on the format of these files? Assuming that these are not just tar files, they must contain encoded meta data that represent filemarks etc., and this encoding must be document somewhere in order to be usable. From your latest log it is obvious that filemarks are used to mark labels.
Any kind of encoding I can think of would require rather ugly code to read/write it, and/or would increase the file size quite a lot. Especially when you want to space backwards, you need to somehow find where the previous tape object (block, filemark etc.) starts. This information cannot be encoded in the actual block data, but must live in a separate world. I guess that with a real streamer there is an interrupt when such kind of data are encountered, and the streamer's firmware can then work with these data, without confusing them with an actual block payload. IMO the description of the SPACE(6) command and what happens when a block/filemark/setmark/end-of-data marker is encountered unexpectedly implies this.

At least it is clear now that labeling is definitely required before you can make use of a tape image file, but the end of your logfile looks like your workstation is searching something on the "tape" after labeling, which it cannot find. As part of this search operation it spaces backwards, which is probably some kind of retry mechanism. At least this is how I read the end of the logfile.

[Pacjunk]

Is there any information on the format of these files? Assuming that these are not just tar files, they must contain encoded meta data that represent filemarks etc., and this encoding must be document somewhere in order to be usable. From your latest log it is obvious that filemarks are used to mark labels. Any kind of encoding I can think of would require rather ugly code to read/write it, and/or would increase the file size quite a lot. Especially when you want to space backwards, you need to somehow find where the previous tape object (block, filemark etc.) starts. This information cannot be encoded in the actual block data, but must live in a separate world. I guess that with a real streamer there is an interrupt when such kind of data are encountered, and the streamer's firmware can then work with these data, without confusing them with an actual block payload. IMO the description of the SPACE(6) command and what happens when a block/filemark/setmark/end-of-data marker is encountered unexpectedly implies this.

I don't know how they are structured. I do know that simh reads them correctly. Whether the .tap file is specially formatted for simh, or simh follows some standard, I don't know.

[uweseimet]

@Pacjunk Do you have a download link for such a file?

[uweseimet]

@Pacjunk Regarding reverse spacing I would like to verify whether your drive actually supports it. Can you please connect your drive to the PiSCSI board and then launch s2pexec and send the following SCSI commands to your drive:

s2pexec>-i SCSI_ID_OF_YOUR_DRIVE -L debug -c 12:00:00:00:ff:00
s2pexec>-c 11:00:ff:ff:ff:00

If for any command "Error: UNIT ATTENTION" is reported (probably for the second command), please send the same command again.
After each command there will be some console output, which I am interested in. The commands do not modify any data.

[Pacjunk]

Here are some files... http://www.bitsavers.org/bits/DEC/vax/vms/TK50/

I was doing some reading, and the labelling conforms to the ANSI tape label format.

Wouldn't it be up to the OS (or driver) to look after tape marks etc, rather than the tape drive inserting them?

I will have to hunt around for appropriate cables to connect the tape drive and piscsi board up simultaneously.

[uweseimet]

@Pacjunk The drive is responsible for creating markers like filemarks or setmarks (support for the latter is optional). This is what the SCSI command SET FILEMARKS is for.. The drive is also responsible for finding these markers with the SCSI command SPACE. (See the SCSI specificiation for details on SET FILEMARKS and SPACE.)
The OS/driver controls when to set/evaluate these markers and to give these markers a meaning. A marker might be a tape label or something else, depending on the OS and/or the driver, or on the location of the marker on the tape.
tar, for instance, uses filemarks differently (it sets a single filemark at the end of an archive) and does not require a tape to be labelled. The Gemar streamer software for the Atari works similar, also without a requirement for labels.

The files you mentioned are in simh format, which appears to be related to an emulation. Are you sure that a driver for real tape drives can make use of this kind of file? Or are you using a special kind of driver, which can deal with simh files? I doubt that the data used by different kinds of tapes drives and operating systems all have the same physical format, even when the data are not compressed. How would a driver for a real tape drive know simh?
@bog-dan-ro I have not been aware of the .tap extension already being used by existing software. This means that using .tap for PiSCSI and SCSI2Pi as discussed in PiSCSI/piscsi#480 is not such a good idea, at least not for the non-tar files supported by SCSI2Pi and (maybe) by PiSCSI. .tap is already reserved for something different.

[Pacjunk]

The real hardware talks to the real drives. The .tap files I believe are either in simh or "DECUS" format. They are not usable on the real machine. This is what emulators are for! As these .tap files are part of preservation efforts, I'm not sure how the community dumps and restores the tapes with the images. Maybe they cluster a real and a virtual machine, then copy "tape to tape", or maybe they just copy the files from the tape to disk, transfer the disk files, then create a new virtual tape. There are probably several ways.

[uweseimet]

@Pacjunk I see. The goal of SCSI2Pi is to support regular tar files and a proprietary image file format that reflects most of the objects that SCSI commands can write to a tape, using the native tape driver of a real machine. This is a different approach than supporting simh files.
Looks as if there are these 3 kinds of relevant formats, all with their special use case:

1. tar archive files. They do not contain markers like filemarks, just the plain data created by tar and compatible tools. On Linux "tar cf /dev/st0 FOLDER_TO_ARCHIVE" creates them, "tar xf /dev/st0" extracts them.
2. simh files, which are platform independent but require special drivers for reading/writing them.
3. raw image files, which work with the native drivers for a platform. They are platform-specific because the respective OS/driver determines the format and decides on the use of filemarks etc.

Do you agree with this list? SCSI2Pi aims at supporting items 1. and 3. Items 1. and 3. have already successfully been tested with tar on Linux and Gemar on the Atari, but tar and Gemar do not make use of tape drive features like spacing, in particular reverse spacing, which you need for your platform.
Supporting item 2. is probably interesting, but it requires a documentation on the simh format at least.

[Pacjunk]

Yep, fine. I didn't expect SCSI2Pi to support simh files. I just used them as a ready-made input file for testing. Looking at the spec it seems that the record length is specified at the beginning and end of the data block to enable reversing. Looks like this sort of behavior will be too complicated for this project (although I assume real physical drives must do it - need to do those tests!)

Spec is here: http://simh.trailing-edge.com/docs/simh_magtape.pdf

[uweseimet]

@Pacjunk Yes, adding the length or previous position also at the end (currently SCSI2Pi only adds it to the beginning) of an object (I use the term object because the specification also uses it, it covers not just data blocks but also filemarks etc.) is something I have already thought about. But I have not made up my mind yet. As far as I can tell, when writing gaps (currently not supported by SCSI2Pi) you can end up at more or less any position. And if you are somewhere in the middle of the "tape", neither and the beginning nor end of an object, any information tied to an object does not help.

I am also not 100% sure whether your driver in normal operation really needs reverse spacing support. It might be that the driver just tries reverse spacing as a last resort because it did not find what it was looking for on the tape. This is why I asked for checking whether your drive really supports reverse spacing. My drive appears to support it, but as already mentioned it is broken and cannot write data anymore. But it does not reject the SPACE command for reverse spacing, i.e. it appears to know what reverse spacing is.

[Pacjunk]

The document at one point talks about physical drives reversing in order to re-read bad blocks. It specifically mentions TK50 which was a common software distribution media before CD-ROMs came along.

I will try to test one (maybe two) of my tape drives in the next couple of days.

[uweseimet]

@Pacjunk A bad block is indeed something I was thinking of as a reason for reverse spacing. This is what I meant with "in normal operation": There cannot be bad blocks in our case, and if the driver thinks that there is a bad block, something must have gone wrong before. But finding out what might have gone wrong is hard. When analyzing the logfile I did not find anything that was obviously wrong or looked like something the driver could regard as a bad block. My guess is that there is still something wrong with the filemarks handling. The rules of how spacing works, how objects are skipped during spacing, and how spacing results are returned in the sense data are quite complicated.
The latest issue_93_3 branch contains some improvements regarding the sense data returned when spacing. Can you please run a new test? Please try to limit the amount of operations to a minimum, e.g. when labelling do it exactly once. Note that the raw file format has changed. Do not reuse any old .tap files but please use new, empty image files. Also note that the default extension has changed to .st. If you want to also use .tap as an extension that is automatically recognized add this line to /etc/s2p.conf:

extensions.sctp=tap

[uweseimet]

After having a superficial look at the simh format, this format probably provides everything I need for item 2 on my list, including gap support. This would be nice, because I prefer existing formats over proprietary ones.

I will create a separate ticket for raw tape files support (be it in simh format or something proprietary). The initial focus of tape support in SCSI2Pi 4.0, which is also mentioned on the SCSI2Pi website, is support for image files that can be used with tar (and as it turned out also with Gemar). This feature has already been successfully tested, and therefore it makes sense to have a release 4.0 that comprises this feature. simh (or similar) support will then become part of release 4.1 or so.

A new ticket will also make it easier to follow any progress, without having to read all comments in the current ticket first.

@Pacjunk As far as testing is concerned, nothing substantial is going to change, because anything that might be wrong with the spacing implementation will also be wrong with a different file format.
This means that it would still be useful if you could run a test (trace level) with the current issue_95_3 branch, and to connect a real tape drive to the board in order to verify whether the drive supports reverse spacing.

[uweseimet]

#100 is the follow-up ticket to #93. Anything not dealing with tar compatibility but with simh files should be added to the new ticket.

@Pacjunk Can you please provide feedback on the issues you had with the PiSCSI web UI? As mentioned in a previous comment I could not find any problem with using the web UI together with SCSI2Pi, except for the python package issues when running the UI with an up to date bullseye. I have already created PiSCSI/piscsi#1481 for this, because this is a problem not caused by SCSI2Pi but also present with PiSCSI.
In case your web UI problem is a different one, which only affects SCSI2Pi but not PiSCSI, please create a separate SCSI2Pi ticket with information on how to reproduce this problem with SCSI2Pi. Thank you.

Just a general technical note on using the web UI with SCSI2Pi: All (protobuf and command line) commands supported by SCSI2Pi are intended to be backwards compatible, as far as PiSCSI supports the respective functionality. Furthermore, SCSI2Pi and PiSCSI use the same port (6868) for communicating with clients. As long as this is the case there is no technical reason why the behavior of the web UI should differ between PISCSI and SCSI2Pi. The UI may not properly handle new devices like SCTP, though. But due to the nature of the protobuf interface, which provides clients with meta data on each device type, even a client that does not know a particular device type may be able to support it to a certain degree. A client knows the device type (SCTP), and it knows whether a device supports image files or which parameters an unknown device type supports etc. This means that a client that makes good use of protobuf can offer essentially everything except a human-readable name for the device type or the names of device-specific statistics. The technical names, which are part of the protobuf interface, can be used as defaults in this case.

[uweseimet]

SCSI2Pi 4.0 release candidate binaries with streamer support for tar files only are now available. There is already an issue_100 branch for #100. I will need some time to change the existing code to read/write the simh format. In issue_100 the .tap extension is supported again by default, this time with the goal to be simh-compatible, which makes this extension the correct choice.

@Pacjunk Do you think I would profit from installing simh on my Linux machine? I assume that I would not be able to use SCSI2Pi in a simh emulation context, even with a Linux-PC with SCSI card. But maybe I am missing something and there is a setup that would help with testing.

[ppuskari]

I am finally able to get some time on this project. I hat I am fallen behind on is how to setup for testing. I have Piscsi device and Linux and compilers etc per not mail to download Piscsi sources and compile and to compile my custom netatalk and Ghostscript ImageWriter II color support etc. What links do I need to also grab or what fork to get the latest tape pieces as well as the test apps we are using in this thread. I can get setup and test that Apple 40meg tape support 3m block addressable tape after a full tape format. Things look promising. And many thanks guys for working on this! Petar Puskarich ***@***.***

…

Sent from my iPhone On Nov 11, 2024, at 11:01 AM, Uwe Seimet ***@***.***> wrote:  Closed #93<#93> as completed. — Reply to this email directly, view it on GitHub<#93 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAIMVUFRSTGG43APNHAD5VT2ADPGVAVCNFSM6AAAAABMWBHWR6VHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJVGI2TGMJSHE2DINI>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[uweseimet]

@ppuskari Sounds good! Note that I have closed this ticket because it deals with tar-compatible tape image files only, and this kind of image files has already been tested and will be supported in SCSI2Pi 4.0, for which I created the first RC today. If you want to you can, of course, test the SCSI2Pi 4.0 RC with tar. For the RC a binary package is available on https://www.scsi2pi.net/packages/develop/.

#100 is the follow-up ticket for this ticket, dealing with supporting simh tape images, which are not tar compatible. Please read the latest comments in this ticket in order to get up to date wrt the tests that need to be done.

There is already an issue_100 branch for #100. This branch is the one you should use for compiling SCSI2Pi and testing anything that is not related to tar compatible image files. There are no special test applications, i.e. testing needs to be done with the native tape drivers for the OS you are using. Testing essentially consists of backup up data to the emulated tape device and restoring these data. Please read the comments in this ticket to learn more.