MHVTL treatment of filemarks

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

MHVTL treatment of filemarks

SteinnGaubert
Hi,

I do a lot of tape conversions and archiving and I've been investigating/playing with mhVTL as a serious alternative for a few weeks now.  I've had some weird problems and while I'm still tracking down the others I have a question on how mhVTL treats filemarks.

It seems that mhVTL treats filemarks as a physical sector instead of an "invisible barrier" (my words) between two physical sectors.  For example, on a physical drive (at least the ones I've tested), reading yields this:
        Physical:
          block 0: DATA
          block 1: DATA
          (read returns EOF)
          block 2: DATA
    but on mhVTL
          block 0: DATA
          block 1: DATA
          block 2: (read returns EOF and no data)
          block 3: DATA

Looking at the tape format code this is clearly a design decision and I'm wondering if I'm missing something as to why it's done this way, or if there's a way to disable this.  I've seen this cause strange symptoms with anything that uses absolute seeks.

Any ideas or thoughts?
Thanks,
Steinn
Reply | Threaded
Open this post in threaded view
|

Re: MHVTL treatment of filemarks

Mark Harvey
Administrator
Hello Steinn,

Many thanks for the detailed problem description..

The filemark info is not part of the actual data stream, but part of the metadata.

Your description of the behaviour of mhVTL is correct, with some minor details missing.

You will get this behaviour if you read the data using the same block size - however the block read following the file mark will return block 2 data, not block 3 as your description sort of indicates.

If you attempt to read each block with a larger size, the mhVTL behaviour will be the same of physical drive. e.g. if each tape block is 64k and you attempt to read specifying a 128k block.

It sounds like you have discovered a bug - i need to verify against a real drive, which won't be until next week at the earliest.

Will you be willing to test any potential patches ?

SteinnGaubert wrote
It seems that mhVTL treats filemarks as a physical sector instead of an "invisible barrier" (my words) between two physical sectors.  For example, on a physical drive (at least the ones I've tested), reading yields this:

        Physical:
          block 0: DATA
          block 1: DATA
          (read returns EOF)
          block 2: DATA
    but on mhVTL
          block 0: DATA
          block 1: DATA
          block 2: (read returns EOF and no data)
          block 3: DATA

Thanks,
Steinn
Regards from Australia
Mark Harvey
Reply | Threaded
Open this post in threaded view
|

Re: MHVTL treatment of filemarks

SteinnGaubert
Mark,

Thanks so much for the quick reply and diving into this!

I didn't mean to imply that mhVTL will return data from block 3 (from my example) - it does not.  

Attached below is quick test case to demonstrate.  I ran this on mhVTL and two physical drives.  I'm using fixed blocks to keep it simple.

Thanks again!  I'm of course happy to test out any patches (and to do so quickly).

mt -f /dev/nst0 rewind
dd if=/dev/zero of=/dev/nst0 bs=512 count=4
dd if=/dev/zero of=/dev/nst0 bs=512 count=4
mt -f /dev/nst0 rewind
dd if=/dev/nst0 of=/dev/null bs=512
mt -f /dev/nst0 tell
mt -f /dev/nst0 seek 4
dd if=/dev/nst0 of=/dev/null bs=512
mt -f /dev/nst0 tell
mt -f /dev/nst0 rewind
mt -f /dev/nst0 fsf
mt -f /dev/nst0 tell

On a physical drive the output is:
4+0 records in
4+0 records out
2048 bytes (2.0 kB) copied, 6.10852 s, 0.3 kB/s
4+0 records in
4+0 records out
2048 bytes (2.0 kB) copied, 1.9853 s, 1.0 kB/s
4+0 records in
4+0 records out
2048 bytes (2.0 kB) copied, 0.0437074 s, 46.9 kB/s
At block 4.
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.0184658 s, 0.0 kB/s
At block 4.
At block 4.


On mhVTL it's:
4+0 records in
4+0 records out
2048 bytes (2.0 kB) copied, 1.13958 s, 1.8 kB/s
4+0 records in
4+0 records out
2048 bytes (2.0 kB) copied, 0.0504578 s, 40.6 kB/s
4+0 records in
4+0 records out
2048 bytes (2.0 kB) copied, 0.0398508 s, 51.4 kB/s
At block 5.
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.0076498 s, 0.0 kB/s
At block 5.
At block 5.


Reply | Threaded
Open this post in threaded view
|

Re: MHVTL treatment of filemarks

SteinnGaubert
Please hold off of "fixing" this.   There's some additional complexity going on between physical and virtual so let me track it down first.

Thanks!
Reply | Threaded
Open this post in threaded view
|

Re: MHVTL treatment of filemarks

Mark Harvey
Administrator
Do you have any references to "REQUEST BLOCK ADDRESS" ?

SSC2 etc..

Sent from my iPhone

On 18 Mar 2014, at 8:45, "SteinnGaubert [via mhVTL - A Linux Virtual Tape Library]" <[hidden email]> wrote:

Just forget this entire "bug" report!

It turns out that Windows is rewriting the block numbers and hiding the fact that on physical tape the filemarks DO count as a block.  EXACTLY like mhVTL is doing today.  Once I bypassed Windows via SCSI Pass Through, the issue was obvious.  

That's why the linux script demonstrated the "issue" so easily.  

So sorry about the bother!

It turns out the real issue is that mhVTL doesn't support the old "REQUEST BLOCK ADDRESS" command which the old software needs.  That is something I might be able to hack in.



If you reply to this email, your message will be added to the discussion below:
http://mhvtl-a-linux-virtual-tape-library.966029.n3.nabble.com/MHVTL-treatment-of-filemarks-tp4025761p4025766.html
To start a new topic under mhVTL - A Linux Virtual Tape Library, email [hidden email]
To unsubscribe from mhVTL - A Linux Virtual Tape Library, click here.
NAML
Regards from Australia
Mark Harvey
Reply | Threaded
Open this post in threaded view
|

Re: MHVTL treatment of filemarks

SteinnGaubert
It may be overkill but the excellent DDS-4 Programmers manual details "request block address" and the "seek block" commands very well.  They're both very simple 6-byte CDB's.  The only weird thing is that block numbering starts from 1 instead of 0.  A lot of older software use them instead of the "newer" read position/locate commands.

Reply | Threaded
Open this post in threaded view
|

Re: MHVTL treatment of filemarks

Mark Harvey
Administrator
Many thanks for reference. I looked
thru my references from QIC/Exabyte/dlt7000 onwards and didn't find it. I don't (didn't) have DDS :)

Sent from my iPhone

On 19 Mar 2014, at 10:41, "SteinnGaubert [via mhVTL - A Linux Virtual Tape Library]" <[hidden email]> wrote:

It may be overkill but the excellent DDS-4 Programmers manual details "request block address" and the "seek block" commands very well.  They're both very simple 6-byte CDB's.  The only weird thing is that block numbering starts from 1 instead of 0.  A lot of older software use them instead of the "newer" read position/locate commands.




If you reply to this email, your message will be added to the discussion below:
http://mhvtl-a-linux-virtual-tape-library.966029.n3.nabble.com/MHVTL-treatment-of-filemarks-tp4025761p4025768.html
To start a new topic under mhVTL - A Linux Virtual Tape Library, email [hidden email]
To unsubscribe from mhVTL - A Linux Virtual Tape Library, click here.
NAML
Regards from Australia
Mark Harvey
Reply | Threaded
Open this post in threaded view
|

Re: MHVTL treatment of filemarks

SteinnGaubert
In reply to this post by SteinnGaubert
I've been meaning to follow-up on exactly what I figured out.  As I posted earlier, it had nothing to do with mhVTL.  

It turns out that both Windows and Linux sometimes rewrite block numbers and hide the fact that on physical tape the filemarks DO count as a block.  EXACTLY like mhVTL is doing today.  Once I bypassed Windows via SCSI Pass Through, the issue was obvious.  It took more work on linux to track it down because different tape drivers (ost.c vs st.c for example) treat filemarks differently.  It was different between physical and virtual because different drivers were being loaded.

So sorry about the bother!