Functionality to add location on disk of library contents

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Functionality to add location on disk of library contents

Galactor
I wonder if it is an easy change to build in the functionality to specify which filesystem supports the virtual tapes that might belong to a given library.  For example, this could be specified in device.conf as a parameter of the the library or within the library contents.

It would make it easier to differentiate between tapes that belonged to a given library I suppose.

You would have to modify mktape to be able to accept a parameter specifying where the tapes should be constructed.
Reply | Threaded
Open this post in threaded view
|

Re: Functionality to add location on disk of library contents

Mark Harvey
Administrator
This question comes up every so often.

It would be possible to do this. However, it's still a long way down that never ending 'to-do list'. And the number of requests for this feature is quite small to force it up the list.

You can currently 'work around' this by manually creating symlinks for each media to the file system in question and running 'mktape'. However, the /opt/mhvtl will still contain a list of all media.

Regards from Australia
Mark Harvey
Reply | Threaded
Open this post in threaded view
|

Re: Functionality to add location on disk of library contents

Galactor
Mark, many thanks for the response.  

The symlinks workaround was not beyond my imagination (indeed I had tried it) although it does not overcome the issue of not being able to discern between volumes and libraries.

One reason why it might be advantageous to create areas for certain libraries is that a file system can often be part of a replication directive.  Imagine that you have a mixture of systems, some requiring robust backup strategies whereupon some backup volumes are duplicated (near real-time) for critical systems but not for the less important ones.  As you will appreciate, replication is usually set up at filesystem or volume group level.  Having just one hard-coded volume makes that difficult with just a single installation of MHVTL - you can solve it, usually if the storage media (I am using Oracle Secure Backup) supports multiple storage nodes.  But that, of course, increases the management costs and you have to dedicate a storage node to the SLA-type.  Not ideal in terms of redundancy - you just want to be able to "write" and not care where.

Symlinks in this case would be risky as you could easily misplace a link to the wrong volume in the belief that you are replicating it whereas in fact you are not.

I have been using MHVTL with OSB for some time now and all the problems about overwriting tapes are resolved.  The product is absolutely brilliant - my congratulations to you.  Hearing my system chunter away as I see my archive logs being backed up and the messages file reporting "consecutive writes" is very satisfying.  I have given courses in Oracle RMAN with the MHVTL as backed-end MML (media management layer) to demonstrate the tape/disk capabilities of RMAN.  I do have a physical DAT72 drive but the performance is not great.

On my Oracle VM (Xen) system I can set up four drives of a LTO-5 STK library and backup four RMAN channels at peaks of 70MB/s - not too far off real systems.

One thing that doesn't work with OSB is volume duplication.  For some reason, OSB can't seem to set up the tapes/drives to do this.  Maybe we should take a look sometime.

Regards,
Graeme
Reply | Threaded
Open this post in threaded view
|

Re: Functionality to add location on disk of library contents

Mark Harvey
Administrator
Thanks for the explaination..

This is one  of (perhaps the first) good use-case for keeping the virtual media on seperate file system(s) per library.

The  most common supplied reason is throughput on the filesystem.

The obvious dis advantage would be 'moving media between libraries' but I supose that can be overcome via symlinks. (not that I'm aware of anybody moving media between libraries)

I'll have a look at it but can't promise anything..

Perhaps a /opt/mhvtl/<library_number>/<media>

So each library number could be a different mount point or link to another favourate file system of choice.

Cheers
Regards from Australia
Mark Harvey
Reply | Threaded
Open this post in threaded view
|

Re: Functionality to add location on disk of library contents

Galactor
Would it not be the case the differing filesystem blocksizes would offer performance benefits (and optimal space usage) given the differing tape blocking factors of certain devices?  Smaller blocksizes where many small files are being backed up?

This is possibly another reason for supporting differing filesystems - they can be formatted with differing block sizes.  I could imagine that my database backups could be written to a channel that uses larger blocksizes (a typical Oracle block size is 8K) and my archive logs to a channel that uses a smaller blocksize (archives are written in 512 byte blocks on UNIX systems).  OSB certainly has functionality to automatically define (using storage selectors) where the various backup types should be sent (full, incrementals, archivelogs and autobackups) so that the DBA doesn't have to manually configure this during backup.

Ever thought of writing to raw disk device in which case each tape would contain a block offset and length?
Reply | Threaded
Open this post in threaded view
|

Re: Functionality to add location on disk of library contents

Mark Harvey
Administrator
Would adding a new entry in device.conf under each library defination be suitable ?

I was thinking along the lines of
 Home directory: /opt/mhvtl/libA/
(I started off thinking the relative path, but I think absolute would be better.)

Then it will be up to each Tape device entries to read/remember the parent directory..

mktape will need to be passed the library id number.

If the entry is not specified, then the default of /opt/mhvtl would apply..

Thoughts / comments ?
Regards from Australia
Mark Harvey
nia
Reply | Threaded
Open this post in threaded view
|

Re: Functionality to add location on disk of library contents

nia
Administrator
Mark Harvey wrote
Would adding a new entry in device.conf under each library defination be suitable ?

I was thinking along the lines of
 Home directory: /opt/mhvtl/libA/
(I started off thinking the relative path, but I think absolute would be better.)

Then it will be up to each Tape device entries to read/remember the parent directory..

mktape will need to be passed the library id number.

If the entry is not specified, then the default of /opt/mhvtl would apply..

Thoughts / comments ?
This sounds good with me , although I would not mind also to make this as non configurable item where each library will go in a separate folder by default under /opt/mhvtl/$libraryId.

Just my two cents.
-nia
Reply | Threaded
Open this post in threaded view
|

Re: Functionality to add location on disk of library contents

Galactor
In reply to this post by Mark Harvey
I don't think there is much difference in the relative/absolute path specification.  I suppose that an absolute specification leaves the system administrator to make a choice.  You could mix the functionality:

Home directory: libA

or

Home directory:

would be created under /opt/mhvtl/libA or /opt/mhvtl respectively.

Home directory: /libA

would be created under the root directory as /libA.

Not specifying "Home directory:"  would default to /opt/mhvtl as currently stands.

I think that device.conf is the best place to put the directive and indeed as part of the library subset.

Regards,
Graeme
Reply | Threaded
Open this post in threaded view
|

Re: Functionality to add location on disk of library contents

Mark Harvey
Administrator
A 'proof of concept' patch.

Anybody willing to test this ?
You will need to manually move the media into the correct location.

This patch will :
 Honour the "Home directory: /some/where/interesting" in device.conf
Or if not defined, default to /opt/mhvtl/<library index>/

e.g.
Library: 10 CHANNEL: 00 TARGET: 00 LUN: 00
 Vendor identification: QUANTUM
 Product identification: Scalar i6000
 Unit serial number: XYZZY_A
 NAA: 10:22:33:44:ab:00:00:00
 Home directory: /opt/mhvtl/fred

If at mount time, the media can't be located in the "Home directory" location, it will fall back to /opt/mhvtl/

Still to do:
- Update mktape to take a library index number (so mktape will create in the correct location)
- make_vtl_media script updated
- Update man pages
- test (basic load/unload tests performed so far)

unique_lib.diff

Cheers
Mark
Regards from Australia
Mark Harvey
nia
Reply | Threaded
Open this post in threaded view
|

Re: Functionality to add location on disk of library contents

nia
Administrator
I have tested this patch .. Appears to be working as expected ..

Thanks
-nia
Reply | Threaded
Open this post in threaded view
|

Re: Functionality to add location on disk of library contents

Mark Harvey
Administrator
Many thanks for confirming..

I'll continue with the updating the utilities to also understand the new media layout.

Looks like a 1.4 release is coming.
Regards from Australia
Mark Harvey
nia
Reply | Threaded
Open this post in threaded view
|

Re: Functionality to add location on disk of library contents

nia
Administrator
Great .... Already made modification for this in the GUI

Regards,

-nia
pkm
Reply | Threaded
Open this post in threaded view
|

Re: Functionality to add location on disk of library contents

pkm
In reply to this post by Mark Harvey
Hello everyone,
    I am able to install mhvtl on a Oracle linux  & able to perform backup/restore using Oracle Secure Backup.
When I change the Home Directory in device.conf file, to say /opt/mhvtl/stk_media, it works fine - It creates all the tape entries under that folder.

# ls /opt/mhvtl/stk_media
CLN101L4  E01002L4  E01005L4  E01008L4  E01011L4  E01014L4  E01017L4  E01020L4  F01032L5  F01035L5  F01038L5
CLN102L5  E01003L4  E01006L4  E01009L4  E01012L4  E01015L4  E01018L4  F01030L5  F01033L5  F01036L5  F01039L5
E01001L4  E01004L4  E01007L4  E01010L4  E01013L4  E01016L4  E01019L4  F01031L5  F01034L5  F01037L5


But if I mount a filesystem (obtained from a filer) to /opt/mhvtl, '/etc/init.d/mhvtl start' command throws - "make_vtl_media failed.." error message.

# mount
...
192.168.4.10://export/mhvtl_fs on /opt/mhvtl type nfs (rw,addr=192.168.4.10)

# cat /etc/mhvtl/device.conf

VERSION: 5

# VPD page format:
# <page #> <Length> <x> <x+1>... <x+n>
# NAA format is an 8 hex byte value seperated by ':'
# Note: NAA is part of inquiry VPD 0x83
#
# Each 'record' is separated by one (or more) blank lines.
# Each 'record' starts at column 1
# Serial num max len is 10.
# Compression: factor X enabled 0|1
#     Where X is zlib compression factor        1 = Fastest compression
#                                               9 = Best compression
#     enabled 0 == off, 1 == on
#
# fifo: /var/tmp/mhvtl
# If enabled, data must be read from fifo, otherwise daemon will block
# trying to write.
# e.g. cat /var/tmp/mhvtl (in another terminal)


Library: 10 CHANNEL: 00 TARGET: 00 LUN: 00
 Vendor identification: STK
 Product identification: L700
 Unit serial number: XYZZY_A
 NAA: 10:22:33:44:ab:00:00:00
 Home directory: /opt/mhvtl/stk_media
 Backoff: 400
# fifo: /var/tmp/mhvtl

Drive: 11 CHANNEL: 00 TARGET: 01 LUN: 00
 Library ID: 10 Slot: 01
 Vendor identification: IBM
 Product identification: ULT3580-TD5
 Unit serial number: XYZZY_A1
 NAA: 10:22:33:44:ab:00:01:00
 Compression: factor 1 enabled 1
 Compression type: lzo
 Backoff: 400
# fifo: /var/tmp/mhvtl


How do I make the mhvtl to create the VTLs on the filesytem/lun mounted on the client..?


Thanks for your time..

Praveen
Reply | Threaded
Open this post in threaded view
|

Re: Functionality to add location on disk of library contents

Mark Harvey
Administrator
mhvtl daemons run as user 'vtl'..

Check the account 'vtl' has read/write/execute permissions to /opt/mhvtl/stk_media after /opt/mhvtl is mounted via NFS.
Sounds like a file permission issue..

I would suggest adding 'INTR' flag to your NFS mount options as well.
Regards from Australia
Mark Harvey
pkm
Reply | Threaded
Open this post in threaded view
|

Re: Functionality to add location on disk of library contents

pkm
Thanks Mark.. It was indeed a permission issue.
Its working absolutely fine after giving access to vtl user...

Thanks again..
-Praveen