Request for comment: device.conf & library_contents.XX

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Request for comment: device.conf & library_contents.XX

Mark Harvey
Administrator
Until the complete SCSI command set has been implemented by this project, customization of library & device types will be unique per backup application.
i.e. What works for NetBackup may not be correct for TSM, which many not be correct for NetWorker.
So far, there is no one configuration which suits all backup software.

I would like to remove the creation of sample configuration from the rc scripts and replace it with the requirement for the user to run some sort of 'create library with XX drive type' script to create the initial configuration files.

This way, a message (or documentation) advising of best parameters could be provided for each backup application.

I'm after your thoughts and comments.

Cheers
Mark
Regards from Australia
Mark Harvey
nia
Reply | Threaded
Open this post in threaded view
|

Re: Request for comment: device.conf & library_contents.XX

nia
Administrator
Mark Harvey wrote
I would like to remove the creation of sample configuration from the rc scripts and replace it with the requirement for the user to run some sort of 'create library with XX drive type' script to create the initial configuration files.
Cheers
Mark
Mark, this sounds like a good idea .. As I am currently working on a Web console GUI, I have already thought of the same thing .. I plan on having a configuration wizard that will ask if user wants to create standard (major vendors selected) or specific backup application recommended configuration along with a custom/expert mode ..I have some of this already working and I like it so far.

I suspect the same thing can be done via a script that people can run after they install MHVTL ..

People in this forum can submit the best configuration that have already tested and certify that it works best for their app which you can use in your script.


Reply | Threaded
Open this post in threaded view
|

Re: Request for comment: device.conf & library_contents.XX

Mark Harvey
Administrator
n.i.a wrote
Mark, this sounds like a good idea .. As I am currently working on a Web console GUI, I have already thought of the same thing .. I plan on having a configuration wizard that will ask if user wants to create standard (major vendors selected) or specific backup application recommended configuration along with a custom/expert mode ..I have some of this already working and I like it so far.

I suspect the same thing can be done via a script that people can run after they install MHVTL ..

People in this forum can submit the best configuration that have already tested and certify that it works best for their app which you can use in your script.
Yep, we can both use the data.

I'd like to use the same php script.

One instance, where you have to specify all required args
/usr/bin/php createNewLibrary.php -drives=ULT3580-TD1 -drive_count=8 -library=L180 -library_vendor=STK -num_slots=40

Perhaps fill in barcodes based on some sort of seeding (-TD1 indicates LTO-1 media, so perhaps L180xxL1 (where xx is the slot number)

If the createNewLibrary.php does not detect these command line options, it can assume it's called via the web interface and act appropriately.

Obviously, 'createNewLibrary.php' needs a better name..
Regards from Australia
Mark Harvey
nia
Reply | Threaded
Open this post in threaded view
|

Re: Request for comment: device.conf & library_contents.XX

nia
Administrator
Yes, I have already did this with a ksh script that is executed by PHP .. but I will look into converting it to PHP to keep the code limited to html+php only .. Below feeds in the variables needed to build both device.conf & library_contents.XX


Reply | Threaded
Open this post in threaded view
|

Re: Request for comment: device.conf & library_contents.XX

Andreas Piesk
In reply to this post by Mark Harvey
good idea but i would like to go a step further and replace device.conf and library_contents.XX with a xml configuration file. i know, it would create a dependency to libxml2 or whatever xml library will be used but it has some advantages:

- DTD enforces syntax correctness
- parameter retrieval is easier, you can use xpath expressions like "//vtl/libraries/library[@id='NBU']/slots"
- it can easily be edited, either using a text editor or by a script using xml functions (php, perl, etc. pp)
- you can specify as many libraries as you want (one for NBU, one for TSM, one for BACULA) because every library has their own set of drives, picker, maps, slots
- the tapes are not tied to a library, so you can assign all tapes to all slots of all libraries as long as you don't use the libaries concurrently

i've attached vtl.dtd and vtl.xml as an example. if you want i can also upload some demo code how to retrieve the attributes. currently i'm changing vtllibrary to use a xml configuration, just to see how it works.

-ap

vtl.dtdvtl.xml
IKP
Reply | Threaded
Open this post in threaded view
|

Re: Request for comment: device.conf & library_contents.XX

IKP
In reply to this post by nia
NIA,

Whilst some may utilise a GUI to configure the library, to get the most from the functionality provided by Mark's software; unless He is planning to remove some of the functionality, the tool needs to be capable of building

a) Multiple libraries in the same configuration file

b) Each library inturn capable of being configured with differeing media types


Therefore I would caution anyone planning to move towards a single configuration approach - the software is far more functional than possible configurable from a single GUI.

In reply to Mark's original query - I attach a copy of one of the environments I currently have running in a number of virtualised environments.

device.conf

Whilst most of the environments (Which are used for training and software functionality testing) are running NBU and NSR, I am currently investigating why BakBone NetVault has issues when communicating with the library. One of which is that there appears to be a discovery / enquire issue when there are more than 4 devices or 4 media slots.


IKP
Reply | Threaded
Open this post in threaded view
|

Re: Request for comment: device.conf & library_contents.XX

Mark Harvey
Administrator
I would love to see the xml version, even if not fully complete.

It would help ease my learning curve into xml :)

The learning curve and the fact it introduces another dependency are probably the two main factor I haven't gone down the xml route earlier.

FWIW: The external dependency is not that big a deal as most distributions would already include the xml base functionality.

php / perl already handle xml with little to no effort, so updating nia's GUI "shouldn't (tm)" be that difficult.

Cheers
Mark
(tm) Famous last words
Regards from Australia
Mark Harvey
nia
Reply | Threaded
Open this post in threaded view
|

Re: Request for comment: device.conf & library_contents.XX

nia
Administrator
In reply to this post by IKP
IKP wrote
Therefore I would caution anyone planning to move towards a single configuration approach - the software is far more functional than possible configurable from a single GUI.

IKP
IKP,

Actually building this type of GUI should add additional power to MHVTL and extra incentive for some one like you who have to keep changing configuration on the fly .. I don't think Mark will be scaling back on any functionality. The GUI will just make easier to scale MHVTL even higher. I personally had a device.conf at one time that had over 43 different libraries + over 200 drives with many different types. I knew I could not run all at the same time, so I spent much time managing that manually .. but having a GUI to Add + Remove+ Modify on the fly easily with few point and click without asking many questions is really nice and can help avoid many mistakes. In my view, the single configuration approach is one out of many options will be included ..
nia
Reply | Threaded
Open this post in threaded view
|

Re: Request for comment: device.conf & library_contents.XX

nia
Administrator
In reply to this post by Andreas Piesk
Mark,

What about having a configuration file (could be xml as suggested) that include many,many different libraries and drives already configured and have a new extra option called ACTIVE=Yes/No for each library and drive (MHVTL will just bypass the INACTIVE devices when it starts up) where then people can easily turn on/off devices as they wish. The GUI can just do that with a single point and click ...


Reply | Threaded
Open this post in threaded view
|

Re: Request for comment: device.conf & library_contents.XX

Mark Harvey
Administrator
n.i.a wrote
Mark,

What about having a configuration file (could be xml as suggested) that include many,many different libraries and drives already configured and have a new extra option called ACTIVE=Yes/No for each library and drive (MHVTL will just bypass the INACTIVE devices when it starts up) where then people can easily turn on/off devices as they wish. The GUI can just do that with a single point and click ...
Oh, I like that idea..

Although, it would be just as easy to have the 'pre-canned' VTL configuration(s) in the web interface and once the user selects the robot(s) they want/need, it populates or adds to the existing config file.

Cheers
Mark
Regards from Australia
Mark Harvey
Reply | Threaded
Open this post in threaded view
|

Re: Request for comment: device.conf & library_contents.XX

Mark Harvey
Administrator
In reply to this post by nia
n.i.a wrote
What about having a configuration file (could be xml as suggested) that include many,many different libraries and drives already configured and have a new extra option called ACTIVE=Yes/No for each library and drive
The idea of being able to 'disable' a (one or more) drive within the library would be a nice addition too.

From a 'support' perspective, it would simulate a hardware vendor pulling a drive out of the library until a replacement is ordered and put back in.
This would be a nice feature.

Cheers
Regards from Australia
Mark Harvey
IKP
Reply | Threaded
Open this post in threaded view
|

Re: Request for comment: device.conf & library_contents.XX

IKP
Mark Harvey wrote
The idea of being able to 'disable' a (one or more) drive within the library would be a nice addition too.

From a 'support' perspective, it would simulate a hardware vendor pulling a drive out of the library until a replacement is ordered and put back in.
This would be a nice feature.

Cheers
From a training perspective, being able not only to disable / re-enable a drive would be of interest, so would being able to change the drives serial number when it is disabled. This would therefore fully simulate an engineer replacing a failed drive and require the software to be reconfigured.

IKP
nia
Reply | Threaded
Open this post in threaded view
|

Re: Request for comment: device.conf & library_contents.XX

nia
Administrator
 If the a drive to be replaced by a hardware vendor SE, then change/replace the device in the device.conf to reflect the new drive configuration .. You can put a new serial number then.

I keep trying not to think of MHVTL as a simulator tool and want to use it as a real VTL.

This goes back to the LTO Options feature discussion where if some one wish to  make an LTO-1 drive read and write from an LTO-5 media or the opposite then they can .. I strongly want to see the developer steer away from making MHVTL as a simulator tool to be used for play and education and more as close as possible to either a real commercial VTL or a real physical tape library and a drive. I want my backup application to interact with MHVTL just like it does with any other real commercial VTL because my main goal is to test/learn my backup applications.

I hope you all understand my point here ...

IKP
Reply | Threaded
Open this post in threaded view
|

Re: Request for comment: device.conf & library_contents.XX

IKP
 
n.i.a wrote
 If the a drive to be replaced by a hardware vendor SE, then change/replace the device in the device.conf to reflect the new drive configuration .. You can put a new serial number then.
This approach would require the entire tape library being stopped and started, in a physical environment especially large Enterprise libraries such as the L700 this is NOT the case.

Also where the single MHVTL environment is presenting multiple libraries to backup systems stopping and re-starting the entire estate is not really feasible. So the question is fore Mark, can He see away to change the serial number on thr fly ?

n.i.a wrote
This goes back to the LTO Options feature discussion where if some one wish to  make an LTO-1 drive read and write from an LTO-5 media or the opposite then they can .. I strongly want to see the developer steer away from making MHVTL as a simulator tool to be used for play and education and more as close as possible to either a real commercial VTL or a real physical tape library and a drive. I want my backup application to interact with MHVTL just like it does with any other real commercial VTL because my main goal is to test/learn my backup applications.
I agree Mark's software should where ever possible act and respond to backup software functionality especially with drive emulation. However, this may cause other issues with software other then NetBackup and NetWorker where the software attempts to be "inteligent" with library functionality. For instance backup software that has knowledge opf extended functionality provided by the library which is not in the emulation.

One resoluition to this would be for a core design decision in line with most commercial VTL products would be to only emulate a small number of libraries - for this list to be documented and only these focused on during software developement.

Cheers

IKP
Reply | Threaded
Open this post in threaded view
|

Re: Request for comment: device.conf & library_contents.XX

Mark Harvey
Administrator
Comments inline.  

Sent from my iPhone

On 25/11/2010, at 18:11, "IKP [via MHVTL - Linux Virtual Tape Library - Community Forums]"<[hidden email]> wrote:

 
n.i.a wrote:
 If the a drive to be replaced by a hardware vendor SE, then change/replace the device in the device.conf to reflect the new drive configuration .. You can put a new serial number then.
This approach would require the entire tape library being stopped and started, in a physical environment especially large Enterprise libraries such as the L700 this is NOT the case.

However, it is recommend to stop the backup application from attempting to use the robot & drives while maintenance is being performed. So cycling the daemons related to the one library is doable (currently) while leaving the other libraries running. Not ideal but better than shutting down everything. 

Also where the single MHVTL environment is presenting multiple libraries to backup systems stopping and re-starting the entire estate is not really feasible. So the question is fore Mark, can He see away to change the serial number on thr fly ?
Just the serial number ?

I could see where h/w vendor swaps an LTO drive and say, swaps a HP drive for an IBM. If I'm going to allow restart/reconfig of devices, I might as well plan for everything :)


n.i.a wrote:
This goes back to the LTO Options feature discussion where if some one wish to  make an LTO-1 drive read and write from an LTO-5 media or the opposite then they can .. I strongly want to see the developer steer away from making MHVTL as a simulator tool to be used for play and education and more as close as possible to either a real commercial VTL or a real physical tape library and a drive. I want my backup application to interact with MHVTL just like it does with any other real commercial VTL because my main goal is to test/learn my backup applications.
I agree Mark's software should where ever possible act and respond to backup software functionality especially with drive emulation. However, this may cause other issues with software other then NetBackup and NetWorker where the software attempts to be "inteligent" with library functionality. For instance backup software that has knowledge opf extended functionality provided by the library which is not in the emulation.

One resoluition to this would be for a core design decision in line with most commercial VTL products would be to only emulate a small number of libraries - for this list to be documented and only these focused on during software developement.

I'm currently implementing a "personality module" where all unique features for drive X can be separate from core functionality. I'm sure that limited free time means that one or two types will be emulated in fine details, while others will be very basic. It all depends on what the vendors publish (which is why IBM LTO is top so far). 

OK. Way too much talk about emulating VTLs. I'm all for emulating real library & drives. 

VTLs have little practical use (now that backup software has discovered disks).

My VTL is good for training, demos, support & coders instead of expensive hardware. 

Which is why I have MAP ports, encryption, cleaning media etc. No commercial VTL would worry about such features as it makes little sense these features in their market.

Real VTLs worry about SPEED. 


Cheers

IKP



Mark
Regards from Australia
Mark Harvey
Reply | Threaded
Open this post in threaded view
|

Re: Request for comment: device.conf & library_contents.XX

Andreas Piesk
In reply to this post by Mark Harvey
Marc, sent you a first draft. the code depends on libxml2 which should be available on all linux systems (part of gnome).

disabling components with 'active=n' is easy, it's only a change of the xpath expression.

-ap
nia
Reply | Threaded
Open this post in threaded view
|

Re: Request for comment: device.conf & library_contents.XX

nia
Administrator
In reply to this post by Mark Harvey
Maybe a long shot , store all of your configs in MySQL DB !
Reply | Threaded
Open this post in threaded view
|

Re: Request for comment: device.conf & library_contents.XX

Mark Harvey
Administrator
You're wright.. It is a long shot.
Way, way too much overhead + dependency hell.

The config information is basically static.
i.e. read each time we start the vtl, written to once and modified very frequently.
Sounds like an ldap database critera..

But no, I'm not going down that path ;)

An ad-hoc format config file is bad - but is working for now.
xml format adds extra dependencies, but is (IMHO), the correct format.
Adding the config file inside a database is just way too much overhead.

Sounds like a story about the three bears..
And xml is just right :)
Regards from Australia
Mark Harvey
Reply | Threaded
Open this post in threaded view
|

Request for comment: xml config file format.

Mark Harvey
Administrator
In reply to this post by Andreas Piesk
Andreas Piesk wrote
Mark, sent you a first draft. the code depends on libxml2 which should be available on all linux systems (part of gnome).

disabling components with 'active=n' is easy, it's only a change of the xpath expression.

-ap
Hi Andreas,

One thing not covered by your xml template layout is the drive type / media type handling capabilities.

I can see what needs to be done, I just can't seem to convert that into xml / dtd format.

I'd like to have a drive_type field similar to the media type field

<!ELEMENT tape                  EMPTY>
<!ATTLIST tape
   id                           ID                                      #REQUIRED
   type                         (CLN|W|S3|X4|L1|L2|L3|L4|L5|TA|JA|JB|JW|JX)             #REQUIRED
   barcode                      CDATA                                   #IMPLIED
>


It would then be convenient if another ELEMENT type was defined which contained the rules.
i.e.
If the drive was type L4, then media type L1 & L2 are READ-ONLY, media type L3 & L4 are READ-WRITE etc..

Something like :

<drive media matching>
 ID -> L4 (drive type is LTO4, and here are the rules to suit this drive type)
 READ-ONLY: L1 L2
 READ-WRITE: L3 L4
 WORM: L3 L4
 ENCRYPTION: L4
</drive media matching>

I'd like to add the following type to the drive ATTLIST
   drive_type                (CLN|W|S3|X4|L1|L2|L3|L4|L5|TA|JA|JB|JW|JX)             #REQUIRED

Hopefully I've explained my problem.

Cheers
Mark
Regards from Australia
Mark Harvey
Reply | Threaded
Open this post in threaded view
|

Re: Request for comment: xml config file format.

Andreas Piesk
i understand what you want to accomplish but i didn't uderstand the drive type list you posted:

   drive_type                (CLN|W|S3|X4|L1|L2|L3|L4|L5|TA|JA|JB|JW|JX)             #REQUIRED

i thought these are media types but i'm not familar with tape technology.
to come up with something useful, i need to know more about the relation between tape drive type and media type. as far as i understand, these are the drive types:

drive_types = {L1, L2, L3, L4, L5)

and these are the media types:

media_types = {CLN, W, S3, X4, L1, L2, L3, L4, L5, TA, JA, JB, JW, JX}

and there are the operation modes:

op_mode = {RO, RW, WORM, ENC} (what about WORM and ENC? are these media dependent?)

each drive type supports 1-n media types/op mode combinations. for example drive type L1 would be:

L1 = {{CLN, ?}, {W, ?}, {S3, ?}, {X4, ?}, {L1, RO}, {L1, RW}, {L1, WORM?}, {L1, ENC?}, ...}

i think such a list could be modelled in a DTD but i'm not sure if it's possible to use IDREFs which would ensure that only valid combinations are in the xml file.

could you give a list of the drive types, media types, op modes and the rule set of at least one drive type so i can get an idea of the relations between these sets?

do you need a drive type at all, i mean, do you need to know that it's a LTO-5 drive or is it sufficient to say: this is a drive which supports LTO5 in RO, RW, WORM and ENC?

regards,
-ap
12