[e-mail] fails on Debian-based Mint and Ubuntu systems

[Tim Jones]

I'm running into an oddity - which I am sure is path related like a number of others - that is causing the vtllibrary and vtltape services to fail at launch:

$ systemctl status [e-mail]
● [e-mail] - Robot Library Daemon for Virtual Tape & Robot Library
   Loaded: loaded (/usr/lib/systemd/system/vtllibrary@.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2019-04-12 20:44:51 MST; 8min ago
     Docs: man:vtllibrary(1)
           man:vtlcmd(1)
  Process: 950 ExecStart=/usr/bin/vtllibrary -F -qmhvtl -v${VERBOSE} ${DAEMON_DEBUG} (code=exited, status=1/FAILURE)
 Main PID: 950 (code=exited, status=1/FAILURE)

Apr 12 20:44:51 linux64 systemd[1]: Started Robot Library Daemon for Virtual Tape & Robot Library.
Apr 12 20:44:51 linux64 /usr/bin/vtllibrary[950]: find_media_home_directory(): Append library id 0 to default path /opt/mhvtl: /opt/mhvtl/0
Apr 12 20:44:51 linux64 vtllibrary[950]: error: Can not find entry for '0' in config file
Apr 12 20:44:51 linux64 systemd[1]: [e-mail]: Main process exited, code=exited, status=1/FAILURE
Apr 12 20:44:51 linux64 systemd[1]: [e-mail]: Failed with result 'exit-code'.

I believe that the issue is in these lines:

Apr 12 20:44:51 linux64 /usr/bin/vtllibrary[950]: find_media_home_directory(): Append library id 0 to default path /opt/mhvtl: /opt/mhvtl/0
Apr 12 20:44:51 linux64 vtllibrary[950]: error: Can not find entry for '0' in config file
Apr 12 20:44:51 linux64 systemd[1]: [e-mail]: Main process exited, code=exited, status=1/FAILURE

Here's the vtllibrary@.service file content:

[Unit]
Documentation=man:vtllibrary(1) man:vtlcmd(1)
Description=Robot Library Daemon for Virtual Tape & Robot Library
Before=mhvtl.target
Requires=mhvtl-load-modules.service
After=mhvtl-load-modules.service
PartOf=mhvtl.target

[Service]
Type=simple
Environment=VERBOSE=0
Environment=DAEMON_DEBUG=
EnvironmentFile=-/etc/mhvtl/mhvtl.conf
ExecStart=/usr/bin/vtllibrary -F -q%i -v${VERBOSE} ${DAEMON_DEBUG}
ExecStop=/usr/bin/vtlcmd %i exit
KillMode=none
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=mhvtl.target

Any ideas as to the cause of the odd search path and failure?

[Mark Harvey]

Is there any chance of capturing the value of %i from the script ?

I.e. is the correct mhVTL.target being built or is it something to do with the .service file ?

Sent from my iPad

On Apr 13, 2019, at 14:02, Tim Jones [via mhVTL - A Linux Virtual Tape Library] <[hidden email]> wrote:

I'm running into an oddity - which I am sure is path related like a number of others - that is causing the vtllibrary and vtltape services to fail at launch:

$ systemctl status [hidden email]
● [hidden email] - Robot Library Daemon for Virtual Tape & Robot Library
   Loaded: loaded (/usr/lib/systemd/system/vtllibrary@.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2019-04-12 20:44:51 MST; 8min ago
     Docs: man:vtllibrary(1)
           man:vtlcmd(1)
  Process: 950 ExecStart=/usr/bin/vtllibrary -F -qmhvtl -v${VERBOSE} ${DAEMON_DEBUG} (code=exited, status=1/FAILURE)
 Main PID: 950 (code=exited, status=1/FAILURE)

Apr 12 20:44:51 linux64 systemd[1]: Started Robot Library Daemon for Virtual Tape & Robot Library.
Apr 12 20:44:51 linux64 /usr/bin/vtllibrary[950]: find_media_home_directory(): Append library id 0 to default path /opt/mhvtl: /opt/mhvtl/0
Apr 12 20:44:51 linux64 vtllibrary[950]: error: Can not find entry for '0' in config file
Apr 12 20:44:51 linux64 systemd[1]: [hidden email]: Main process exited, code=exited, status=1/FAILURE
Apr 12 20:44:51 linux64 systemd[1]: [hidden email]: Failed with result 'exit-code'.

I believe that the issue is in these lines:

Apr 12 20:44:51 linux64 /usr/bin/vtllibrary[950]: find_media_home_directory(): Append library id 0 to default path /opt/mhvtl: /opt/mhvtl/0
Apr 12 20:44:51 linux64 vtllibrary[950]: error: Can not find entry for '0' in config file
Apr 12 20:44:51 linux64 systemd[1]: [hidden email]: Main process exited, code=exited, status=1/FAILURE

Here's the vtllibrary@.service file content:

[Unit]
Documentation=man:vtllibrary(1) man:vtlcmd(1)
Description=Robot Library Daemon for Virtual Tape & Robot Library
Before=mhvtl.target
Requires=mhvtl-load-modules.service
After=mhvtl-load-modules.service
PartOf=mhvtl.target

[Service]
Type=simple
Environment=VERBOSE=0
Environment=DAEMON_DEBUG=
EnvironmentFile=-/etc/mhvtl/mhvtl.conf
ExecStart=/usr/bin/vtllibrary -F -q%i -v${VERBOSE} ${DAEMON_DEBUG}
ExecStop=/usr/bin/vtlcmd %i exit
KillMode=none
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=mhvtl.target

Any ideas as to the cause of the odd search path and failure?

---

If you reply to this email, your message will be added to the discussion below:

http://mhvtl-a-linux-virtual-tape-library.966029.n3.nabble.com/vtllibrary-mhvtl-service-fails-on-Debian-based-Mint-and-Ubuntu-systems-tp4026247.html

To start a new topic under mhVTL - A Linux Virtual Tape Library, email [hidden email]
To unsubscribe from mhVTL - A Linux Virtual Tape Library, click here.
NAML

[Tim Jones]

Hi Mark,

I dug through this last night and this morning and it appears to be an issue with the mhvtl-device-generator binary and the differences in the systemd configurations on the two platform types.

On a working CentOS 7 system, the mhvtl.target generates the [e-mail], [e-mail], and [e-mail] files (I'm emulating a 2U, 24 slot with 2 drives) and the mhvtl.target.wants is a marker file in /etc/systemd/system/multi-user.target.wants folder.  The mhvtl.target.wants folder is placed into /run/systemd/generator/ and it contains the proper @10, @11, and @12 service files.

On the debian-based systems, I end up with the mhvtl.target.wants folder in /etc/systemd/system/ and the marker files within that folder are not properly numbered for the same configuration of a 24 slot, 2 drive setup - instead, the files are simply vtllibrary@.service and vtltape@.service.  This implies to me that the generator is not expanding the template properly (%i = NULL or empty) on these systems.

Thoughts?  I'm not a fan of systemd, and these types of glitches drive it even further up my list of things that simply should go away in Linux.

Tim

[Tim Jones]

I have verified that this is the case.  By manually creating the proper @## vtllibrary and vtltape service symlink entries in the /etc/systemd/system/mhvtl.target.wants folder, the Debian-based systems do the correct thing :).

[Mark Harvey]

Moving '/usr/lib/systemd/system-generators/mhvtl-device-conf-generator to /lib/systemd/system-generators/' helped tremendously.

And modifying mhvtl-load-modules.service  to find modprobe in /sbin instead of /usr/sbin.

Note: There is a warning from the kernel: When the upstream kernel enforces the check - I'm in big trouble. Hopefully I find time to get the user-space ported over to tcmurunner before then :)
Of course, anybody with more kernel space experience than I is welcome to submit a work around for this.

[  173.663180] ------------[ cut here ]------------
[  173.663182] Bad or missing usercopy whitelist? Kernel memory overwrite attempt detected to SLUB object 'dma-kmalloc-512' (offset 0, size 32)!
[  173.663198] WARNING: CPU: 0 PID: 1650 at mm/usercopy.c:81 usercopy_warn+0x81/0xa0
[  173.663199] Modules linked in: ch(+) osst st mhvtl(OE) binfmt_misc snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm snd_timer snd intel_powerclamp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_rapl_perf soundcore input_leds serio_raw joydev mac_hid vboxguest sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid vboxvideo(CE) ttm drm_kms_helper aesni_intel syscopyarea sysfillrect sysimgblt aes_x86_64 crypto_simd cryptd glue_helper mptsas mptscsih psmouse mptbase scsi_transport_sas fb_sys_fops ahci libahci i2c_piix4 e1000 drm pata_acpi video
[  173.663237] CPU: 0 PID: 1650 Comm: vtllibrary Tainted: G         C OE     4.18.0-17-generic #18-Ubuntu
[  173.663238] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[  173.663239] RIP: 0010:usercopy_warn+0x81/0xa0
[  173.663240] Code: 10 99 41 51 4d 89 d8 48 c7 c0 d9 96 0f 99 49 89 f1 48 89 f9 48 0f 45 c2 48 c7 c7 30 ad 10 99 4c 89 d2 48 89 c6 e8 c1 c5 df ff <0f> 0b 48 83 c4 18 c9 c3 48 c7 c6 ca 95 12 99 49 89 f1 49 89 f3 eb 
[  173.663259] RSP: 0018:ffffb5a500c1fcf8 EFLAGS: 00010282
[  173.663260] RAX: 0000000000000000 RBX: ffff9c75c0098c00 RCX: 0000000000000006
[  173.663261] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff9c76dfc164b0
[  173.663262] RBP: ffffb5a500c1fd10 R08: 0000000000000001 R09: 0000000000000252
[  173.663262] R10: 0000000000000004 R11: 0000000000000000 R12: 0000000000000020
[  173.663263] R13: 0000000000000000 R14: ffff9c75c0098c20 R15: 0000000000000020
[  173.663264] FS:  00007f52f66ce740(0000) GS:ffff9c76dfc00000(0000) knlGS:0000000000000000
[  173.663265] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  173.663266] CR2: 00007f0998f8a000 CR3: 000000003418a003 CR4: 00000000000606f0
[  173.663268] Call Trace:
[  173.663273]  __check_heap_object+0xc2/0x110
[  173.663274]  __check_object_size+0x14c/0x178
[  173.663278]  vtl_sg_copy_user+0xcf/0x120 [mhvtl]
[  173.663280]  vtl_c_ioctl+0x39e/0x4e8 [mhvtl]
[  173.663282]  do_vfs_ioctl+0xa8/0x620
[  173.663286]  ? ipcget+0x202/0x270
[  173.663288]  ksys_ioctl+0x67/0x90
[  173.663290]  __x64_sys_ioctl+0x1a/0x20
[  173.663292]  do_syscall_64+0x5a/0x110
[  173.663296]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  173.663297] RIP: 0033:0x7f52f68023c7
[  173.663298] Code: 00 00 90 48 8b 05 c9 3a 0d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 99 3a 0d 00 f7 d8 64 89 01 48 
[  173.663317] RSP: 002b:00007ffcb41f9c28 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  173.663319] RAX: ffffffffffffffda RBX: 00007ffcb41f9c90 RCX: 00007f52f68023c7
[  173.663319] RDX: 00007ffcb41f9c90 RSI: 0000000000000203 RDI: 0000000000000003
[  173.663320] RBP: 00007ffcb41faed4 R08: 0000000000000001 R09: 0000000000000004
[  173.663320] R10: 0000556779bddad8 R11: 0000000000000246 R12: 0000000000000005
[  173.663321] R13: 00007ffcb41f9cc0 R14: 0000000000000000 R15: 00007ffcb41f9e50
[  173.663322] ---[ end trace 0137f5f0aa058b4a ]---

[Tim Jones]

Nabble must be doing some weird caching on browsers.  I saw the email note about this back on the 14th, but it's just shown up in my browser today (Apr 26th).

[Tim Jones]

Mark Harvey wrote

Moving '/usr/lib/systemd/system-generators/mhvtl-device-conf-generator to /lib/systemd/system-generators/' helped tremendously.

And modifying mhvtl-load-modules.service  to find modprobe in /sbin instead of /usr/sbin.

Success on all of my Debian-based systems.  Hopefully the 4.X kernel crash can be sorted when a drive is loaded and the library attempts to unload to the slot ...