Kernel 4.15 and later break mhvtl kernel module

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Kernel 4.15 and later break mhvtl kernel module

scsirob
Hi Mark,

Have you checked compiling mhVTL against recent kernels? In 4.15 a change to timer functions was introduced which seems to break the kernel module compilation. I'd try to fix this myself if I understood the intricacies, but was wondering if you had a chance to do so?

Rob
Reply | Threaded
Open this post in threaded view
|

Re: Kernel 4.15 and later break mhvtl kernel module

Mark Harvey
Administrator
I have had a patch submitted (a couple of months ago) but just haven’t had time to pull it and try it out..

I’m out of touch for next couple of weeks but plan on catching up on backlog once back from leave.

Sent from my iPad

On Apr 16, 2018, at 20:07, scsirob [via mhVTL - A Linux Virtual Tape Library] <[hidden email]> wrote:

Hi Mark,

Have you checked compiling mhVTL against recent kernels? In 4.15 a change to timer functions was introduced which seems to break the kernel module compilation. I'd try to fix this myself if I understood the intricacies, but was wondering if you had a chance to do so?

Rob



To start a new topic under mhVTL - A Linux Virtual Tape Library, email [hidden email]
To unsubscribe from mhVTL - A Linux Virtual Tape Library, click here.
NAML
Regards from Australia
Mark Harvey
Reply | Threaded
Open this post in threaded view
|

Re: Kernel 4.15 and later break mhvtl kernel module

scsirob
There's a patch for 4.14 from November of last year. That patch doesn't address the timer changes in 4.15.
Hope you can have a look at it.
Reply | Threaded
Open this post in threaded view
|

Re: Kernel 4.15 and later break mhvtl kernel module

Mark Harvey
Administrator
Patches pushed into github over last few hrs.
I haven't tested yet, but looks good.
Appreciate any feedback on success/failure of these patches :)
Regards from Australia
Mark Harvey
Reply | Threaded
Open this post in threaded view
|

Re: Kernel 4.15 and later break mhvtl kernel module

scsirob
Hi Mark,

The patch installs and compiles fine on CentOS 7 with the latest 4.16-5 ml kernel. But something isn't right. When accessing the library for the first time, I do get proper output but at the end I  get a kernel oops. See trace below. After that the system is still responsive and I can do other things.

The second access to the same library results in a kernel panic that I have not been able to capture yet, other than a screenshot. After this the system hangs.




:[  101.193869] ------------[ cut here ]------------
:[  101.193871] Bad or missing usercopy whitelist? Kernel memory overwrite attempt detected to SLUB object 'dma-kmalloc-512' (offset 0, size 32)!
:[  101.193877] WARNING: CPU: 0 PID: 3092 at mm/usercopy.c:81 usercopy_warn+0x8e/0xb0
:[  101.193878] Modules linked in: ch(+) osst st mhvtl(O) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter intel_pmc_core intel_powerclamp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device aesni_intel crypto_simd glue_helper cryptd intel_rapl_perf snd_pcm sg input_leds pcspkr video snd_timer snd soundcore i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c
:[  101.193918]  sd_mod sr_mod cdrom ata_generic pata_acpi crc32c_intel ahci libahci ata_piix e1000 libata serio_raw dm_mirror dm_region_hash dm_log dm_mod dax
:[  101.193926] CPU: 0 PID: 3092 Comm: vtllibrary Tainted: G           O     4.16.5-1.el7.elrepo.x86_64 #1
:[  101.193926] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
:[  101.193927] RIP: 0010:usercopy_warn+0x8e/0xb0
:[  101.193928] RSP: 0018:ffffc90000957cd8 EFLAGS: 00010286
:[  101.193929] RAX: 0000000000000000 RBX: ffffffff8208cfa0 RCX: 0000000000000006
:[  101.193930] RDX: 0000000000000000 RSI: 0000000000000082 RDI: ffff88007fc169d0
:[  101.193931] RBP: ffffc90000957cf8 R08: 0000000000000000 R09: 0000000000000242
:[  101.193931] R10: 0000000000000004 R11: 0000000000000241 R12: 0000000000000020
:[  101.193932] R13: ffff880000098220 R14: 0000000000000000 R15: 0000000000000020
:[  101.193933] FS:  00007f3e06961740(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000
:[  101.193934] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
:[  101.193935] CR2: 00007fc4fddf1038 CR3: 000000007af40005 CR4: 00000000000606f0
:[  101.193937] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
:[  101.193938] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
:[  101.193939] Call Trace:
:[  101.193942]  __check_heap_object+0xc4/0x130
:[  101.193943]  __check_object_size+0xdb/0x1b0
:[  101.193945]  vtl_sg_copy_user+0xed/0x180 [mhvtl]
:[  101.193947]  vtl_c_ioctl+0x2e3/0x7cc [mhvtl]
:[  101.193949]  do_vfs_ioctl+0xaa/0x610
:[  101.193950]  SyS_ioctl+0x79/0x90
:[  101.193952]  do_syscall_64+0x79/0x1b0
:[  101.193954]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
:[  101.193955] RIP: 0033:0x7f3e0625d107
:[  101.193956] RSP: 002b:00007ffdf807c058 EFLAGS: 00000206 ORIG_RAX: 0000000000000010
:[  101.193957] RAX: ffffffffffffffda RBX: 00007ffdf807c110 RCX: 00007f3e0625d107
:[  101.193958] RDX: 00007ffdf807c110 RSI: 0000000000000203 RDI: 0000000000000003
:[  101.193958] RBP: 00000000000000fe R08: 000000000060fe98 R09: 0000000000000000
:[  101.193959] R10: 0000000000000000 R11: 0000000000000206 R12: 0000000000100000
:[  101.193960] R13: 0000000000000000 R14: 00007ffdf807c2f0 R15: 000000000060fe98
:[  101.193960] Code: 07 82 48 0f 45 f2 4c 89 44 24 10 48 89 c2 48 89 4c 24 08 48 89 1c 24 4d 89 d8 4c 89 d1 48 c7 c7 08 d0 08 82 31 c0 e8 82 3a e2 ff <0f> 0b 48 83 c4 18 5b 5d c3 49 c7 c1 91 cc 09 82 4c 89 cb 4d 89
:[  101.193982] ---[ end trace 73f9499224a00a1b ]---

Reply | Threaded
Open this post in threaded view
|

Re: Kernel 4.15 and later break mhvtl kernel module

Mark Harvey
Administrator
OK, I've setup an environment (CentOS 7 + elrepo) and reproduced the error.

Apr 30 16:01:40 mhappliance kernel: ch 3:0:0:0: [ch0] type #4 (dt): 0x1f4+4 [data transfer]
Apr 30 16:01:40 mhappliance kernel: ------------[ cut here ]------------
Apr 30 16:01:40 mhappliance kernel: Bad or missing usercopy whitelist? Kernel memory overwrite attempt detected to SLUB object 'dma-kmalloc-512' (offset 0, size 32)!
Apr 30 16:01:40 mhappliance kernel: WARNING: CPU: 0 PID: 5077 at mm/usercopy.c:81 usercopy_warn+0x8e/0xb0
Apr 30 16:01:40 mhappliance kernel: Modules linked in: ch(+) osst st mhvtl(O) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd glue_helper cryptd vmw_balloon intel_rapl_perf joydev pcspkr input_leds sg vmw_vmci i2c_piix4 shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs
Apr 30 16:01:40 mhappliance kernel: libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc32c_intel serio_raw vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt vmxnet3 fb_sys_fops ttm ata_piix vmw_pvscsi drm libata dm_mirror dm_region_hash dm_log dm_mod dax
Apr 30 16:01:40 mhappliance kernel: CPU: 0 PID: 5077 Comm: vtllibrary Tainted: G           O     4.16.6-1.el7.elrepo.x86_64 #1

I'm assuming it has something to do with the following WARNING:
kernel: Bad or missing usercopy whitelist? Kernel memory overwrite attempt detected to SLUB object 'dma-kmalloc-512' (offset 0, size 32)!

I'll need to check out what has changed with the usercopy() call..
Regards from Australia
Mark Harvey
Reply | Threaded
Open this post in threaded view
|

Re: Kernel 4.15 and later break mhvtl kernel module

Mark Harvey
Administrator
Been researching this one.

Root cause is due to additional sanity checking of data being copied between kernel space and user space.

As of 4.16... a single oops is logged as a method to audit modules not doing the 'right thing'.
The actual memory copy is not blocked (yet) - so apart from the scary oops being logged first use after the module being loaded, everything should behave correctly.

I (we) still have time to figure out how to 'white-list' the memory buffer(s) used.

https://outflux.net/blog/archives/category/security/
Hardened usercopy whitelisting.

https://patchwork.kernel.org/patch/10153431/
+config HARDENED_USERCOPY_FALLBACK
+ bool "Allow usercopy whitelist violations to fallback to object size"
+ depends on HARDENED_USERCOPY
+ default y
+ help
+  This is a temporary option that allows missing usercopy whitelists
+  to be discovered via a WARN() to the kernel log, instead of
+  rejecting the copy, falling back to non-whitelisted hardened
+  usercopy that checks the slab allocation size instead of the
+  whitelist size. This option will be removed once it seems like
+  all missing usercopy whitelists have been identified and fixed.
+  Booting with "slab_common.usercopy_fallback=Y/N" can change
+  this setting.
Regards from Australia
Mark Harvey
Reply | Threaded
Open this post in threaded view
|

Re: Kernel 4.15 and later break mhvtl kernel module

scsirob
Thanks Mark,

I found this reference to whitelist test code that may help? Use of vm_mmap() looks like the key.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6c3521400c345fa2575a6f5b212c215db38c5d93

+static void do_usercopy_kernel(void)
+{
+ unsigned long user_addr;
+
+ user_addr = vm_mmap(NULL, 0, PAGE_SIZE,
+    PROT_READ | PROT_WRITE | PROT_EXEC,
+    MAP_ANONYMOUS | MAP_PRIVATE, 0);
+ if (user_addr >= TASK_SIZE) {
+ pr_warn("Failed to allocate user memory\n");
+ return;
+ }
+
+ pr_info("attempting good copy_to_user from kernel rodata\n");
+ if (copy_to_user((void __user *)user_addr, test_text,
+ sizeof(test_text))) {
+ pr_warn("copy_to_user failed unexpectedly?!\n");
+ goto free_user;
+ }
+
+ pr_info("attempting bad copy_to_user from kernel text\n");
+ if (copy_to_user((void __user *)user_addr, vm_mmap, PAGE_SIZE)) {
+ pr_warn("copy_to_user failed, but lacked Oops\n");
+ goto free_user;
+ }
+
+free_user:
+ vm_munmap(user_addr, PAGE_SIZE);
+}