VMware Tool Patch for Linux Kernel 2.6.37/38

Posted by February 26, 2011 in Technology

This post is not longer maintained since VMWare has announced the latest version of 3.1.3 (416484) which is able to work with the linux 2.6.38 (which is being used in Ubuntu 11.04). I’ve tested the latest version, and I’ve updated my virtual ubuntu as 11.04, and verified it works fine in everything. Thank you for all nice and wonderful comments and suggestions on this post.

This is a note for patching VMware tool for linux kernel 2.6.37/38 guest OS while using VMware Fusion.

Here is my environment for your reference.

  • Host OS: Mac OS 10.6.6
  • Guest OS: Ubuntu 10.10 + Kernel Updated (2.6.38)
  • VMware Fusion: 3.1.2 (332101)

Since the vmware-tool in vmware fusion is not aware of the changes of the latest kernel structure, it needs to be patched in order to get correct kernel module for vmware tool such like file sharing, clipboard sharing, etc.

Here is the walk-through steps:

  1. Install VMware Tool
  2. Copy vmware-tools-distrib directory into a local directory (say, ~/tmp/vmware-tools-distrib) by 
     mkdir -p ~/tmp/ && tar -C ~/tmp -xvzf /media/VMware\ Tools/VMwareTools-8.4.5-332101.tar.gz
  3.  cd ~/tmp/vmware-tools-distrib/lib/modules/source
  4. tar -xvf vmci.tar && tar -xvf vmhgfs.tar && tar -xvf vsock.tar
  5. copy patch files (0001-vmwaretool-vmci-Linux-2.6.37.patch, 0001-vmwaretool-vmhgfs-Linux-2.6.37.patch, 0001-vmwaretool-vsock-Linux-2.6.37.patch) into ~/tmp/vmware-tools-distrib/lib/modules/source
  6.  cd vmci-only && git apply --ignore-whitespace ../0001-vmwaretool-vmci-Linux-2.6.37.patch && cd ..
  7.  cd vmhgfs-only && git apply --ignore-whitespace ../0001-vmwaretool-vmhgfs-Linux-2.6.37.patch && cd ..
    • Caution: For now (05/07/2011), this will pass the compile error, but a segment fault error is caused while creating/removing a file on the filesystem. If I find a resolution, I will update it. (Thanks, Chris.) 
  8.  cd vsock-only && git apply --ignore-whitespace ../0001-vmwaretool-vsock-Linux-2.6.37.patch && cd ..
  9.  rm -rf vmci.tar vmhgfs.tar vsock.tar
  10.  tar -cvf vmci.tar vmci-only && tar -cvf vmhgfs.tar vmhgfs-only && tar -cvf vsock.tar vsock-only
  11.  rm -rf vmci-only vmhgfs-only vsock-only
  12. Install the vmware tool by 
     cd ~/tmp/vmware-tools-distrib && ./vmware-install.pl

Here is the patches for your references:

  1. 0001-vmwaretool-vmci-Linux-2.6.37.patch: 
    From ad10b5458d4aa6d5df7b913174185d395943ce32 Mon Sep 17 00:00:00 2001
From: YOUNGWHAN SONG <breadncup@gmail.com>
Date: Fri, 25 Feb 2011 22:16:19 -0800
Subject: [PATCH] Linux 2.6.37 Patch

---
 shared/compat_semaphore.h |    4 ++--
 vmci_drv.c                |   12 +++++++-----
 2 files changed, 9 insertions(+), 7 deletions(-)

diff --git a/shared/compat_semaphore.h b/shared/compat_semaphore.h
index c2902f0..0add974 100644
--- a/shared/compat_semaphore.h
+++ b/shared/compat_semaphore.h
@@ -28,7 +28,7 @@
 #endif
 
 
-#if defined CONFIG_PREEMPT_RT && LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 31)
+#if (defined CONFIG_PREEMPT_RT && LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 31)) || LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 37)
    /*
     * The -rt patch series changes the name of semaphore/mutex initialization
     * routines (across the entire kernel).  Probably to identify locations that
@@ -41,7 +41,7 @@
       #define DECLARE_MUTEX(_m)  DEFINE_SEMAPHORE(_m)
    #endif
    #ifndef init_MUTEX
-     #define init_MUTEX(_m) semaphore_init(_m)
+      #define init_MUTEX(_m) sema_init(_m,1)
    #endif
 #endif
 
diff --git a/vmci_drv.c b/vmci_drv.c
index ac64f02..14ac3b5 100644
--- a/vmci_drv.c
+++ b/vmci_drv.c
@@ -73,7 +73,7 @@ static int vmci_probe_device(struct pci_dev *pdev,
 static void vmci_remove_device(struct pci_dev* pdev);
 static int vmci_open(struct inode *inode, struct file *file);
 static int vmci_close(struct inode *inode, struct file *file);
-static int vmci_ioctl(struct inode *inode, struct file *file, 
+static long vmci_ioctl(struct file *file, 
                       unsigned int cmd, unsigned long arg);
 static unsigned int vmci_poll(struct file *file, poll_table *wait);
 #if LINUX_VERSION_CODE < KERNEL_VERSION(2, 6, 19)
@@ -93,7 +93,7 @@ static struct file_operations vmci_ops = {
    .owner   = THIS_MODULE,
    .open    = vmci_open,
    .release = vmci_close,
-   .ioctl   = vmci_ioctl,
+   .unlocked_ioctl   = vmci_ioctl,
    .poll    = vmci_poll,
 };
 
@@ -496,9 +496,8 @@ vmci_close(struct inode *inode,  // IN
  *-----------------------------------------------------------------------------
  */
 
-static int
-vmci_ioctl(struct inode *inode,  // IN
-           struct file *file,    // IN
+static long
+vmci_ioctl(struct file *file,    // IN
            unsigned int cmd,     // IN
            unsigned long arg)    // IN
 {
@@ -506,10 +505,12 @@ vmci_ioctl(struct inode *inode,  // IN
    return -ENOTTY;
 #else
    int retval;
+   lock_kernel();
    VMCIGuestDeviceHandle *devHndl =
       (VMCIGuestDeviceHandle *) file->private_data;
 
    if (devHndl == NULL) {
+      unlock_kernel();
       return -EINVAL;
    }
 
@@ -669,6 +670,7 @@ vmci_ioctl(struct inode *inode,  // IN
       break;
    }
 
+   unlock_kernel();
    return retval;
 #endif
 }
-- 
1.7.4.1
  2. 0001-vmwaretool-vmhgfs-Linux-2.6.37.patch: 
    From 2d1de0f0ea3ef2fce534944431c096baed1e423b Mon Sep 17 00:00:00 2001
From: YOUNGWHAN SONG <breadncup@gmail.com>
Date: Fri, 25 Feb 2011 23:07:15 -0800
Subject: [PATCH] vmware:vmhgfs: Support new linux kernel 2.6.37

---
 super.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/super.c b/super.c
index f6f26ff..c5627f5 100644
--- a/super.c
+++ b/super.c
@@ -70,7 +70,7 @@ struct super_operations HgfsSuperOperations = {
 #ifndef VMW_USE_IGET_LOCKED
    .read_inode    = HgfsReadInode,
 #endif
-   .clear_inode   = HgfsClearInode,
+   .evict_inode   = HgfsClearInode,
    .put_super     = HgfsPutSuper,
    .statfs        = HgfsStatfs,
 };
-- 
1.7.4.1
  3. 0001-vmwaretool-vsock-Linux-2.6.37.patch: 
    From dea787ae80d9a6267be21928efc164d0b4f1d4c6 Mon Sep 17 00:00:00 2001
From: YOUNGWHAN SONG <breadncup@gmail.com>
Date: Fri, 25 Feb 2011 23:13:28 -0800
Subject: [PATCH] vmwaretool:vsock:Linux-2.6.37

---
 shared/compat_semaphore.h |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/shared/compat_semaphore.h b/shared/compat_semaphore.h
index 802d174..0add974 100644
--- a/shared/compat_semaphore.h
+++ b/shared/compat_semaphore.h
@@ -28,7 +28,7 @@
 #endif
 
 
-#if defined CONFIG_PREEMPT_RT && LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 31)
+#if (defined CONFIG_PREEMPT_RT && LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 31)) || LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 37)
    /*
     * The -rt patch series changes the name of semaphore/mutex initialization
     * routines (across the entire kernel).  Probably to identify locations that
@@ -41,7 +41,7 @@
       #define DECLARE_MUTEX(_m)  DEFINE_SEMAPHORE(_m)
    #endif
    #ifndef init_MUTEX
-      #define init_MUTEX(_m) semaphore_init(_m)
+      #define init_MUTEX(_m) sema_init(_m,1)
    #endif
 #endif
 
-- 
1.7.4.1

p.s. Thanks to Vyacheslav, I corrected the conditional branch definition. (03/15/2011)

You May Also Like

MAME dual monitor thumbnail

How To Use Dual Monitor in MAME in macOS (Sonoma)

A Great Clipboard tool for macOS

54 Comments

  3. Hello

    I am looking at this line:
    #if defined (CONFIG_PREEMPT_RT && LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 31)) || LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 37)

    Should not it be something like this to compile:
    #if (defined CONFIG_PREEMPT_RT && LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 31)) || LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 37)
    ?

    Reply

  4. I’m getting an additional error:

    make[1]: Entering directory `/usr/src/linux-2.6.38-gentoo'
  CC [M]  /tmp/vmware-root/modules/vmxnet3-only/vmxnet3.o
/tmp/vmware-root/modules/vmxnet3-only/vmxnet3.c: In function 'vmxnet3_restore_vlan':
/tmp/vmware-root/modules/vmxnet3-only/vmxnet3.c:2213:27: error: 'VLAN_GROUP_ARRAY_LEN' undeclared (first use in this function)

    This can be fixed by changing the constant to VLAN_N_VID, as a git commit log says that it was renamed.

    Reply

  7. cd vmci-only && git apply –ignore-whitespace ../0001-vmwaretool-vmci-Linux-2.6.37.patch && cd ..
    ../0001-vmwaretool-vmci-Linux-2.6.37.patch:42: trailing whitespace.
    static long vmci_ioctl(struct file *file,
    error: patch failed: vmci_drv.c:73
    error: vmci_drv.c: patch does not apply

    cd vmhgfs-only && git apply –ignore-whitespace ../0001-vmwaretool-vmhgfs-Linux-2.6.37.patch && cd ..
    error: patch failed: super.c:70
    error: super.c: patch does not apply

    Problems…… pls help!

    Reply

    1. Hi Zorzhes,

      Sorry for delayed answer since I’ve been busy in workplace.

      It looks your environment and mine are different. Could you check yours with mine? Please, refer to this:

      Host OS: Mac OS 10.6.6
      Guest OS: Ubuntu 10.10 + Kernel Updated (2.6.38)
      VMware Fusion: 3.1.2 (332101)

      Thanks,
      Daniel

      Reply
  11. Pingback: Ubuntu Desktop 11.04 的 Unity 三大崩潰 | YORKXIN×YORKXIN

  13. Thanks for suggesting these patches, but unlinking files on a hgfs-mounted filesystem seg faults for me. A strace of “rm test1 “ends with:

    —-

    unlinkat(AT_FDCWD, “test1”, 0
    +++ killed by SIGSEGV +++
    Segmentation fault

    —-

    Ubuntu 11.04 guest kernel: Linux ubuntu 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:50 UTC 2011 i686 i686 i386 GNU/Linux
    Mac OS X host: Darwin 10.7.0 Darwin Kernel Version 10.7.0: Sat Jan 29 15:17:16 PST 2011; root:xnu-1504.9.37~1/RELEASE_I386 i386
    VMware Fusion: Version 3.1.2 (332101)

    Reply

      1. Thank you for the comment, Chris.

        I’ve tested, and yes, there is a bug for causing segmentation fault with above patches. I will try to debug for it, and update it. For a while, I put your precious comment on my main post as well.

        Thank you!
        Daniel

        Reply

  14. This seems to have worked for me (openSUSE 11.4 (x86_64), same version of Fusion as you). Thanks much for posting this.

    Reply

  15. This has been bugging for a couple months. VMWare owes you some $$$ for fixing their customers’ problems! thanks for the patches.

    Reply

  16. This also works on Fedora 15, however trying to read any of the files on the hgfs share does cause the mentioned seg fault, the only thing you can really do is see what files are on the share

    this happens on fedora 15 kernel versions 2.6.38.6-27 and 2.6.38.6-26.rc1

    Reply

    1. That’s true, Tom.

      I planned to fix it, but it’s hard to find free time to look at. If I find the resolution, I will update it.

      Thanks,
      Daniel

      Reply

    2. All is sorted with the latest version of VMware Tools (8.4.7) in the latest version of VMware Fusion (3.1.3) for Fedora, recommend an update.

      Reply

  17. Hi,

    I have been struggling terribly trying to fix my file sharing. I found your patches but am struggling. Here is error… Any assistance is greatly appreciated. Thank You in advance!

    Host OS: Mac OS 10.6.7
    Guest OS: Ubuntu 11.4 + Kernel Updated (2.6.38)
    VMWare 3.1.3 (416484)

    The patches are not installing. I receive the following errors for vmhgfs and vmci – for vsock I receive directory not found.

    jseeman@ubuntu:~/tmp/vmware-tools-distrib/lib/modules/source$ cd vmhgfs-only && git apply –ignore-whitespace ../0001-vmwaretool-vmhgfs-Linux-2.6.37.patch && cd ..
    error: patch failed: super.c:70
    error: super.c: patch does not apply

    Regards, Jill

    Reply

    1. Hi Jill,

      Sorry for delayed response, and I’ve tested exactly same as your environment, but I don’t have any issue without any patch I’ve updated here. So, Jill, please don’t apply anything posted here, and just use as VMWare 3.1.3 (416484) with updated Ubuntu (11.04). That should be fine to yours.

      Thanks,
      Daniel

      Reply

    2. It depends on a nemubr of things if both machines run Windows and the virus is one that spreads across a network and you have no virus checker on your host machine then yes, it can spread. If both machines run Windows and you move files between the machines and you have no virus checker on the host then yes.If you have a virus checker on the host then no.If the host and the guest run significantly different software (i.e. one is Windows and the other is Linux) then the virus won’t infect both machines under any circumstance.

      Reply

  18. Daniel,

    I couldn’t get the shares to work so I upgraded my VMWare fusion version 3.1.3 after my first attempt at installing. I just tried to install the tools again and it seems to install successfully but then it asks if i would like the installer to lauch the configuration file. I say YES. It then goes through a sequence of steps and always finishes with:

    Creating a new initrd boot image for the kernel.
    update-initramfs: Generating /boot/initrd.img-2.6.38-8-generic
    initctl: Job failed to start
    Unable to start services for VMware Tools

    Execution aborted.

    Found VMware Tools CDROM mounted at /media/VMware Tools. Ejecting device
    /dev/sr0 …

    It seems to be installed as my video drivers look better etc however there is definitely nothing in my /mnt/hgfs and I have two shares configured in my VMWare fushion settings.

    No worries about the delay, I appreciate your help whenever you have time.

    Thank You.
    Jill

    Reply

    1. Hmmmm…that’s weird. I have no problem of sharing files over /mnt/hgfs with the Ubuntu 11.4 and the VMware 3.1.3 (416484), and didn’t see any error while updating.

      Just in case, I share how I did to upgrade.

      1. Update Ubuntu 11.04 from 10.10, and reboot the virtual machine.
      2. Install VMware Tool by clicking menu from “Virtual Machine” -> “Install VMware Tools”
      3. Untar /media/VMware\ Tools/VMwareTools-8.4.7-416484.tar.gz in a temporary directory.
      4. Launch vmware-install.pl in the directory as a root user privilege (e.g., “sudo ./vmware-install.pl”), and choose everything as default one.

      I believe you’ve done like above, but you may want to try again from the beginning. If you previously untarred or applied patches, please remove them all and get fresh vmware tools, and try again.

      Let me know if it keeps causing errors.

      Daniel

      Reply

  19. Hello,

    Well I tried again by completely uninstalling the vmware tools. To do this I removed all directories I could find related to vmware, restarted ubuntu and then reinstalled. Unfortunately the exact same response ended and with the same error.

    Can you provide instructions on how to ensure I have removed everything required? I am not sure if there are patches still resident as I have been trying to get this to work from different users suggestions I read on the web.

    As I am new to Linux I see directories and files everywhere outside the home directory.

    Does the error message give us any clues?

    initctl: Job failed to start
    Unable to start services for VMware Tools

    If I can’t get this to work, do you have another suggestion to set up sharing files between guest and host?

    Thank you again
    Jill

    Reply

  20. Sorry I first used the sudo vmware-uninstall-tools.pl before removing the directories. Jill

    Reply

  21. Hi Daniel,

    After too many hours of frustration I blew away my virtual and started fresh…after a clean install, the tools installed correctly and my shares are now visible.

    Thank you for all your time.

    Cheers…

    Reply

  22. Hi Daniel,

    After too many hours of frustration I blew away my virtual and started fresh…after a clean install, the tools installed correctly and my shares are now visible.

    Cheers…

    Reply

  23. As all say, new fusion updates, use new tools… see all shared folders and … hit a snag. As long as I edit the files in the Ubuntu VM not worries, but when I modify the file outside the VM, then the metadata of the files becomes unreadable by ubuntu. Looked for errors and just could not find in all /var/log/ anything that would look like a hgfs error or a connection error.

    Anyone, any idea?

    Cheers

    Reply

  24. Hi Daniel,

    For some reason my shared drives unmounts itself.

    Is there a command that can mount a persistent drive? Or can I mount the drive on startup?

    At the moment I have to run this sudo mount -t vmhgfs .host:/ /mnt/hgfs to get the folders to show.

    Regards,
    Jill

    Reply

  27. Thanks to Vyacheslav for the guide !
    I’ve just copied the parch into the source folder and execute the patch 0001-vmwaretool-vmci-Linux-2.6.37.patch but got this line errors:

    0001-vmwaretool-vmci-Linux-2.6.37.patch:42 : trailing withespace
    static long vmci-ioctl(struct file *file)
    error: patch failde: shared/compat_semaphore.h:28
    error: share/compat_semaphore.h: patch does not apply
    error: vmci_drv.c: No such file or directory

    executing 0001-vmwaretool-vmhgfs-Linux-2.6.37.patch got this error:

    erro: patch failed: super.c:70
    error: super.c: patch does not apply

    My configuration:

    HOST OS :Mac OsX 10.6.8
    GUEST OS : Backtrack 5 kernel 2.6.38
    Vmware 4.0.2 (491587)

    Thanks in advance for any kind of answer .

    Antonio

    Reply

  28. I have worked and tested over this for weeks. Thanks it worked great all but shared folders. I am using workstation 6.5 but I have 7 on another box and I used the tools from it. Version of tools 8.4.5-324285 and the VM is Debian 6.06 Squeeze. And it worked great. I thought I was done when I seen the whitespace warnings lol. But only error was the vmhgfs.

    I couldn’t care less about shared folders as long as I can copy and paste and get the hosts shared clipbrd the host has a network clipbrd I need no shared folders.

    Thanks a million. I am wondering if the patches will fix the older tools in my 6.5 workstation 6.5.0-118166 and will test it soon. I was very surprised when I rebooted and all was well even the vmnet. Great job!

    Reply

  29. I tryed this same patched tools on a Debian Wheezy this morning and it sorta worked. The Wheezy needed something I couldn’t find to do the patch so I used the patched 1 I used above and like I say it sorta worked. But the VMXnet failed and another module besides the folder share. So if anybody wonders it will get you everything on Wheezy too just with a failed compile or 2. But everything works when done least it did here.

    This patch is awesome as the open tools on the Debian repository for Wheezy work but the copy only goes from host to guest so no copy files to host. Text both ways. Just make sure you have all the requires and Squeeze is perfect with this 8 version and the patch.

    Reply

    1. Thank you for the note and comment, jombo. I’m glad to see it still works (sort of..though..)

      As I mentioned at the top of this posting, the patch here I’ve provided was obsolete, and currently, I don’t have the platform. So, unfortunately, I cannot virtually help this directly for now. Sorry about that, and wonder if you’ve communicated with VMWare. Hope they would help your issue.

      Thanks,
      Daniel

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *