Home
Join

23 Replies

  • I've used this in the past and had really good results

    https://www.duplicatefilesfixer.com/Opens a new window

    Pepper graySpice (3) flagReport
    2 found this helpful thumb_up thumb_down
  • I have been using TreeSize for years to find duplicate files.  It will even find duplicate files with different names.

    TreeSize Free | JAM Software (jam-software.com)Opens a new window

    Pepper graySpice (7) flagReport
    1 found this helpful thumb_up thumb_down
  • AllDup works great.  https://www.alldup.de/alldup_help/alldup.phpOpens a new window

    Pepper graySpice (1) flagReport
    Was this post helpful? thumb_up thumb_down
  • I'll be giving those three above a go with the demo (or just trying it if free), I appreciate it!  Most of my google searching had resulted in dubious software at best, straight up scam software at worst.  I don't believe any of those 3 showed up and that in and of itself is why the Spiceworks community is amazing. 

    Pepper graySpice (3) flagReport
    3 found this helpful thumb_up thumb_down
  • Just to give you a couple more options: I've been using Total Commander for a while: https://www.ghisler.com/Opens a new window to find duplicate files by name, hash... and also Find Same Images Ok: https://www.softwareok.com/?Download=Find.Same.Images.OKOpens a new window, It will find similar images based on a percentage  but it requieres quite a bit of 'clicking'.

    Was this post helpful? thumb_up thumb_down
  • DDP wrote:

    I have been using TreeSize for years to find duplicate files.  It will even find duplicate files with different names.

    TreeSize Free | JAM Software (jam-software.com)Opens a new window

    I was thinking of TreeSize too. But JAM Software TreeSize Free​ is probably not supported for the context of the OP. He would probably need JAM Software TreeSize Professional​ or some other edition to run in an environment with AD.

    MisterFerret wrote:

    I'll be fixing the structure and permissions as part of this, but is there anything I should keep in mind while I go?

    Yes. Users may have used different names for copies. Some copies might not be duplicates but different versions of same file instead. There may be valid reasons to have several versions of same file. And then there may be hard and soft links too as far as supported by the file systems involved. So with such links, you won't have that much storage occupied as the links themselves occupy very little storage.

    MisterFerret wrote:

    Anyone took on a project like this before?
    • I did not understand to which project you refer and who assigned which one to you?
    • Do you refer to the file server migration project?
    • Or do you refer to the file server cleanup project?
    • Or do you refer to the establishment of SOP project for the file server?
    • Or do you refer to the setup of controls project?

    If you were assigning all those projects to you and not management, I fear that you'll create more troubles. I understand your concerns. But you'll need management support to end current practices. I guess that you should be able to get management support but will discover that you might not get authorization to do as you preferred, but get some constraints imposed. And this may lead to not all tasks assigned to you but that all users of file server get tasks of doing some cleanup of their no longer needed files. Eventually, the file server might get split into file server and repository service.

    Pepper graySpice (1) flagReport
    Was this post helpful? thumb_up thumb_down
  • This is built in to ccleaner. I was very happy with it used it in production.

    Pepper graySpice (1) flagReport
    Was this post helpful? thumb_up thumb_down
  • You didn't specify what OS the file server is hosted on, if it is Windows Server 2012 R2 and above then the feature File Server Resource Manager will be your friend. It has to be activated through the Server Manager > Manage > Add Roles and Features interface, and is located under File and Storage Services > File and iSCSI Services. Click on the "add features" button when the additional window pops up. Activating the feature might require a restart of the server, so plan accordingly to minimize impact to the environment.

    You might want to consider implementing it on the new file server as well if it is Windows Server based, since it allows you to filter file types and assign quotas to folders, which will make the management of content in shares much easier and adds another layer of defense to your server if implemented properly. Adding a SMTP server to the configuration allows you to send explanatory mails when users try to copy unauthorized files to the protected share, near their quota limit and it can send scheduled reports to you regarding usage as well.

    You can assign a file policy to the root folder where the share(s) reside on (monitoring only, no need to enforce it) and once it has had a chance to run you will be able to pull reports for large files, duplicate files, files with a specific extension, files by modify date (for unused files) and so forth.

    Pepper graySpice (5) flagReport
    4 found this helpful thumb_up thumb_down
  • Or just enable data de-duplication and don't worry about it.  Doesn't matter how many copies people make then the server will only store 1 copy

    Its much easier than trying to get a heard of cats to manage their files properly

    Pepper graySpice (11) flagReport
    7 found this helpful thumb_up thumb_down
  • I've used DigitalVolcano's Duplicate Cleaner for several years now.  Does everything I need it to do.

    https://www.digitalvolcano.co.uk/duplicatecleaner.htmlOpens a new window 

    Cheers.

    Was this post helpful? thumb_up thumb_down
  • I like dupeguru as it can search network drives, so I can use from my work computer rather than directly installing and using it on the server.

    Was this post helpful? thumb_up thumb_down
  • However, I use it more for finding duplicate folders, word, excel, pdf files rather than images.

    Was this post helpful? thumb_up thumb_down
  • Little more info, this is a VM (Hyper-V if it matters) running Server 2019 that I had done an in place upgrade from 2012 about 2 years ago.  I'll be moving things to a new Hyper-V host and building a new VM as the file server rather than an export/import.  This company has been just browsing the network for files rather than using mapped drives, so I'll be fixing that particular headache, setting it up right this time is my goal.  As a plus, I've been given free reign to accomplish this however I see fit, apparently the last 3 IT folks tried and then gave up after they saw this mess.  I don't intend to do that.

    FSRM seems like a great idea to help keep it from become what it is now again, so that will absolutely be setup on the new server. Part of the removing duplicate files is to trim down the extraneous data before I move it both for the time, space, and sanity savings.  I do have some cooperation from other departments to slim down and organize a bit, but in the end it will probably be a lot of unilateral decisions on my part.
    I gave the department heads a worksheet to help me (and them really) with a warning that if they don't help then don't get to complain if they don't like how I handle it.
    Pepper graySpice (3) flagReport
    Was this post helpful? thumb_up thumb_down
  • I have used Beyond Compare for this exact project. It was well worth the cost of a license.

    Pepper graySpice (2) flagReport
    2 found this helpful thumb_up thumb_down
  • MisterFerret wrote:

    This company has been just browsing the network for files rather than using mapped drives, so I'll be fixing that particular headache, setting it up right this time is my goal.

    • Is drive mapping still best practice or are shortcuts to shared preferred now?
    Pepper graySpice (1) flagReport
    1 found this helpful thumb_up thumb_down
  • I was going to say...just enable data de-dupe and setup proper shares. If you take large sections that "shouldn't be that way" and move them elsewhere then make proper drive mapping to have them organize things in a better way on their own the filesystem deduping will handle the rest of it. I wouldn't try to mess with a clients folder structures. They're just going to either complain or screw it up again afterwards.

    Pepper graySpice (5) flagReport
    1 found this helpful thumb_up thumb_down
  • itaintbroke wrote:

    I have used Beyond Compare for this exact project. It was well worth the cost of a license.

    THIS ^^^^^^^^

    Pepper graySpice (1) flagReport
    Was this post helpful? thumb_up thumb_down
  • I would like to point out that MS File service does have Deduplicate as one of the options that needs to be turned on. When I managed a system that had 4TB to start it was one of the first things I did. After 8 years we were up to 6TB of data, but the DeDup had saved us about 5 TB of space. It's at least a starting point while you check out the other options and get things in order.

    https://learn.microsoft.com/en-us/windows-server/storage/data-deduplication/install-enableOpens a new window

    Pepper graySpice (2) flagReport
    1 found this helpful thumb_up thumb_down
  • I second kevinweller​  This is included in CCleaner for free.

    A nifty free tool, not to find dupes but to get an overview of usage and see where to focus your efforts, is WinDirStat  

    Was this post helpful? thumb_up thumb_down
  • I'd not heard of some of the more generalized solutions mentioned above which is a pleasant learning experience, but for what it's worth I've had to de-duplicate a big image cache before and used Adobe Lightroom and this Lightroom plugin (http://www.bungenstock.de/teekesselchen/Opens a new window) to detect duplicate images. When it was done, I then exported the images into a new file structure. Might be useful if you're not getting enough image de-duplication from other solutions.

    Pepper graySpice (1) flagReport
    Was this post helpful? thumb_up thumb_down
  • I'm a little surprised that I haven't seen probably the 2 best reminders in a situation like this:  1.  Create multiple backups before you start! :)   and  2. Ensure that mgmnt knows what you are planning to do, and discuss with them and anyone that has been there a long time about the history, structure, reasonings, files etc.  It's amazing what you can learn and this also may help you with the screaming ....  and lastly  Communicate Communicate Communicate to them what is going on, what the plan is and ensure you don't create a panic.  Just my 3cents...

    Pepper graySpice (1) flagReport
    Was this post helpful? thumb_up thumb_down
  • thelanranger wrote:

    I was going to say...just enable data de-dupe and setup proper shares. If you take large sections that "shouldn't be that way" and move them elsewhere then make proper drive mapping to have them organize things in a better way on their own the filesystem deduping will handle the rest of it. I wouldn't try to mess with a clients folder structures. They're just going to either complain or screw it up again afterwards.

    Been there done that. We had a file server with user folders as well as a well-organized central repo for departments such as sales, customer service, etc. where people have a lot of shared files. They just ended up having their own duplicates. After we deduped it, it didn't matter much, but it was still frustrating that they refused to do things properly.

    Pepper graySpice (2) flagReport
    Was this post helpful? thumb_up thumb_down
  • Data deduplication on a file server is an a huge saving grace.. 

    For example


    Was this post helpful? thumb_up thumb_down

Read these next...

  • Simple command to monitor Windows 10 temperature?

    Simple command to monitor Windows 10 temperature?

    Hardware

    I feel like this has probably been address before, although I was wondering if someone is aware of a simple command I can run to report the internal temperature of a Windows 10 PC?I think all computers monitor the temperature, although I've only found thi...

  • Remote access to DVR?

    Remote access to DVR?

    Security

    Hi!I have an older Hikvision DVR that I need to provide remote access to. The users would be mainly accessing it from their smartphones. I tested their software, iVMS, by assigning one of my public IP's to the DVR and it worked fine. However the issue is ...

  • Snap! -- Survival Kits, Forest Bubble on Mars, AI Movie Plots, Leprosy & Livers

    Snap! -- Survival Kits, Forest Bubble on Mars, AI Movie Plots, Leprosy & Livers

    Spiceworks Originals

    Your daily dose of tech news, in brief. Welcome to the Snap! Flashback: Back on December 6, 1907, Mathematical Logician J. Barkley Rosser Born (Read more HERE.) Bonus Flashback: Back on December 6, 1998, International Space Station assemb...

  • Spark! Pro Series - 6 December 2022

    Spark! Pro Series - 6 December 2022

    Spiceworks Originals

    Today in History: 6 December 1240 – Mongols led by Batu Khan occupy and destroy Kyiv after an 8 day siege; out of 50,000 people in the city only 2,000 survive 1849 – Harriet Tubman escapes from slavery in Maryl...

  • The most boring but interesting Phishing Attempt I've seen

    The most boring but interesting Phishing Attempt I've seen

    Security

    Hello There,We've recently had a phishy email come through to one of our employees with an attachment to something work related. But here's the interesting part: The email was spoofed. When checked, the address was that of our own domain, however the emai...