Home
Join

24 Replies

  • If you had 1G+ Internet up and down, not 250M/250M, you could probably wrangle a way to get the data physically on-premises faster. But, you have a 250M/250M. 

    If the data needs to be physically onsite, it'll be fastest to just ship drives like you currently are (even compared to WAN acceleration, among other methods). 

    Pepper graySpice (5) flagReport
    1 found this helpful thumb_up thumb_down
  • If the data doesn't need to be physically onsite, such as cloud-adjacent sharing, then you have a whole bunch of options to share and side-load options through Internet Data Centers (IDC... depending on where both parties are physically locating their data) or tenant-to-tenant sharing through hyperscaler technologies.

    Was this post helpful? thumb_up thumb_down
  • even with a 10G/10G internet overnighting a disk is faster. Just locally copying will take a few hours.

    Pepper graySpice (5) flagReport
    1 found this helpful thumb_up thumb_down
  • AaronJBerger wrote:

    I assisting a company that is looking for cloud storage for large video files so they can upload the videos at one site and download them at another.

    The current solution is manually shipping usb hard drives with the video files which are around 5TB or larger to a different state.

    Is there a cloud storage provider that can handle this size video files, that is also relatively easy for the users at both end to use? Currently the link at each site is 250/250 but we can upgrade this to a faster link as needed.

    Obviously the main issues likes with the size of the files and the data transfer speeds.

    But you need to decide if it is for storage or transfers as large storage would have a significant cost involve.

    We use AWS FXS with multi-AZ for global transfer of files.

    Pepper graySpice (2) flagReport
    2 found this helpful thumb_up thumb_down
  • There are lots of cloud storage providers that can handle that. Yes

    • Wasabi
    • Back Blaze
    • MS Azure
    • AWS
    • Etc
    • Etc

    But as noted upload download speeds will be dependent on your ISP connections

    Pepper graySpice (1) flagReport
    2 found this helpful thumb_up thumb_down
  • I'm not sure what you're doing exactly but it seems to me that if you're looking for 'increased efficiency' it would likely be better to setup some kind of VPN between the sites and transfer it directly. Whether you're sending it to a cloud service or via hard drive you're transferring the data multiple times. If you could establish some mechanism of permanent storage at each location that was able to replicate itself via VPN or the like then you would immediately "double the speed" as you would not have to copy the data two times (either from system to disk then back to system or from system to cloud then back to system).

    Pepper graySpice (2) flagReport
    Was this post helpful? thumb_up thumb_down
  • How often does such a 5TB transfer need to happen?  Daily? Weekly?

    5TB is a lot of data to move around on a regular basis, especially over distance.  Even at 1Gbps, it would take 11 hours (with no latency issues - latency increases with distance, eventually becoming the limiting factor, not bandwidth). So that's at least 11 hours to upload, and then another 11 hours to download, or a 22 hour minimum transfer time (with upgraded Internet).


    Pepper graySpice (1) flagReport
    1 found this helpful thumb_up thumb_down
  • This may be a stupid suggestion BUT if you can upgrade the internet connection, could you set up an SFTP server and save the files to that and download them at the other end? or even upload them to the destination from the source?

    I needed something similar and i just used to FTP the file to the destination which removed the need for cloud and other hardware.

    Pepper graySpice (3) flagReport
    Was this post helpful? thumb_up thumb_down
  • Have you thought about building a site to site VPN or FTP site with creds?

    With a video file that big, it would be considerably faster to send a hard drive still.

    Remember this would be one way, it would take just as long to download at the other site.

    Pepper graySpice (2) flagReport
    Was this post helpful? thumb_up thumb_down
  • The issue you are running into is the size of the data you are moving. 5TB takes a long time no matter what your connection is.

    What I would do, if I was in your situation, is create a file sharing system (Windows file share server, NAS, etc.) and house the data locally. Upgrade your internet speed to as fast as you can get. At the remote sites you would then use your local storage as a cloud service that they can either download the data or stream it. If they need to get the data faster then upgrade there connection as well. If the data always needs to be at the remote site as well I would set up a syncing system that will allow it to sync at off peak times automatically.

    This solution will reduce the transfer time since it will only need to be downloaded at the remote sites (essentially halving the data transfer time). Using a third party cloud based solution will be expensive and slow since you would have to upload the file first then they could download it once it was done. You also do not have control over the data speeds for some sites. If you have a 1gig internet speed you may not be able to download or upload at a gig from the cloud storage place. If you house your data in-house and have a gig internet then you know you can transfer data at a gig.

    Was this post helpful? thumb_up thumb_down
  • If you really do want to transfer that much data regularly, you might want to consider a WAN accelerator - not something I have experience with, but I'm aware they exist. For instance, Veeam (Enterprise and higher), has a WAN accelerator for speeding up moving data to another site.

    A WAN accelerator helps dedupe and compress data to allow it to transfer in less time.

    As mentioned many times above, you should strongly consider direct transfer to a self-hosted SFTP site or something.  Cut out the cloud middle-man and you can cut your transfer time in half (ish).

    Did you want to use a cloud intermediary because of unstable connections, and the ability to resume a transfer?

    Was this post helpful? thumb_up thumb_down
  • phildrew wrote:

    If you really do want to transfer that much data regularly, you might want to consider a WAN accelerator - not something I have experience with, but I'm aware they exist. For instance, Veeam (Enterprise and higher), has a WAN accelerator for speeding up moving data to another site.

    A WAN accelerator helps dedupe and compress data to allow it to transfer in less time.

    As mentioned many times above, you should strongly consider direct transfer to a self-hosted SFTP site or something.  Cut out the cloud middle-man and you can cut your transfer time in half (ish).

    Did you want to use a cloud intermediary because of unstable connections, and the ability to resume a transfer?

    WAN accelerators require a matching device on both ends of the WAN Link which is not possible for cloud storage solutions.  

    It could work for a site-to-site VPN setup between physical sites

    Pepper graySpice (1) flagReport
    1 found this helpful thumb_up thumb_down
  • Crazy Rabbit wrote:

    The issue you are running into is the size of the data you are moving. 5TB takes a long time no matter what your connection is.

    What I would do, if I was in your situation, is create a file sharing system (Windows file share server, NAS, etc.) and house the data locally. Upgrade your internet speed to as fast as you can get. At the remote sites you would then use your local storage as a cloud service that they can either download the data or stream it. If they need to get the data faster then upgrade there connection as well. If the data always needs to be at the remote site as well I would set up a syncing system that will allow it to sync at off peak times automatically.

    This solution will reduce the transfer time since it will only need to be downloaded at the remote sites (essentially halving the data transfer time). Using a third party cloud based solution will be expensive and slow since you would have to upload the file first then they could download it once it was done. You also do not have control over the data speeds for some sites. If you have a 1gig internet speed you may not be able to download or upload at a gig from the cloud storage place. If you house your data in-house and have a gig internet then you know you can transfer data at a gig.

    I'm leaning towards site-to-site VPN with NAS/Server at each end. Minimum of 1G at each end, though it looks like 10G is the only way to get the transfer down under 20hrs.

    How stable would this be with a 5TB copy over 1G? Would we require manual resuming of transfers due to interruptions/timeouts etc during the copy?

    Was this post helpful? thumb_up thumb_down
  • You can look at something like this siteOpens a new window to find a speed that is acceptable to you guys.

    If you use something like a sync or SFTP, those will have protocols that make sure there are no errors in the transfer. You can find other protocols that can assure a complete transfer. Remember that your internet speed will also be shared with others that are using it and also the ISP can fluctuate. Here is what 10GB would look like if it were to maintain full speed the entire time. 1Gbps would take approx 8 hours vs 10gbps is closer to 40 min. It's all about cost.


    Was this post helpful? thumb_up thumb_down
  • Under the circumstances, I'm seeing two basic limitations here:

    * File size limits

    * Speed of transfer limits

    Also, page blob (not block blob) limits in Azure are 8 tebibytes (about 8,800 gigabytes).   

    If you don't care about speed of transfer, then just pick up a file-splitter software, chop your giganto file into 1Tb increments, and re-join them at their destination.

    If speed of transfer does matter, buy a few 10Tb drives in USB3 cradles, and use them to mail the files to the destination overnight.

    Hope this helps - 

      T

    1 found this helpful thumb_up thumb_down
  • phildrew​ molan​ Most WAN accelerators do not work well with compressed files, pictures and vids. It is like trying to zip these type of files and after hours you save like only 1%.

    Gorfmaster1​ Crazy Rabbit​ In most cases, we would have subscription to "consumer" Internet for such usage as some "consumer" broadband offers 1Gbps or even 2Gbps as a 2nd line in the office (doubles up as Guest networks). Then we only solve our "uploading" part while the other side handles their own "downloading issues".

    1 found this helpful thumb_up thumb_down
  • Is it just a single end point?  Why stage it "in the cloud" instead of just transferring it?

    As pointed out above, it's going to take a while, and adding a point in the middle just makes it take at least twice as long.

    If the data doesn't change often, or completely every time, then it's possible a transfer system that works on the block level like zfs send rather than a file level might shave some time off.

    Was this post helpful? thumb_up thumb_down
  • How stable would this be with a 5TB copy over 1G? Would we require manual resuming of transfers due to interruptions/timeouts etc during the copy?

    Any transfer has a chance to crash mid transfer, but in this case I would find/use a sync/transfer/backup software that allows for this and uses an auto resume feature.

    I'm leaning towards site-to-site VPN with NAS/Server at each end. Minimum of 1G at each end, though it looks like 10G is the only way to get the transfer down under 20hrs.

    A site to site VPN is great for security but it will incorporate overhead on the connection making your transfer take longer. Even a 5% overhead can equal an extra 30-60 minutes to transfer a file that size. If you go with a NAS solution you could use a Synology​ NAS and use there DDNS connectivity option. They also have SYNC software to replicate NASes. I believe QNAP Systems, Inc​ also does this but I do not have experience with them specifically. As Gorfmaster1​ has mentioned, a 1gig connection should take about 8 hours to complete for the 5TB transfer on a fibre connection. I would expect it to take closer to 12+ hours since sustaining a full 1 gig at all times is not always going to happen. If you have the option to run a 10G fiber for internet service then yes, it would definitely help a ton in the transfer speed. You will also need to make sure you have those speeds internally and find a NAS/Server with drives that can match that speed at least. Having a 10G connection but only 6G drives or connecting to a gig switch will bottleneck you. A 10G connection will need SFP+ minimum and NVMe drives. It will also need to match on both sites.

    If you go with a NAS solution that replicates to an off site NAS I would look at the data being transmitted. If it is one file that is 5TB then there is nothing you can do about the transfer time. If it is many files that are zipped/compressed into one file then I would not compress them with this solution. The SYNC time can be reduced by using the option to SYNC only changed files. Once a master image is made on the second NAS then when changes are done on the master NAS it will check the differences and only copy the changed files. This will significantly reduce the transfer times.

    Pepper graySpice (1) flagReport
    Was this post helpful? thumb_up thumb_down
  • You need 1G internet speed up and down or keep sending disk files overnight LOL what kind of storage you are required to have? do you need 200TB 500tb At the upload and download requirements sounds like you need a lot storage?

    how many users needs access to the files? maybe use OneDrive and sync files with on-premise appliance like Synology?

    Was this post helpful? thumb_up thumb_down
  • If you are just editing or viewing the file at one of the sites it would likely be easier and faster to leave it at one site and set up a way to remote to the location where the file exists and use it that way.  A VPN connection between the two sites and remote desktop from one side to the other.  You can view, edit, whatever.  The file only stays on one side but is usable by both.

    Was this post helpful? thumb_up thumb_down
  • Another issue with transferring these files over the internet will be data caps. Not all providers have them on business connections but some do. Depending how often you're transferring these 5TB files you could hit a data cap very quickly. 

    Was this post helpful? thumb_up thumb_down
  • molan wrote:

    phildrew wrote:

    If you really do want to transfer that much data regularly, you might want to consider a WAN accelerator - not something I have experience with, but I'm aware they exist. For instance, Veeam (Enterprise and higher), has a WAN accelerator for speeding up moving data to another site.

    A WAN accelerator helps dedupe and compress data to allow it to transfer in less time.

    As mentioned many times above, you should strongly consider direct transfer to a self-hosted SFTP site or something.  Cut out the cloud middle-man and you can cut your transfer time in half (ish).

    Did you want to use a cloud intermediary because of unstable connections, and the ability to resume a transfer?

    WAN accelerators require a matching device on both ends of the WAN Link which is not possible for cloud storage solutions.  

    It could work for a site-to-site VPN setup between physical sites

    Yes, fully agreed.  OP says that the client wants to " upload the videos at one site and download them at another" - a perfect use case for WAN accelerators.


    adrian_ych wrote:

    phildrew​ molan​ Most WAN accelerators do not work well with compressed files, pictures and vids. It is like trying to zip these type of files and after hours you save like only 1%.

    Yes, a WAN accelerator might only compress a video by 1%, but at the file sizes being discussed here, 1% of 5TB is still 50 GB, which is nothing to sneeze at if you are trying to reduce transfer times - that could easily be 30-60 minutes of saved time. Multiply that by how frequently this needs to happen (which we still don't know), and it might be a desirable option.

    Keep in mind that transfer times for most of these file transfer calculators being quoted go out the window when you have WAN latency in the picture (here's one that does take latency into account: https://wintelguy.com/wanperf.plOpens a new window). For instance - a 250 Mbps connection has TCP overhead - you can only get 237 Mbps of your data down the wire. However, if you add 37 ms of latency (74 ms round-trip) to the connection that bandwidth becomes 158 Mbps (63% of the theoretical bandwidth).  So, while a calculator can say the file can transfer in 48 hours, the reality could be that it will take much closer to 72 hours.

    Was this post helpful? thumb_up thumb_down
  • phildrew​, when you are talking latency, the number you get from a test is the round trip number. It is not possible to get latency only one way since it tests the time it takes for packets to get to the destination and back. Your 37 ms is the round trip number, it would not double to 74 ms as you state.

    You are correct though that most tools do the raw math without taking into account latency and fluctuating bandwidth available (among other things). A 250 mb/s may only get about half of that as constant bandwidth.
    Pepper graySpice (1) flagReport
    Was this post helpful? thumb_up thumb_down
  • Crazy Rabbit wrote:

    phildrew​, when you are talking latency, the number you get from a test is the round trip number. It is not possible to get latency only one way since it tests the time it takes for packets to get to the destination and back. Your 37 ms is the round trip number, it would not double to 74 ms as you state.

    You are correct though that most tools do the raw math without taking into account latency and fluctuating bandwidth available (among other things). A 250 mb/s may only get about half of that as constant bandwidth.

    My bad - ping latency is the round-trip time! Thanks for the correction.

    Was this post helpful? thumb_up thumb_down

Read these next...

  • Simple command to monitor Windows 10 temperature?

    Simple command to monitor Windows 10 temperature?

    Hardware

    I feel like this has probably been address before, although I was wondering if someone is aware of a simple command I can run to report the internal temperature of a Windows 10 PC?I think all computers monitor the temperature, although I've only found thi...

  • Remote access to DVR?

    Remote access to DVR?

    Security

    Hi!I have an older Hikvision DVR that I need to provide remote access to. The users would be mainly accessing it from their smartphones. I tested their software, iVMS, by assigning one of my public IP's to the DVR and it worked fine. However the issue is ...

  • Snap! -- Survival Kits, Forest Bubble on Mars, AI Movie Plots, Leprosy & Livers

    Snap! -- Survival Kits, Forest Bubble on Mars, AI Movie Plots, Leprosy & Livers

    Spiceworks Originals

    Your daily dose of tech news, in brief. Welcome to the Snap! Flashback: Back on December 6, 1907, Mathematical Logician J. Barkley Rosser Born (Read more HERE.) Bonus Flashback: Back on December 6, 1998, International Space Station assemb...

  • Spark! Pro Series - 6 December 2022

    Spark! Pro Series - 6 December 2022

    Spiceworks Originals

    Today in History: 6 December 1240 – Mongols led by Batu Khan occupy and destroy Kyiv after an 8 day siege; out of 50,000 people in the city only 2,000 survive 1849 – Harriet Tubman escapes from slavery in Maryl...

  • The most boring but interesting Phishing Attempt I've seen

    The most boring but interesting Phishing Attempt I've seen

    Security

    Hello There,We've recently had a phishy email come through to one of our employees with an attachment to something work related. But here's the interesting part: The email was spoofed. When checked, the address was that of our own domain, however the emai...