Add support for Zip64 for archives > 4GB

[SteveL-MSFT]

The .Net ZipFile class currently has a limitation of 2GB sizes, this limits usefulness on VM scenarios where VHDs are routinly larger than 2 GB

[iSazonov]

@SteveL-MSFT Do you have in mind that we need to use this project SharpZipLib?
Currently corefx support Zip64 only for unzip dotnet/corefx#11264

[SteveL-MSFT]

@iSazonov since SharpZipLib appears to be available under MIT License, I see no reason to not use that lib.

[iSazonov]

@SteveL-MSFT Thanks!
😕 they haven't .Net Core support icsharpcode/SharpZipLib#106
It is seems we are still blocked in both cases.

[kilasuit]

This is a pain point compress-archive, which I was looking to make use of for Zipping up Process dumps (some 7Gb +) but it looks like I'd be better wrapping around 7zip to do this instead of relying on a more native command.

[SteveL-MSFT]

Looks like SharpZipLib now has NetStandard1.3 support so that works for us

[kilasuit]

@SteveL-MSFT is there any update on this at all?

[SteveL-MSFT]

@kilasuit no update right now, it's in the backlog, but not a high priority right now given other work that's happening

[JohnLBevan]

@kilasuit FYI: There is a preexisting 7zip wrapper available as a temporary workaround / to save you rolling your own. https://github.com/thoemmi/7Zip4Powershell

[rjmholt]

I can't find a corefx issue for this? Is this something we should open an issue on them for?

Some possibly related issues:

• https://github.com/dotnet/corefx/issues/11668
• https://github.com/dotnet/corefx/issues/24083
• https://github.com/dotnet/corefx/issues/11669

[SteveL-MSFT]

@rjmholt I think rewriting this module in C# and using SharpZipLib might be better than waiting for corefx, plus it would work for older versions of PowerShell

[rjmholt]

@SteveL-MSFT agreed, but just want corefx to be conscious of the ask

[SteveL-MSFT]

@rjmholt no harm in opening an issue for corefx. I suspect it's not high on their priority since existing OSS libraries support zip better than the built in support unless ASPNETCORE has a need for it.

[adam-birds-test]

Any update on where this is at?

[SteveL-MSFT]

@adam-birds-hwt unfortunately, this is currently not in our priority queue. I'd like to see about addressing this in PS7.1 timeframe as it's not a small work item.

[boscorelly]

Hi, same issue on windows Server 2016/2019

[iSazonov]

This comment says that .Net support Zip64.
After the fix I'd expect that the module works in PowerShell 7.0.

@ericstj or @ianhays could tell us (please) where a problem is in this module and that API we could use to re-write the module on C#.

[ericstj]

The implementation in .NETCore still loads into memory when being opened for update. When it does this it uses a managed array which limits the size to 2GB.

If you open only the archive for Read or Write only, then entries can be created that exceed 2GB. Let us know more about the scenario and we can provide a recommendation. /cc @carlossanlop

[rjmholt]

@adityapatwardhan does the archive module perform archive update/modification functions?

[iSazonov]

@rjmholt Yes, it is does.

If MSFT approves I'd port the module to C# (in PowerShell/Module repo) and address the issue too.

[rjmholt]

@SteveL-MSFT and @adityapatwardhan are probably the right people to make that call

[ChrisLynchHPE]

Any status update on this, and #96? It would be great to not have to build some custom Advanced Function to handle archives, or worse yet to use an external binary (like 7z.exe).

[michalzobec]

we waiting for creating of ZIP archives almost for 4 years ... it is sad. :(

[iSazonov]

In .Net 5.0 I still see issues in the API. We need investigate in depth all issues in the API and ask .Net team to fix its in .Net 6.0 milestone.

[Bearstonk]

I'd like to emphasize that the file size limitation of compress archive has a renewed sense of urgency now that applications like 7zip are being banned from government and DOD systems and there's no CLI replacement really available outside of third party. It would be preferable for a native application to be implemented. Or simply incorporate gzip or tar since they're fully featured already.

[kilasuit]

@Bearstonk - I believe that this is semi-planned in the PowerShell Roadmap post by @SteveL-MSFT due to the Exec Order on CyberSecurity.

How soon this happens will be a different matter that only Steve can properly comment on.

[ayousuf23]

Closing this issue as zip64 is now supported.

[ayousuf23]

Please download the latest preview for zip64 support.

[JairoCaychoFinTech]

@ayousuf23 Could you give me more context, what does it mean to download latest preview for zip64?
I found this link...is it correct?
http://www.artpol-software.com/Download.aspx
Since zip64 is now supported, just by using Compress-Archive I will be able to compress without limits?

[JairoCaychoFinTech]

For all the ones that ended up here I found a solution without upgrading to the just released (4 days ago) preview version .NET Framework 4.8.1.
You can download the latest (as today 13 of August 2022 is 6.0.4) .NET open-source developer platform from:
https://dotnet.microsoft.com/en-us/download/dotnet

Then in your powershell you go to the folder where is your downloaded script and execute the following command:
.\dotnet-install.ps1 -Channel Current
If you want to check that everything is ok, just run:
dotnet --list-sdks

And you should get something like this:

Documentation:
https://docs.microsoft.com/en-us/dotnet/core/install/windows?tabs=net60
https://docs.microsoft.com/en-us/dotnet/core/install/how-to-detect-installed-versions?pivots=os-windows

[kilasuit]

@JairoCaychoFinTech what @ayousuf23 meant was with the latest preview of this module you now get zip64 support, which the release of it on the PowerShellGallery does show install instructions

[JairoCaychoFinTech]

@JairoCaychoFinTech what @ayousuf23 meant was with the latest preview of this module you now get zip64 support, which the release of it on the PowerShellGallery does show install instructions

Thanks for the clarification. I'm just starting with scripting so I need a lot of explanations!

[santisq]

For anyone still facing this issue and wanting to use a PowerShell / .NET native solution, I created this function. It is fully compatible with PowerShell 5.1 and PowerShell Core and there shouldn't be a file size limitation because it doesn't use a MemoryStream, instead it copies from FileStream to IO.Compression.WrappedStream. I haven't encountered a size limitation so far and is a lot faster than the Core version of Compress-Archive (haven't tested the preview version of the cmdlet yet).

[NRL-LEastham]

For anyone still facing this issue and wanting to use a PowerShell / .NET native solution, I created this function. It is fully compatible with PowerShell 5.1 and PowerShell Core and there shouldn't be a file size limitation because it doesn't use a MemoryStream, instead it copies from FileStream to IO.Compression.WrappedStream. I haven't encountered a size limitation so far and is a lot faster than the Core version of Compress-Archive (haven't tested the preview version of the cmdlet yet).

I've just tried this on a 3.7GB file and it cuts the original file short as the CopyTo has the same Stream was too long issue. The archive completes but only with 2GB of the original file.

[santisq]

I've just tried this on a 3.7GB file and it cuts the original file short as the CopyTo has the same Stream was too long issue. The archive completes but only with 2GB of the original file.

@NRL-LEastham I'm honestly unable to replicate. Have tried to compress folders with size up to 20Gb without any problems. Are you using the latest module version 1.0.4? If you want to please raise an Issue in the repo to troubleshoot.

[NRL-LEastham]

I've just tried this on a 3.7GB file and it cuts the original file short as the CopyTo has the same Stream was too long issue. The archive completes but only with 2GB of the original file.

@NRL-LEastham I'm honestly unable to replicate. Have tried to compress folders with size up to 20Gb without any problems. Are you using the latest module version 1.0.4? If you want to please raise an Issue in the repo to troubleshoot.

Sorry was against the clock on Friday. I ended up just using a function call that adds the switches to the command line of the installed 7z CLI. Was on the latest version and my 3.7gb T-SQL file just generated the stream error on a CopyTo command instead. Was a 2 vCore 8GB VM and although the zip was created the original file size was reported as 2.7GB instead so must have truncated. If I get time I'll test on my local machine and if still an issue will raise it on your repo.