gh-102950: Implement PEP 706 – Filter for tarfile.extractall #102953

encukou · 2023-03-23T14:21:11Z

See PEP-706 for details.

I might have overengineered the tests a bit, moving boilerplate to a set of helpers. Hopefully the tests themselves are clear enough to make it worth it.

Issue: Implement PEP 706 – Filter for tarfile.extractall #102950

ethanfurman

Here's what I have so far; still working on tarfile.py.

Doc/library/shutil.rst

Lib/shutil.py

Doc/library/tarfile.rst

Lib/tarfile.py

Thanks to Ethan for spotting these

encukou · 2023-03-31T16:18:07Z

@gpshead, did you want to take a look?

encukou · 2023-04-03T15:48:58Z

I have made the requested changes; please review again

hroncok

I reviewed the changes in Doc/.

hroncok · 2023-04-04T15:04:21Z

Doc/library/tarfile.rst

+
+.. versionadded:: 3.12
+
+The *tar* format is designed to capture all details of a UNIX-like ecosystem,


Suggested change

The *tar* format is designed to capture all details of a UNIX-like ecosystem,

The *tar* format is designed to capture all details of a UNIX-like filesystem,

I also tend to say POSIX rather than UNIX at this point.

POSIX is more precise, but this intentionally hand-wavy terminology: I think UNIX-like is more understandable to more people than POSIX-like.

Doc/library/tarfile.rst

Doc/library/shutil.rst

Doc/library/tarfile.rst

Lib/shutil.py

encukou · 2023-04-17T09:51:38Z

If there are no objections, I plan to merge this around Wednesday. Please let me know if you're working on a review.

hroncok · 2023-04-17T10:15:48Z

Lib/tarfile.py

+        except TarError as e:
+            if self.errorlevel > 0:
+                raise


After reading the docs, I thought only OSErrors will be raised with errorlevel == 1. This seems to raise even TarErrors with errorlevel 1 or 2. What do I miss?

Oh dear. What a rabbit hole. Thanks for raising it, and for a very helpful private discussion.
This warrants an addition to the PEP: https://discuss.python.org/t/23149/26
Please voice your concerns there!

I've updated this PR with my preferred solution, so you can see the areas that need to change and so I get a CI check. It doesn't mean the discussion is over, of course.

hroncok · 2023-04-17T10:22:09Z

If there are no objections, I plan to merge this around Wednesday. Please let me know if you're working on a review.

I still plan to go trough all of this. I think the last remaining part is ReplaceTests but I've already looked over that part before this PR was opened. Feel free to merge if you don't hear from me by Wednesday.

Lib/tarfile.py

Doc/library/tarfile.rst

Lib/test/test_tarfile.py

encukou · 2023-04-20T10:06:12Z

aarch64 Debian clang fails test_tarfile with ValueError: overflow in number field. Will investigate.
wasm32-wasi fails symlink-related tests. Not surprising. Will investigate.
ArchLinux TraceRefs fails in freezing modules, presumably unrelated
Fedora LTO+PGO fails in test_tools, presumably unrelated

encukou · 2023-04-20T12:00:46Z

On WASI, all symlink tests are skipped for now. The mechanism to skip them is a bit of a shortcut that doesn't work here, so on WASI I'll skip the tarfile symlink tests explicitly.

encukou · 2023-04-20T13:21:40Z

And aarch64 Debian clang showed an issue in tests: I assumed addfile will work on a regular file, but with the old USTAR_FORMAT, it fails if the owner user ID is too high. Will fix the test.

encukou · 2023-04-24T08:56:47Z

The remaining buildbot failures are unrelated and happened on main builds as well. One was fixed in main (#103561) and the other is still there.
Crossing fingers, merging.

kulikjak · 2023-04-25T09:34:04Z

Hi, with this integrated, I see one new test failure on Solaris:

FAIL: test_modes (test.test_tarfile.TestExtractionFilters.test_modes)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "...../cpython-main/Lib/test/test_tarfile.py", line 3641, in test_modes
    self.expect_file('all_bits', mode='?rwsrwsrwt')
  File "...../cpython-main/Lib/test/test_tarfile.py", line 3406, in expect_file
    self.assertEqual(got, mode)
AssertionError: '?rwsrwsrwx' != '?rwsrwsrwt'
- ?rwsrwsrwx
?          ^
+ ?rwsrwsrwt
?          ^

The issue seems to be the fact that setting sticky bit on files as regular user doesn't do anything on Solaris, and from my limited testing, the same seems to be true on some other systems as well (I tried FreeBSD and chmod +t on file as a regular user also did nothing).

encukou · 2023-04-25T09:40:12Z

Thanks for the report! The tests are intentionally pretty strict. I'll relax this one. (Edit: see #103831)

…ython#102953) Backport of af53046

…H-102953) (GH-103832) See [Backporting & Forward Compatibility in PEP 706](https://peps.python.org/pep-0706/#backporting-forward-compatibility). - Backport b52ad18 - Backport c8c3956 - Remove the DeprecationWarning - Adjust docs - Remove new `__all__` entries

…H-102953) (GH-104128) - Backport b52ad18 - Backport c8c3956 - Remove the DeprecationWarning - Adjust docs - Remove new `__all__` entries Co-authored-by: Petr Viktorin <[email protected]>

…H-102953) (#104382) Backport of c8c3956

encukou · 2023-05-16T13:24:47Z

A test_gdb timeout looks unrelated. And the equivalent PR buildbot was green before merging.

…H-102953) (#104548) Backport of c8c3956

* Post 3.8.16 * [3.8] Update copyright years to 2023. (pythongh-100852) * [3.8] Update copyright years to 2023. (pythongh-100848). (cherry picked from commit 11f9932) Co-authored-by: Benjamin Peterson <[email protected]> * Update additional copyright years to 2023. Co-authored-by: Ned Deily <[email protected]> * [3.8] Update copyright year in README (pythonGH-100863) (pythonGH-100867) (cherry picked from commit 30a6cc4) Co-authored-by: Ned Deily <[email protected]> Co-authored-by: HARSHA VARDHAN <[email protected]> * [3.8] Correct CVE-2020-10735 documentation (pythonGH-100306) (python#100698) (cherry picked from commit 1cf3d78) (cherry picked from commit 88fe8d7) Co-authored-by: Jeremy Paige <[email protected]> Co-authored-by: Gregory P. Smith <[email protected]> * [3.8] Bump Azure Pipelines to ubuntu-22.04 (pythonGH-101089) (python#101215) (cherry picked from commit c22a55c) Co-authored-by: Hugo van Kemenade <[email protected]> * [3.8] pythongh-100180: Update Windows installer to OpenSSL 1.1.1s (pythonGH-100903) (python#101258) * pythongh-101422: (docs) TarFile default errorlevel argument is 1, not 0 (pythonGH-101424) (cherry picked from commit ea23271) Co-authored-by: Owain Davies <[email protected]> * [3.8] pythongh-95778: add doc missing in some places (pythonGH-100627) (python#101630) (cherry picked from commit 4652182) * [3.8] pythongh-101283: Improved fallback logic for subprocess with shell=True on Windows (pythonGH-101286) (python#101710) Co-authored-by: Oleg Iarygin <[email protected]> Co-authored-by: Steve Dower <[email protected]> * [3.8] pythongh-101981: Fix Ubuntu SSL tests with OpenSSL (3.1.0-beta1) CI i… (python#102095) [3.8] pythongh-101981: Fix Ubuntu SSL tests with OpenSSL (3.1.0-beta1) CI issue (pythongh-102079) * [3.8] pythonGH-102306 Avoid GHA CI macOS test_posix failure by using the appropriate macOS SDK (pythonGH-102307) [3.8] Avoid GHA CI macOS test_posix failure by using the appropriate macOS SDK. * [3.8] pythongh-101726: Update the OpenSSL version to 1.1.1t (pythonGH-101727) (pythonGH-101752) Fixes CVE-2023-0286 (High) and a couple of Medium security issues. https://www.openssl.org/news/secadv/20230207.txt Co-authored-by: Gregory P. Smith <[email protected]> Co-authored-by: Ned Deily <[email protected]> * [3.8] pythongh-102627: Replace address pointing toward malicious web page (pythonGH-102630) (pythonGH-102667) (cherry picked from commit 61479d4) Co-authored-by: Blind4Basics <[email protected]> Co-authored-by: C.A.M. Gerlach <[email protected]> Co-authored-by: Hugo van Kemenade <[email protected]> * [3.8] pythongh-101997: Update bundled pip version to 23.0.1 (pythonGH-101998). (python#102244) (cherry picked from commit 89d9ff0) * [3.8] pythongh-102950: Implement PEP 706 – Filter for tarfile.extractall (pythonGH-102953) (python#104548) Backport of c8c3956 * [3.8] pythongh-99889: Fix directory traversal security flaw in uu.decode() (pythonGH-104096) (python#104332) (cherry picked from commit 0aeda29) Co-authored-by: Sam Carroll <[email protected]> * [3.8] pythongh-104049: do not expose on-disk location from SimpleHTTPRequestHandler (pythonGH-104067) (python#104121) Do not expose the local server's on-disk location from `SimpleHTTPRequestHandler` when generating a directory index. (unnecessary information disclosure) (cherry picked from commit c7c3a60) Co-authored-by: Ethan Furman <[email protected]> Co-authored-by: Gregory P. Smith <[email protected]> Co-authored-by: Jelle Zijlstra <[email protected]> * [3.8] pythongh-103935: Use `io.open_code()` when executing code in trace and profile modules (pythonGH-103947) (python#103954) Co-authored-by: Tian Gao <[email protected]> * [3.8] pythongh-68966: fix versionchanged in docs (pythonGH-105299) * [3.8] Update GitHub CI workflow for macOS. (pythonGH-105302) * [3.8] pythongh-105184: document that marshal functions can fail and need to be checked with PyErr_Occurred (pythonGH-105185) (python#105222) (cherry picked from commit ee26ca1) Co-authored-by: Irit Katriel <[email protected]> * [3.8] pythongh-102153: Start stripping C0 control and space chars in `urlsplit` (pythonGH-102508) (pythonGH-104575) (pythonGH-104592) (python#104593) (python#104895) `urllib.parse.urlsplit` has already been respecting the WHATWG spec a bit pythonGH-25595. This adds more sanitizing to respect the "Remove any leading C0 control or space from input" [rule](https://url.spec.whatwg.org/GH-url-parsing:~:text=Remove%20any%20leading%20and%20trailing%20C0%20control%20or%20space%20from%20input.) in response to [CVE-2023-24329](https://nvd.nist.gov/vuln/detail/CVE-2023-24329). I simplified the docs by eliding the state of the world explanatory paragraph in this security release only backport. (people will see that in the mainline /3/ docs) (cherry picked from commit d7f8a5f) (cherry picked from commit 2f630e1) (cherry picked from commit 610cc0a) (cherry picked from commit f48a96a) Co-authored-by: Miss Islington (bot) <[email protected]> Co-authored-by: Illia Volochii <[email protected]> Co-authored-by: Gregory P. Smith [Google] <[email protected]> * [3.8] pythongh-103142: Upgrade binary builds and CI to OpenSSL 1.1.1u (pythonGH-105174) (pythonGH-105200) (pythonGH-105205) (python#105370) Upgrade builds to OpenSSL 1.1.1u. Also updates _ssl_data_111.h from OpenSSL 1.1.1u, _ssl_data_300.h from 3.0.9. Manual edits to the _ssl_data_300.h file prevent it from removing any existing definitions in case those exist in some peoples builds and were important (avoiding regressions during backporting). (cherry picked from commit ede89af) (cherry picked from commit e15de14) Co-authored-by: Gregory P. Smith <[email protected]> Co-authored-by: Ned Deily <[email protected]> * Python 3.8.17 * Post 3.8.17 * Updated CI to build 3.8.17 --------- Co-authored-by: Łukasz Langa <[email protected]> Co-authored-by: Benjamin Peterson <[email protected]> Co-authored-by: Ned Deily <[email protected]> Co-authored-by: Miss Islington (bot) <[email protected]> Co-authored-by: HARSHA VARDHAN <[email protected]> Co-authored-by: Gregory P. Smith <[email protected]> Co-authored-by: Jeremy Paige <[email protected]> Co-authored-by: Hugo van Kemenade <[email protected]> Co-authored-by: Steve Dower <[email protected]> Co-authored-by: Owain Davies <[email protected]> Co-authored-by: Éric <[email protected]> Co-authored-by: Oleg Iarygin <[email protected]> Co-authored-by: Steve Dower <[email protected]> Co-authored-by: Dong-hee Na <[email protected]> Co-authored-by: Blind4Basics <[email protected]> Co-authored-by: C.A.M. Gerlach <[email protected]> Co-authored-by: Pradyun Gedam <[email protected]> Co-authored-by: Petr Viktorin <[email protected]> Co-authored-by: Sam Carroll <[email protected]> Co-authored-by: Ethan Furman <[email protected]> Co-authored-by: Jelle Zijlstra <[email protected]> Co-authored-by: Tian Gao <[email protected]> Co-authored-by: Irit Katriel <[email protected]> Co-authored-by: stratakis <[email protected]> Co-authored-by: Illia Volochii <[email protected]>

So that the script can be used on Debian 12. > Changed in version 3.11.4: Added the filter parameter. -- https://docs.python.org/3.11/library/tarfile.html The implementation was in python/cpython#102953 Before this change, Debian 12 has Python 3.11.2, so this script errors on a call with `filter=` After this change the script does not error.

Implement PEP 706 – Filter for tarfile.extractall

cde089c

encukou requested a review from gpshead March 23, 2023 14:21

encukou requested a review from ethanfurman as a code owner March 23, 2023 14:21

Add a blurb

2395c92

ethanfurman requested changes Mar 23, 2023

View reviewed changes

eryksun reviewed Mar 23, 2023

View reviewed changes

Lib/tarfile.py Show resolved Hide resolved

encukou added 4 commits March 27, 2023 14:34

Remove unneeded backslashes in docs

561a9d4

Fix typos in docs & comments

54b5644

Thanks to Ethan for spotting these

Clarify the arguments to register_unpack_format's +function* argument

7867fc3

tests: Clean up unused variables and unnecessary f-strings

734190d

hroncok reviewed Apr 4, 2023

View reviewed changes

gpshead reviewed Apr 5, 2023

View reviewed changes

encukou added 5 commits April 6, 2023 17:43

Make filter keyword-only in shutil

d55ae42

Address documentation review

c795bb3

Fix ReST syntax

5d850cc

Merge main branch to resolve conflict in Whatsnew

81e9050

Remove typo

de11090

hroncok reviewed Apr 17, 2023

View reviewed changes

Improve errorlevel handling

b0c0674

gpshead reviewed Apr 19, 2023

View reviewed changes

Lib/tarfile.py Show resolved Hide resolved

Doc/library/tarfile.rst Show resolved Hide resolved

hroncok reviewed Apr 19, 2023

View reviewed changes

Lib/test/test_tarfile.py Outdated Show resolved Hide resolved

Lib/test/test_tarfile.py Show resolved Hide resolved

encukou added 2 commits April 19, 2023 15:58

Fix the tests to match the PEP update

3796fdf

Document type of errorlevel

08ab551

encukou added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 19, 2023

Skip symlink-related test failures on WASI

0474842

Ignore unrelated errors in NoneInfoTests_Misc.test_add

a255634

encukou added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Apr 20, 2023

encukou merged commit af53046 into python:main Apr 24, 2023

encukou deleted the tarfile-dir-traversal-sqsq branch April 24, 2023 08:58

encukou added a commit to encukou/cpython that referenced this pull request Apr 25, 2023

pythongh-102950: Implement PEP 706 – Filter for tarfile.extractall (p…

b52ad18

…ython#102953) Backport of af53046

mcepl mentioned this pull request May 3, 2023

[3.10] gh-102950: Implement PEP 706 – Filter for tarfile.extractall (GH-102953) #104128

Merged

ambv pushed a commit that referenced this pull request May 15, 2023

[3.9] gh-102950: Implement PEP 706 – Filter for tarfile.extractall (G…

98016f7

…H-102953) (#104382) Backport of c8c3956

ambv pushed a commit that referenced this pull request May 17, 2023

[3.8] gh-102950: Implement PEP 706 – Filter for tarfile.extractall (G…

79e63e5

…H-102953) (#104548) Backport of c8c3956

CharlieZhao95 mentioned this pull request Sep 4, 2023

Update missing Python CVE (CVE-2007-4559) psf/advisory-database#22

Closed

jwhitlock mentioned this pull request Oct 16, 2023

Update to Python 3.10.13 mozilla/fx-private-relay#4010

Merged

This comment was marked as spam.

Sign in to view

edgarrmondragon mentioned this pull request May 10, 2024

Update tarfile.extractall and friends to use filter='data' by default #118875

Open

maxwell-k mentioned this pull request Jul 16, 2024

Support Python 3.11.2 maxwell-k/dotlocalslashbin#7

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-102950: Implement PEP 706 – Filter for tarfile.extractall #102953

gh-102950: Implement PEP 706 – Filter for tarfile.extractall #102953

encukou commented Mar 23, 2023 •

edited by bedevere-bot

Loading

ethanfurman left a comment

encukou commented Mar 31, 2023

encukou commented Apr 3, 2023

hroncok left a comment

hroncok Apr 4, 2023

gpshead Apr 5, 2023

encukou Apr 6, 2023

encukou commented Apr 17, 2023

hroncok Apr 17, 2023 •

edited

Loading

encukou Apr 18, 2023

hroncok commented Apr 17, 2023

encukou commented Apr 20, 2023

encukou commented Apr 20, 2023

encukou commented Apr 20, 2023

encukou commented Apr 24, 2023 •

edited

Loading

kulikjak commented Apr 25, 2023

encukou commented Apr 25, 2023 •

edited

Loading

encukou commented May 16, 2023

This comment was marked as spam.


		.. versionadded:: 3.12

		The tar format is designed to capture all details of a UNIX-like ecosystem,

gh-102950: Implement PEP 706 – Filter for tarfile.extractall #102953

gh-102950: Implement PEP 706 – Filter for tarfile.extractall #102953

Conversation

encukou commented Mar 23, 2023 • edited by bedevere-bot Loading

ethanfurman left a comment

Choose a reason for hiding this comment

encukou commented Mar 31, 2023

encukou commented Apr 3, 2023

hroncok left a comment

Choose a reason for hiding this comment

hroncok Apr 4, 2023

Choose a reason for hiding this comment

gpshead Apr 5, 2023

Choose a reason for hiding this comment

encukou Apr 6, 2023

Choose a reason for hiding this comment

encukou commented Apr 17, 2023

hroncok Apr 17, 2023 • edited Loading

Choose a reason for hiding this comment

encukou Apr 18, 2023

Choose a reason for hiding this comment

hroncok commented Apr 17, 2023

encukou commented Apr 20, 2023

encukou commented Apr 20, 2023

encukou commented Apr 20, 2023

encukou commented Apr 24, 2023 • edited Loading

kulikjak commented Apr 25, 2023

encukou commented Apr 25, 2023 • edited Loading

encukou commented May 16, 2023

This comment was marked as spam.

encukou commented Mar 23, 2023 •

edited by bedevere-bot

Loading

hroncok Apr 17, 2023 •

edited

Loading

encukou commented Apr 24, 2023 •

edited

Loading

encukou commented Apr 25, 2023 •

edited

Loading