Optimize pathlib path construction

Pathlib is slow. One of the most obvious symptoms is that pathlib.PurePath objects are slow to construct. We should be able to speed construction up without making other parts of pathlib slower.

Two possible approaches:

Optimize the existing machinary of path construction: __new__(), _from_parts(), _parse_parts(), _parse_args().
Perform less work in the constructor: defer parsing, joining and normalization until needed.

Linked PRs

I'd like to land #101363 before I put the first PR up for this issue.

pythonGH-101362 - Optimise pathlib by deferring path normalisation

pythonGH-101362: Check pathlib.Path flavour compatibility at import time

pythonGH-101362: Call join() only when >1 argument supplied to pathli…

pythonGH-101362: Optimise PurePath(PurePath(...))

Does the PR cope with WindowsPurePath and PosixPurePath and converting between them?

Could you clarify? The PRs maintain the behaviour that attempting to instantiate pathlib.PurePath will give you a PureWindowsPath or a PurePosixPath depending on your platform, but pathlib doesn't support converting between Windows and POSIX paths as I understand it.

This behaviour should be preserved:

Python 3.10.10 (tags/v3.10.10:aad5f6a, Feb  7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from pathlib import *
>>> p = PureWindowsPath("a/b/c")
>>> p
PureWindowsPath('a/b/c')
>>> PurePosixPath(p)
PurePosixPath('a/b/c')
>>> PurePath(p)
PureWindowsPath('a/b/c')
>>> PosixPath(p)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__3847v3x7pw1km\lib\pathlib.py", line 962, in __new__
    raise NotImplementedError("cannot instantiate %r on your system"
NotImplementedError: cannot instantiate 'PosixPath' on your system
>>> WindowsPath(p)
WindowsPath('a/b/c')
>>>

Right! That will be broken by #101667 as things stand:

>>> from os import fspath
>>> from pathlib import *
>>> p = PureWindowsPath("a/b/c")
>>> p
PureWindowsPath('a/b/c')
>>> fspath(p)
'a\\b\\c'
>>> PurePosixPath(fspath(p))
PurePosixPath('a\\b\\c')
>>> PurePosixPath(p)
PurePosixPath('a\\b\\c')

It doesn't appear to be documented or tested behaviour, and it feels odd to me that PurePosixPath(p) can give a different result to PurePosixPath(fspath(p)). I'll try to find a good way forwards!

This feature doesn't really work with drives or roots:

>>> PurePosixPath(PureWindowsPath('//server/share/dir'))
PurePosixPath('\\\\server\\share\\/dir')
>>> PurePosixPath(PureWindowsPath('c:/dir'))
PurePosixPath('c:\\/dir')
>>> PurePosixPath(PureWindowsPath('/dir'))
PurePosixPath('\\/dir')

As far as I can tell, no one has ever logged a bug about it.

However, using PurePosixPath(PureWindowsPath(...).as_posix()) works everywhere:

>>> PurePosixPath(PureWindowsPath('//server/share/dir').as_posix())
PurePosixPath('//server/share/dir')
>>> PurePosixPath(PureWindowsPath('c:/dir').as_posix())
PurePosixPath('c:/dir')
>>> PurePosixPath(PureWindowsPath('/dir').as_posix())
PurePosixPath('/dir')

So I'm tempted to conclude that converting with PurePosixPath(PureWindowsPath(...)) alone is not supported, and that it only happened to work in some limited circumstances. Users can add .as_posix() to signal their intent and make it work reliably. What do you reckon?

I think the direct conversion should either be consistent with fspath(p) or p.parts, but no strong preference which. As you say, neither makes a huge amount of sense, and the workaround (.as_posix()) has been there for a long time.

Might be worth adding a short note to the docs warning that the constructor can't reliably convert from different pathlib objects, and maybe specify whether it will fspath or .parts them.

On another perf note, is it possible that parsing the path up front isn't necessary? Obviously it'll save the most time to keep a single string literal around and parse it later, but I don't personally have a good feel for whether that's common or not. (Obviously if it's available pre-parsed then keep it.)

pythonGH-101362: Check pathlib.Path flavour compatibility at import t…

pythonGH-101362: Optimise PurePath(PurePath(...)) (pythonGH-101667)

pythonGH-101362: Omit path anchor from `pathlib.PurePath()._parts`

Merge branch 'main' into nodeaddict

Merge branch 'main' into inlinecomp2

Merge branch 'main' into pythongh-101362-exclude-anchor-from-parts

For anyone following along, I think this ticket can be resolved if/when this PR lands:

GH-101362: Omit path anchor from pathlib.PurePath()._parts #102476

Merge branch 'main' into pythongh-101362-exclude-anchor-from-parts

GH-101362: Omit path anchor from `pathlib.PurePath()._parts` (GH-102476)

Resolving this issue. It's now much cheaper to construct pathlib.PurePath and Path objects. Some operations like str() and joinpath() are also cheaper.

Future work:

Optimize pathlib.PurePath.__fspath__() #102783
Path.rglob performance issues in deeply nested directories compared to glob.glob(recursive=True) #102613
Add C implementation of os.path.splitroot() #102511
Add C implementation of os.path.join()

pythonGH-101362: Omit path anchor from `pathlib.PurePath()._parts` (p…

Replace regex approach to ls-files output parsing

Navigation Menu

Optimize pathlib path construction #101362

Linked PRs

7 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Optimize pathlib path construction #101362

Description

Linked PRs

Activity

barneygale commented on Jan 29, 2023

zooba commented on Feb 22, 2023

barneygale commented on Feb 22, 2023

zooba commented on Feb 24, 2023

barneygale commented on Feb 24, 2023

barneygale commented on Feb 24, 2023

zooba commented on Feb 24, 2023

7 remaining items

barneygale commented on Mar 22, 2023

barneygale commented on Apr 9, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions