pythongh-12005: Align FileIO.readall between _pyio and _io

cmaloney · cmaloney · commit 50389df0a5f3 · 2025-02-05T13:28:47.000-08:00
Utilize `bytearray.resize()` and `os.readinto()` to reduce copies
and match behavior of `_io.FileIO.readall()`.

There is still an extra copy which means twice the memory required
compared to FileIO because there isn't a zero-copy  path from
`bytearray` -&gt; `bytes` currently.

On my system reading a 2GB file
`./python -m test -M8g -uall test_largefile -m test.test_largefile.PyLargeFileTest.test_large_read -v`

Goes from ~2.7 seconds -&gt; ~2.2 seconds
diff --git a/Lib/_pyio.py b/Lib/_pyio.py
@@ -1674,22 +1674,31 @@ def readall(self):
                 except OSError:
                     pass
 
-        result = bytearray()
+        result = bytearray(bufsize)
+        bytes_read = 0
         while True:
-            if len(result) >= bufsize:
-                bufsize = len(result)
-                bufsize += max(bufsize, DEFAULT_BUFFER_SIZE)
-            n = bufsize - len(result)
+            if bytes_read >= bufsize:
+                # Parallels _io/fileio.c new_buffersize
+                if bufsize > 65536:
+                    addend = bufsize >> 3
+                else:
+                    addend = 256 + bufsize
+                if addend < DEFAULT_BUFFER_SIZE:
+                    addend = DEFAULT_BUFFER_SIZE
+                bufsize += addend
+                result.resize(bufsize)
+
+            assert bufsize - bytes_read > 0, "Should always try and read at least one byte"
             try:
-                chunk = os.read(self._fd, n)
+                n = os.readinto(self._fd, memoryview(result)[bytes_read:])
             except BlockingIOError:
-                if result:
+                if bytes_read:
                     break
                 return None
-            if not chunk: # reached the end of the file
+            if n == 0:  # Reached the end of the file
                 break
-            result += chunk
-
+            bytes_read += n
+        result.resize(bytes_read)
         return bytes(result)
 
     def readinto(self, buffer):
diff --git a/Misc/NEWS.d/next/Library/2025-02-05-13-19-15.gh-issue-129005.Sb69L_.rst b/Misc/NEWS.d/next/Library/2025-02-05-13-19-15.gh-issue-129005.Sb69L_.rst
@@ -0,0 +1,2 @@
+``_pyio.FileIO.readall()`` now allocates, resizes, and fills a data buffer
+using the same algorithm ``_io.FileIO.readall()`` uses.

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,2 @@`
	`1`	+``_pyio.FileIO.readall()`` now allocates, resizes, and fills a data buffer
	`2`	+using the same algorithm ``_io.FileIO.readall()`` uses.