Skip to content

feat: Postgres build#5381

Draft
nielsenko wants to merge 26 commits into
serverpod:mainfrom
nielsenko:feat-postgres-build
Draft

feat: Postgres build#5381
nielsenko wants to merge 26 commits into
serverpod:mainfrom
nielsenko:feat-postgres-build

Conversation

@nielsenko

@nielsenko nielsenko commented Jun 24, 2026

Copy link
Copy Markdown
Collaborator

TO BE DONE

Fixes: #5119

@nielsenko nielsenko force-pushed the feat-postgres-build branch 6 times, most recently from ae31246 to 889620b Compare June 24, 2026 14:09
@nielsenko nielsenko self-assigned this Jun 24, 2026
@nielsenko nielsenko force-pushed the feat-postgres-build branch 23 times, most recently from 2362fe1 to eb2e271 Compare June 26, 2026 10:41
@nielsenko nielsenko force-pushed the feat-postgres-build branch 2 times, most recently from b41ef5b to c22c9a5 Compare July 2, 2026 13:59
@nielsenko nielsenko force-pushed the feat-postgres-build branch from c22c9a5 to 3713200 Compare July 2, 2026 22:10
nielsenko added 15 commits July 3, 2026 00:19
Previously every suite scaffolded into a fixed <repo>/temp* directory, which
forced hand-assigned unique names per file (to avoid collisions) and a
.gitignore entry. Generate projects in a fresh Directory.systemTemp.createTemp
instead: collisions are impossible by construction going forward, nothing
lands in the repo tree, and it matches the cwd-independence the generated
projects already have (their path deps are absolute via SERVERPOD_HOME).
The generated server now uses embedded PostgreSQL
Drop the --concurrency=1 that the suite needed only for the shared Docker
database; the generated projects now use embedded PostgreSQL, so the suites
isolate themselves and can run at the default concurrency.
…tor bundles

- Replace Zonky extension-less binaries with Serverpod bundles (PG + PostGIS + pgvector) from GitHub Releases.
- Add a BinaryArtifact seam; BinaryStore generic over it.
- Add ServerpodBundleArtifact + .tar.xz extraction; keep ZonkyArtifact for back-compat/tests.
- Supervisor sets PROJ_LIB to bundle share/proj so PostGIS transforms survive cache relocation.
- Validate prefetch(target:) against published suffixes; post-extract shape check before caching.
- README: shipped PostGIS 3.5.4 + pgvector 0.8.3; add artifact URL/platform-mapping test.
…s + CI

- tool/build_postgres builds PG 16.13 + PostGIS 3.5.4 + pgvector 0.8.3 via zig cc/c++ into a relocatable .tar.xz.
- GitHub Actions matrix builds per-runner, publishes an embedded-postgres-v<bom> release (PG_VERSION from the tag).
- zigshim adapts macOS flags zig mishandles (-bundle, libc++ headers, UBSan, headerpad).
- zigshim: -Wno-unknown-warning-option on real compiles only; honors ZIG_VERSION for anyzig pinning.
- build-core force-keeps backend symbols so extensions resolve at load.
- package.sh archives STAGE at tar root (extract resolves bin/postgres); codesign arm64-only.
- Windows scaffold: x86_64-windows-gnu under MSYS2, geo .dll beside postgres.exe (UNVALIDATED).
- macOS x64 verified end-to-end.
- BinarySource { auto, download, build }; set via EmbeddedPostgresOptions.binarySource or SERVERPOD_PG_SOURCE.
- auto falls back to a local build only on a 404 (not transient errors); build forces it; download is unchanged.
- BundleBuilder runs tool/build_postgres/build-all.sh (package-relative or SERVERPOD_PG_BUILD_DIR), streams output.
- BinaryStore.ensure takes (source, builder); widens claim stale/timeout windows when a build may run.
- BinaryFetchException carries statusCode so auto can distinguish 404.
- Lets CI build on normal runs (force, no publish); publish stays tag-triggered.
…le in CI

- From-source relocatable PG 16.13 + PostGIS 3.5.4 + pgvector 0.8.3 bundle, downloaded per-platform at runtime.
- No Docker to provision a DB for tests or local dev.
- Toolchain: Apple clang on macOS (zig-clang 21 miscompiled the backend), zig cc glibc-2.28 on Linux, mingw gcc on Win.
- Static GEOS/PROJ/json-c; PostGIS -j1 (rtpostgis.sql.tmp race); raster/protobuf/address_standardizer/NLS off.
- Windows: export GEOS/PROJ DLL symbols, force-include <cstdint> for PROJ, materialize PostGIS symlinks as files.
- macOS: rewrite install_names/rpaths to @loader_path (re-sign on arm64).
- CI: build each bundle once in a seed job, hand to test jobs via artifact; restore-only cache + gated save.
- Run embedded-PG integration suites on Linux, macOS, Windows.
…t lock on start

- A postmaster mid-shutdown removes postmaster.pid early but holds the Unix socket lock until it fully exits.
- A launch then passed the pid liveness check but crashed: "lock file ... already exists".
- Port-retry only covered TcpTransport, so the Unix-socket case had no recovery.
- On start, read run/.s.PGSQL.<port>.lock; if it names a live PID, wait for exit (reuse waitForPidExit).
- add ubuntu-24.04-arm + macos-15 legs to seed_embedded_pg and the two embedded
  test matrices, so every <os>-<arch> ServerpodBundleArtifact serves (Abi.current)
  is built/tested, not just x64
- the embedded-PG test jobs are server-side `dart test` (no Flutter), and Flutter
  ships no Linux-arm64 SDK, so swap flutter-action -> setup-dart on them
@nielsenko nielsenko force-pushed the feat-postgres-build branch from 3713200 to d01d13d Compare July 2, 2026 22:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Custom built Postgres to support PostGIS and pgvector

1 participant