Skip to content
BenchBarBenchBar

Decisions

One line per non obvious choice: the decision, then the reason. The decisions of the app work live in macos/DECISIONS.md. The 0.5.5, 0.5 and 0.4 runs come first, the 0.3 easy install run follows.

  • benchbar docs checks the page with one curl -I (3 seconds) before opening it: a 404 opens the start page with a warning, and no answer (offline, DNS, a slow network) opens the page anyway, so the check can only improve on a plain open, never block it.
  • benchbar docs --print prints the URL without opening it or touching the network: agents and scripts want the link, not a browser window.
  • Topics are short names with aliases (doctor and repair, teams, profiles and lock), kept in one table in lib/frappe-local/docs.sh; an unknown topic exits 1 with the list, so a typo never opens a wrong page.
  • The doctor see: line goes under every [FAIL], after fix:, and never under [WARN] or in --json: AGENTS.md promises parsers the existing lines byte for byte, and a new line starting with see: is ignored by anything that reads [FAIL] and fix:. Every check id has an anchor in the doctor guide, so no per check table is needed.
  • --version keeps benchbar X.Y.Z as the first line and adds BenchBar app X.Y.Z (path) only when an app is installed: the app, install.sh and the tests read the first line. The app version is read from the XML Info.plist with awk, the helper report already used, instead of plutil or defaults, so the tests can fake an app with a small file.
  • report --json sends the human lines to stderr and prints one object on stdout, with schema_version and cli_version like every other document; redactions counts the lines where something was replaced (the sum of what REDACTIONS.txt lists). --json and --print together are refused.
  • The bug report form’s version fields are prefilled by the ids bug_report.yml uses: macos and version.

0.5.5: repo foundation

  • The Code of Conduct’s contact is a private advisory on GitHub until a contact email exists: the git address is a noreply address that cannot receive mail, and an advisory is private to the maintainers.
  • The AI policy welcomes AI assisted pull requests with a one line disclosure instead of banning them: BenchBar is itself built with coding agents (AGENTS.md), and a ban nobody can check only hides the tool.
  • good first issue items are kept for people working by hand: they exist to teach the codebase, and an agent solving them teaches nobody.
  • The bug form requires the benchbar report zip and doctor output: those two answer most first questions, and the zip is already redacted.
  • Blank issues are off and questions go to Discussions Q&A: the issue list stays a list of things to fix.
  • scripts/social-preview.swift uses AppKit, not Python: the Mac has no imaging library by default and the repo already builds with Swift; scripts/app-icon.py only writes SVG text.
  • The social preview and the og:image come from one script with a size argument: one design, two sizes (1280x640 for GitHub, 1200x630 for the docs site), no drift between them.
  • The runner strip in the preview is cut from docs/images/runners.png, with the grey tiles made transparent: the frames stay the ones the app draws, without rendering them again.
  • Dependabot watches /macos for Swift: the package list lives in macos/project.yml (XcodeGen) and the generated .xcodeproj is not committed, so Swift updates only start once a Package.resolved is in the repo.
  • Vouch waits until drive-by pull requests appear: a trust list for a project with one maintainer and few contributors is ceremony.

0.5.5: docs site

  • The site reads docs/ in place through a glob loader with base: ".." in site/src/content.config.ts, not a symlink: Starlight 0.42 supports any Astro loader for the docs collection, a symlink breaks on Windows clones and in some editors, and entry.filePath stays a real repo path, so “Last updated” comes from the file’s own git history.
  • editLink.baseUrl is https://github.com/askysh/benchbar/edit/main/site/: Starlight appends entry.filePath, which is ../docs/<page>.md relative to site/, and the URL resolves to edit/main/docs/<page>.md.
  • Page URLs are the lowercased file paths under docs/ (DECISIONS.md is /decisions/), set by the loader’s generateId, so the URLs the README and the app link to follow the file names.
  • ROADMAP.md and CONTRIBUTING.md stay at the repo root with no frontmatter; site/scripts/prepare.mjs copies them into site/.generated/ (ignored by git) before each build with a title from their # heading, their own edit link and last commit date. A missing CONTRIBUTING.md is skipped, and the sidebar leaves it out.
  • Links in docs/ stay relative .md paths so GitHub renders them; a remark plugin (site/src/remark-repo-links.mjs) turns them into site URLs, and links to other repo files into GitHub links, at build time.
  • Screenshots in the moved README content use markdown image syntax, not <img width>: Astro optimises markdown images (webp, a fifth of the size) and ignores raw <img> paths.
  • The heading slugs come from github-slugger, which keeps underscores, so ### port_clash is #port_clash; tests/test-docs.sh checks a heading for every check id in FL_CHECK_ORDER plus port_block.
  • The CLI reference documents only flags that benchbar --help shows (tests/test-docs.sh checks it). report --out DIR and app update --json without --dry-run exist in the code but not in --help, so they are left out until --help names them.
  • Astro and Starlight run under Bun (bun --bun astro build): Astro 7 needs Node 22.12 or later, and Bun alone avoids a second runtime in CI and on a contributor’s Mac. @astrojs/markdown-remark is a direct dependency because Astro 7 only runs remark plugins through it.
  • The docs workflow is the official Astro Pages workflow with its steps written out (withastro/action would run bun install without --frozen-lockfile and has no place for the link check).
  • Docs only changes skip the CLI and app jobs through a changes job (dorny/paths-filter) and if: conditions, not a workflow paths-ignore: a workflow that never starts leaves a required check pending forever, a skipped job counts as a pass. The matrix shards skip step by step on an Ubuntu runner, because a matrix job skipped as a whole reports one check under its unexpanded name. A push to main skips the same way: the pull request already ran everything, and workflow_dispatch runs all jobs.
  • The docs workflow runs on every pull request and skips its build job the same way when no docs file changed, so “Docs build” can be required too. It also runs tests/test-docs.sh, which the CLI shards skip on a docs only change.
  • The link check is linkinator over site/dist with fragment checks on, skipping every external URL: GitHub and the site’s own domain (before DNS) would make the check flaky, and the internal links are the ones a docs change breaks.
  • The Umami script uses the benchbar.akashmishra.com website of Akash’s Umami, limited with data-domains so local previews are not counted, and og:image points at https://benchbar.akashmishra.com/og.png, which the build copies from docs/images/og-image.png when that file exists.

0.5.5: readme

  • The README keeps only what a visitor needs to decide and start (why, install, quick start, features) and links four deep pages of the docs site: a 3,200 word README had become the manual, which nobody can navigate on GitHub, and the docs site now has search and a sidebar.
  • The README runs about 800 words, not the 1,100 the brief suggested: the fixed order and one line per item leave no room for more without padding.
  • The logo and the hero image switch with the reader’s appearance through <picture>; the icons are exported from AppIcon.icon with Icon Composer’s ictool (Default and Dark renditions), so they match the app icon exactly.

0.5: app installs

Designed against frappe/bench develop (c9d1250) and frappe version-15; the code moved on since the design, and the code won where they disagreed:

  • Everything per bench goes through the existing per bench state (.benchbar/benches/<name>-<hash>.env, fl_bstate_get/set); the site app cache is a sibling file, <name>-<hash>.site-apps, not the <name>.site-apps of the design.
  • install-app and migrate run with the bench’s own Redis up (fl_bench_redis_up/down from 0.4), because frappe v16 connects to it during both; the exit handler stops what was started, however the command ends.
  • Never get-app --resolve-deps: it reads hooks.py through the GitHub API (no private repos) and can rmtree an existing app. required_apps is read from the local hooks.py after the clone, resolved through apps.tsv, and cloned after a second plan and question.
  • Never get-app --overwrite, and stdin is /dev/null: an app that exists is reported with the manual git fetch and git checkout, and a surprise click.confirm fails instead of hanging.
  • --skip-assets, then one bench build --app X (or --apps a,b with the required apps): one build, after every dependency is in place.
  • Access is checked before any change with git ls-remote under GIT_TERMINAL_PROMPT=0 and GIT_SSH_COMMAND='ssh -o BatchMode=yes': a private repo without a key or token fails in a second with a fix line (ssh -T git@HOST and ssh-add for SSH, which also covers host aliases; the SSH URL or gh auth setup-git for HTTPS) instead of waiting on a hidden prompt.
  • A repo URL with a user name or token is refused: it would land in .git/config, app list --json and later a committed lockfile. Remote URLs are always shown without userinfo.
  • get-app is always given the resolved URL, also for names like erpnext that bench could resolve itself: the preflight checked exactly that URL, and apps.tsv entries without a repo map to github.com/frappe/<name>, which is what bench uses.
  • The app name is the repo name (lower case, - as _) or --name; the folder bench actually created is found by comparing apps/ before and after, so Raven becoming raven is followed.
  • A get-app that fails half way moves the new folder to .benchbar/backups/<ts>/apps/ only while sites/apps.txt does not list it. Once bench has listed it, the folder and the line stay and the fix (bench remove-app NAME) is printed: benchbar never edits sites/ (AGENTS.md), and doctor’s apps_txt check keeps reporting it. The first version removed the line itself; the review before the 0.5 PR reverted that.
  • A dirty app is left alone by add (it only installs on sites) but refused by update: add never touches the code, update moves it.
  • app update is git fetch plus merge --ff-only, never bench update: one app changes, nothing is rebased or reset. A diverged branch stops with the git log --graph to look at; an app ahead of its remote is “unchanged”. On a failure the reset --hard OLD is printed, never run.
  • A shallow clone is fetched with --shallow-since=@<HEAD time - 1s>, so old..new exists locally for the changelog and the fast forward check.
  • update --dry-run runs git fetch: the changelog needs the new commits. Only the app’s .git changes; the working tree, the sites and benchbar’s state do not, and the test snapshots exactly that.
  • Every site that has the app is backed up (bench --site S backup) before the merge, --skip-backup opts out: the repo rule is a backup before every change, and migrate is the step that can hurt data.
  • Which sites have an app comes from bench --site S list-apps --format json with a 15 second limit per site, cached per bench: it needs MariaDB, and doctor (on a timer in the app), app list --no-sites and the lockfile check must never need it. A failed read keeps the last good list and says so in sites_error.
  • app update --json is only the plan (--dry-run --json): the app shows the changelog, then runs the update with --yes and a live log, and a JSON stream of an apply would need its own format.
  • The doctor checks apps_txt and app_branch_policy have no repair action: fixing either means changing which code runs, a person’s call.
  • --site stays the global option, so app install NAME --site S and app add X --site S parse like every other command.
  • The fake bench of the tests now has an apps/erpnext folder: it always listed erpnext in apps.txt, which the new apps_txt check rightly fails.
  • app add takes the app name from the repo name (or --name), never from apps.tsv, when given a URL: apps.tsv is only the catalog for names.
  • The git mock passes everything to the real git under MOCK_GIT_REAL=1, and the bench mock’s get-app then really clones (--origin upstream, shallow with MOCK_BENCH_SHALLOW=1) from bare repos in the test folder over file://. MOCK_GIT_LSREMOTE_EXIT fakes a private repo. No test touches the network.

0.5: team profiles

  • A team profile lives outside BenchBar: ~/.config/benchbar/profiles/NAME.toml, then each folder of BENCHBAR_PROFILE_PATH (colon separated), so an organisation’s repos and branches never land in config/ or in this repo, and a team shares its recipe through its own config repo.
  • Built in profiles are looked up first, and a team file named like one (v15-lts.toml) is reported as invalid instead of loaded: --profile v15-lts must mean the same thing on every Mac. A later file with an earlier file’s name is hidden, and profile list says so.
  • A team profile names a built in base and never its own Python, Node or MariaDB: those formulae, the MariaDB range and the doctor checks are tested per built in profile. Only frappe_branch may be overridden.
  • PROFILE in the bench’s state stays the base, and TEAM_PROFILE names the team profile: every daily command, doctor included, works from the base alone, and follows the team’s app branches (the app_branch_policy check, app add NAME) while the file exists and parses. A missing or broken team file falls back to the base instead of breaking doctor.
  • The team’s apps plug into the existing policy lookup (fl_lookup_app_policy asks the loaded team profile first), so phase 01, app add NAME and required apps all resolve a team app the same way.
  • The phases get the team profile’s name (--profile acme) and load it themselves: phase 00 needs only the base, phase 01 the apps, site and pins, and benchbar install stays a thin wrapper.
  • bundle and [[apps]] exclude each other: a bundle is the built in app list, [[apps]] the team’s own, and mixing them would make the order of apps.txt depend on rules nobody wrote down.
  • An optional commit in [[apps]] goes through phase 01’s existing pin (fetch --all --tags, checkout COMMIT), which leaves a detached HEAD; exact pins for identical benches are the lockfile’s job, a profile is the recipe.
  • profile create --from-bench reads the bench only: remote URLs without user info, current branches (the policy branch for a detached app, else the app is skipped with a warning), the base from apps/frappe’s version, the default site name and the scheduler choice. It writes no commits and nothing from sites/ except that name; the written file is parsed back before it is offered, and an existing file is shown as a diff and backed up.
  • The format is the strict TOML subset of lib/frappe-local/toml.sh, shared with the lockfile: one parser, and a file this parser accepts is valid TOML. schema = 1 is optional in a profile (written by create) and refused when it is another number.
  • profile create takes the checkout’s lock like other writing commands; profile list and show do not.

0.5: the team lockfile

  • The lockfile is an awk parsed TOML subset (toml.sh, shared with team profiles), not tomllib: the Command Line Tools’ /usr/bin/python3 is 3.9 without it, the bench’s env Python is missing exactly when a check matters, doctor stays pure shell, and a fallback would mean two parsers that can disagree. Every refusal names the line (benchbar.toml:3: not supported: inline tables).
  • The code lives in teamlock.sh: lock.sh already is the checkout’s run lock, and the two must not be confused.
  • Lookup is --lock PATH, BENCHBAR_LOCK, LOCK_FILE in the bench’s own state file (fl_bstate_set, not the <name>.env of the design), then <bench>/benchbar.toml. A relative --lock is taken from the current folder when it exists there, else from the bench, so --lock apps/acme/benchbar.toml works from anywhere. The path is remembered by write, check and apply (not in a dry run); BENCHBAR_LOCK never is, and list --json ignores it since it names one bench, not all.
  • The app that holds the lockfile gets no commit: committing the lockfile moves that app’s HEAD, so its pin would always be one commit behind and every teammate would see commit_ahead.
  • Frappe is an ordinary [[app]]: one code path, and the entry order is apps.txt’s, so apply clones in dependency order.
  • Repos compare as host and path (git@github.com:acme/x.git equals https://github.com/acme/x): teammates clone over SSH or HTTPS as they like. An SSH host alias from ~/.ssh/config is not resolved and reads as another repo (a warning only).
  • lock check is read only and offline: git reads, apps.txt, and the site app cache that app list fills, so it works with MariaDB stopped. The one outside call is bench --version (15 seconds at most) for bench_version_mismatch; doctor’s lock_drift leaves that kind out and calls nothing.
  • app_missing is the only fail level: the bench lacks code the team’s sites need. Everything else is a warning, and doctor reports the whole lock as one lock_drift warning pointing at lock check.
  • Drift exits 1 like a failing doctor, through the dispatcher (|| exit 1), so lock check --json prints only the JSON.
  • lock apply stops at code: no new-site, install-app, migrate or bench update. Missing sites and site apps are printed as benchbar site add and benchbar app install lines, and a changed app ends with the reminder to migrate: site data is the user’s to change.
  • A pinned commit is reached with git merge --ff-only, after git fetch REMOTE SHA when it is not local (protocol v2 serves any reachable commit), and --unshallow for a shallow clone as the fallback. The design’s fetch --depth 1 SHA was dropped: a depth 1 fetch cuts the history between HEAD and the pin, so the fast forward check fails.
  • An app that is ahead of its pin, diverged from it, or dirty is skipped with a warning and apply still exits 0: local work outranks the lock, and a second apply must say unchanged, not fail forever.
  • A branch switch happens only on a clean tree: the locked branch is fetched into its remote tracking ref (--depth 1 for a shallow clone), then checked out as the local branch when there is one, else created with --track. checkout -B is never used on an existing branch, since it would drop that branch’s local commits.
  • A freshly cloned app whose branch tip is ahead of the pin is moved to the pin with checkout -B BRANCH SHA: the clone is seconds old and holds no local work, so this is not the reset the rules forbid.
  • Cloning goes through app add’s own steps (access check, get-app --skip-assets, the half clone rollback); setup requirements --python and --node run only for apps whose code changed in place (get-app already did it for new clones), and one bench build --apps a,b covers all of them.
  • lock write refreshes the site app lists from bench (it is the one lock command that may ask the database) and falls back to the cache with a warning. A dirty or detached app is refused unless --allow-dirty, which pins the commit as it is (a detached app then takes its policy branch).
  • The written file is parsed back before it is shown, so write can never produce a file check rejects. decisions of the app work live in macos/DECISIONS.md. The 0.5 and 0.4 runs come first, the 0.3 easy install run follows.

0.5: benchbar pull

Upstream facts behind the design, frappe version-15 and version-16 (the same in both), from the design notes:

  • F1 frappe/commands/site.py restore: --encryption-key is the gpg passphrase for -enc backup files, not the site encryption_key; without it restore reads backup_encryption_key from the site config (get_or_generate_backup_encryption_key() in frappe/utils/backups.py).
  • F2 decrypt_backup in frappe/utils/backups.py runs gpg --yes --passphrase {passphrase} ... in a shell, so the passphrase shows in the process list.
  • F3 frappe/utils/password.py: get_encryption_key() generates and saves a new key when encryption_key is missing, and decrypt() then fails with “Encryption key is invalid! Please check site_config.json”. Restore never copies the key.
  • F4 restore_backup calls _new_site(..., force=True): on an existing site it recreates the database; a fresh name is created with a generated db_name (get_sites in frappe/utils/bench_helper.py accepts any name).
  • F5 _new_site installs only frappe, install_apps and --install-app; restore never checks that the dumped apps exist on the bench.
  • F6 restore --force only skips the downgrade question and turns a missing __Auth into a warning; it does nothing for missing apps.
  • F7 bench backup calls new_backup(force=True), which first runs delete_temp_backups() and removes files in private/backups older than keep_backups_for_hours (default 23).
  • F8 migrate’s pre_schema_updates and after_migrate call get_hooks(app_name=app) for every installed app; _load_app_hooks re-raises ImportError for an app not on the bench. remove-from-installed-apps edits only the list.
  • F9 are_emails_muted() is flags.mute_emails or cint(conf.get("mute_emails")); there is no disable_email key. The scheduler stops with pause_scheduler in the config or disable-scheduler.

Decisions:

  • The default source is the latest existing backup: a plain pull writes nothing on the server.
  • --new-backup needs the production site name typed, even with --yes, because bench backup also deletes older backups (F7). Without a terminal the name comes from --confirm-site SITE, still typed by whoever runs the command; --yes alone never takes a backup.
  • Restore goes only into a new site name; --replace first runs bench --site NAME backup --with-files locally, because restore over an existing site recreates its database (F4).
  • The keys come from the live production site_config.json, read by a Python snippet on the server that prints only the one value asked for, not from the downloaded site_config_backup.json (the design): the database password and Redis settings of production never reach the Mac, and it also works when the config backup is encrypted. --from-dir has no server and reads the site_config_backup.json in the folder.
  • Only encryption_key is written into the new site (F3); backup_encryption_key is used only as the gpg passphrase (F1), so the copy’s own backups are not encrypted with the production key.
  • The key flows through pipes only (ssh stdout into a Python snippet’s stdin); it is never on a command line, in a shell variable, on disk outside site_config.json, in the output or in the log. The snippet writes the file the way update_site_config does (indent 1, sorted keys, temp file and rename) with plain Python, so it needs no frappe import and runs under the test mocks.
  • The MariaDB root password and the Administrator password reach frappe on stdin: a small wrapper reads the secret, puts it into sys.argv inside the process and calls frappe.utils.bench_helper.main(), which is what bench itself runs from sites/. The design passed --db-root-password on the command line (accepted in 0.3 for new-site); the wrapper keeps it out of ps at no cost. The option used is --mariadb-root-password, the spelling new-site already uses.
  • Encrypted backups are decrypted here with gpg --batch --pinentry-mode loopback --passphrase-fd 0 (F2); gpg is needed only then, and a missing one stops before the download with brew install gnupg.
  • Restore --force is never passed: it does not help with missing apps (F6) and would hide a downgrade. Instead pull stops before the download when production frappe is newer than the bench’s.
  • Missing apps stop the run before the download with the bench get-app --branch B URL commands (F5, F8). The design offered benchbar app add; that command is being built separately, so pull only prints the commands for now.
  • --skip-app runs remove-from-installed-apps and prints that its doctypes and tables stay as orphans (F8). frappe cannot be skipped.
  • Tokens in an app’s git remote URL (https://user:token@host/...) are cut before the URL is shown or logged.
  • The production app list comes from the text form of list-apps (name, version, branch), parsed by its first columns; --from-dir reads the installed_apps global from the dump (tabDefaultValue, what frappe.get_installed_apps() reads), after decryption when the dump is encrypted.
  • Migrate runs when any app version differs from production, is unknown, or an app was skipped; the design ran it only when local code was newer, but an older local app with the same schema is the rare case and a needless migrate is cheap.
  • Email is muted (mute_emails), the scheduler paused (pause_scheduler and disable-scheduler) and host_name set before the first start (F9): a copy must never mail customers or poll inboxes. --keep-scheduler leaves the scheduler alone.
  • The Administrator password is reset only with ADMIN_PASSWORD or after a yes at the prompt; otherwise the production password keeps working. The design’s --admin-password-prompt flag was left out: without --yes pull asks anyway.
  • A decrypt probe after the restore counts the encrypted __Auth rows that decrypt and that fail and prints only the counts, so a wrong key shows up at once, not at the first email sync.
  • Restore and migrate run with the bench’s own Redis (fl_bench_redis_up, from site add): frappe v16 connects to it during site setup, and the code has moved on since the design.
  • The staging folder is <bench>/.benchbar/pulls/<site>-<backup timestamp>/, not <timestamp of the run>: a rerun of the same pull finds the partial files and resumes. It is mode 0700, removed file by file after a successful run, kept after a failed one.
  • rsync gets --partial --append-verify --info=progress2 only when the local rsync is 3.x; macOS ships openrsync (2.6.9 compatible), which has neither, so it gets --partial --progress, and the partial file is the basis of the next transfer. scp is used when the server has no rsync; it cannot resume.
  • A file whose local size already matches the server’s is not downloaded again.
  • The free space check wants 3x the backup (download, decrypted copy or database, extracted files) on the bench’s volume, before any transfer.
  • SSH settings come only from ~/.ssh/config (ProxyJump, keys, ports); benchbar adds BatchMode=yes under --yes or --json and one control connection (ControlMaster=auto, ControlPersist=60) closed with -O exit when the run ends. Its socket is in /tmp/benchbar-ssh.XXXXXX, because socket paths are limited to 104 bytes and bench paths can be long.
  • The host must look like a Host alias ([A-Za-z0-9@._-], not starting with -) and every remote word is single quoted, so no argument becomes an ssh option or remote shell code.
  • Remote commands are an allowlist (the site check, list-apps, git rev-parse and remote get-url, ls, wc -c, the key read, command -v rsync, and bench backup after the gate); the test fails on anything else.
  • Remote frappe runs through env/bin/python -m frappe.utils.bench_helper from sites/: non login SSH shells often lack bench on PATH.
  • pull --json streams one object per line on stdout and moves every human line to stderr; done is always the last line, written by the exit handler after a refusal or a failure too.
  • Only the source (PULL_SOURCE) and the remote bench are remembered, per bench, so benchbar pull alone repeats the last pull; no password is stored.
  • test-pull fails first when an ssh, rsync, scp, gpg or df mock is not the first one on PATH: during development a mock that was not yet executable let /usr/bin/ssh look up the host prod, which did not resolve, so nothing connected.

0.4: roadmap

  • Work happens in a second git worktree (~/dev/benchbar-work): ~/.local/bin/benchbar and the app run the checkout in ~/dev/benchbar, so a feature branch checked out there would change the CLI in daily use. That checkout stays on main and is fast forwarded after each merge.

  • Pulling a production site moved from the 0.7 plan to Ideas, as a wizard that also handles the encryption key and the app list mismatch: without those two it restores a site that cannot decrypt its passwords or fails on missing apps, and it is not scheduled yet.

  • The old 0.4 items the brief did not schedule (worker restart on Python changes, more speed sources, a runner gallery, running one scheduler event) moved to Ideas; runner import from a zip was dropped from the list because 0.3 already ships it.

  • App installs, team profiles, the lockfile and pull were scheduled as 0.6, then folded into 0.5 (Akash, 2026-09-26): they are being built at the same time as the 0.5 app work, and one bigger release is less to write up and tag than two. Patch numbers stay for fixes.

0.4: the v16 profile

Checked in frappe version-16 at 012667b and bench develop at c9d1250 (September 2026), not from memory:

  • v16 needs Python 3.14 exactly (requires-python = ">=3.14,<3.15") and Node 24 (engines.node >=24); frappe itself only warns below Node 18 at build time, so the profile’s pin is what enforces 24.
  • frappe v16 refuses no MariaDB version: check_compatible_versions in frappe/database/mariadb/setup_db.py only warns below 10.6 or above 11.8, at new-site and restore. 10.11 gets no warning at all. The docs’ 11.8 is a recommendation.
  • wkhtmltopdf stays installed on v16: pdf_generator on Print Format defaults to wkhtmltopdf and a v16 patch sets every existing format to it; Chromium is opt in per format. So the check became pdf_engine (wkhtmltopdf everywhere, plus Chromium on v16) instead of swapping one tool for the other.
  • Chromium is looked up the way frappe does (find_or_download_chromium_executable): chromium_path from common_site_config, else <bench>/chromium/chrome-mac/headless_shell. A missing one is a warning with bench setup-chrome, not a repair action: frappe downloads it on first use anyway, and its download removes <bench>/chromium first, which benchbar should not trigger on its own.
  • pkgconf and mariadb-connector-c are system dependencies of every profile: v16 pins mysqlclient==2.2.7, which does not build without them. pkg-config is only an alias of pkgconf in Homebrew, and brew list pkg-config does not follow it, so the formula is named pkgconf.
  • bench sets PKG_CONFIG_PATH itself from brew --prefix mariadb-connector-c, only for frappe 16 or newer on darwin, when it builds the env or installs an app. benchbar adds the same folder to its own exports (shell block, repair, phase 01) so a pip build outside bench finds it too. The launchd plist and the runner set no PKG_CONFIG_PATH, so nothing undoes bench’s value; the running bench needs none.
  • Missing build formulae are a warning, not a failure: a v15 bench runs without them and a v15 machine set up before 0.4 should not turn red.
  • bench 5.31 depends on uv and uses it for the env by default (BENCH_DISABLE_UV=1 turns that off), so “uv or pipx” only decides how the bench command itself is installed. uv wins when it is on PATH, as in the Frappe docs; an existing pipx bench is reported and never moved.
  • honcho 2.0.0 imports importlib.metadata, not pkg_resources; only honcho 1.x breaks on Python 3.12 or newer without setuptools.
  • The v16 CI row is a separate job running tests/test-profile-v16.sh on macOS /bin/bash 3.2: the rest of the suite asserts v15 strings on purpose, and a second full run would only repeat them.

0.4: doctor hardening

  • Full Disk Access: frappe/bench#1730 (crontab “Operation not permitted” during bench init) is still open with no fix merged as of 2026-09-25; the rollback bench offers is a prompt (click.confirm), not automatic. benchbar passes --no-backups, which skips python-crontab entirely, and keeps a doctor check plus a warning before bench init, since bench setup backups would still need it.
  • The probe is crontab -l: “no crontab for” and exit 1 is a normal answer, only “Operation not permitted” counts. Run from BenchBar the check reports ok without probing: the access that matters is the Terminal’s, not the app’s.
  • The toolchain is four checks (toolchain_node, toolchain_yarn, toolchain_pkgconfig, plus mariadb_version), one line each in doctor’s existing one line per check format. Python is not repeated: env_python already compares the env’s version with the profile.
  • Tools are resolved on the launchd PATH the agent uses, not the caller’s: the forum case “nvm’s node is not seen by bench” is exactly a node that only exists on the shell’s PATH. env/bin/node wins when bench put one there.
  • The MariaDB server version comes from the binary of the process listening on 3306 (ps), else the installed formula’s client; doctor never logs in, so it never reads the Keychain. The range is frappe’s own: below 10.6 unsupported, above 11.8 untested, both warnings.
  • honcho_setuptools imports honcho.command with the interpreter in honcho’s shebang instead of importing pkg_resources directly: honcho 2.0 does not need it, and a bare pkg_resources probe would warn on every healthy Python 3.12 venv. The repair installs setuptools into that venv (pipx or uv), never into the bench env, and adopt skips it like honcho_install.
  • fork_safety reads the plist only: the variables reach honcho and every worker through the agent’s environment, and Procfile.lean does not need to repeat them. benchfg sets them itself.
  • orphans counts listeners on the bench’s ports only when neither the agent nor a honcho runs the bench, so a bench started with benchfg is not reported.
  • Found while testing: doctor --json with a failing check went through the ERR trap on its way to exit 1 and printed [FAIL] Last command failed on stdout after the JSON. The dispatcher now exits 1 itself, and test-json parses a failing report.

0.4: several benches

  • Per bench settings live in the checkout, .benchbar/benches/<name>.env, not in the bench: benchbar writes only its service files into a bench, and a folder name is already the agent’s unique key (com.benchbar.<name>), so it is a safe file name too.
  • Keys that were per bench but global before 0.4 (PROFILE, SITE_NAME, AUTOSTART, HONCHO_BIN, APP_BUNDLE, APPS) are read from state.env for the default bench until its own file has them, and moved by the next writing command. Doctor and status stay read only, and an upgrade needs no migration step.
  • When the default bench changes, its old settings move into its own file first, so AUTOSTART=off of the old default never leaks to the new one.
  • A second install, adopt or service keeps the default bench (--make-default changes it): “set up a v16 bench to try something” must not change what benchup starts.
  • A bench with no stored profile gets the one whose Frappe branch matches apps/frappe/frappe/__init__.py (__version__ = "16.x" is v16-lts), before the default profile: doctor on a freshly found v16 bench would otherwise judge it by v15 rules.
  • The shell block’s PATH lines follow the default bench’s profile, and every bench renders the block the same way. Before, a v16 bench would have put Python 3.14 and Node 24 first in the user’s shell, and the two benches’ doctors would have called each other’s block outdated forever.
  • Phase 00 writes the shell block only when it is missing: it knows a profile but not which bench is the default. service and repair keep it current.
  • honcho and socketio are matched by their working folder (lsof -a -p PID -d cwd): both start with a relative path (honcho start -f Procfile.lean, node apps/frappe/socketio.js), so the command line is identical in every bench. A process whose folder cannot be read is still matched, the pre 0.4 behaviour, since it is usually exiting.
  • The runner (template v3) clears only its own bench’s stale socketio at start, with the same working folder test; before, starting a second bench killed the first bench’s socketio.
  • The launchctl mock now stops only the agent’s own honcho, so the tests can run two benches at once.

0.4: Codex review findings on PRs 10 to 12

  • Bench paths are resolved once, in fl_abs_path (symlinks, . and ..), so state, ports, the default bench and agents see one spelling per bench; a stored BENCH_DIR from before is compared through fl_same_path. A local review before pushing found a symlinked spelling clashing with its own ports and getting its own state file.
  • The per bench state file is <name>-<8 hex of the canonical path>.env (cksum of pwd -P, so every spelling of a bench maps to one file; a plain <name>.env from the first 0.4 builds is read and renamed on the next write, by the default bench, or by a bench no other known bench shares the folder name with): ~/frappe-bench and ~/dev/frappe-bench are both detected candidates and would otherwise share one file. The agent label still uses the folder name only; two benches with the same folder name remain unsupported, as before 0.4.
  • A bench installed with uv is looked up in uv tool dir --bin, which follows UV_TOOL_BIN_DIR and XDG_BIN_HOME, instead of assuming ~/.local/bin.
  • Phase 01 records the default bench only after the site is verified, so a failed install --make-default leaves benchup pointing at the bench that worked.
  • With --make-default, the shell block is rendered for the bench being set up even in a dry run, so the plan shows the PATH change the real run makes.
  • The orphans finding (a machine wide pgrep for honcho) was already fixed by the working folder match in the multi bench PR.

0.4: port blocks

  • bench init already chooses ports: make_ports in bench’s config/common_site_config.py takes max+1 over the benches in the same parent folder. benchbar keeps that and only moves a bench when it clashes with another bench it knows (anywhere under ~ or ~/dev, or registered by an agent) or a port is taken by a listener.
  • The block is written with frappe’s own bench set-config -g (-p for the two integer ports) and the Redis files with bench setup redis, not by editing common_site_config.json or config/redis_*.conf: the AGENTS.md rule “never edit sites/” holds, and the files look exactly as bench would write them. The config is backed up first.
  • redis_socketio is written equal to redis_cache and no 12000 + n port is reserved: bench keeps the two equal “for backward compatibility”, and neither frappe v15 nor v16 connects to it (v16 realtime uses redis_queue).
  • Only a newcomer moves: the default bench (or the one about to become it) never changes ports on its own, and a bench moves only when it clashes with an established bench, one with a benchbar agent or the default. Found by the tests: without that rule, adopting the first bench on a Mac with two more unregistered benches moved the first bench off 8000.
  • Moving ports is the port_block action of the same plan as the service files (a check that exists only while a plan moves ports, never in doctor): one confirmation covers it, and a cancelled adopt really changed nothing. Codex found the first version asking separately and moving the ports before the plan could still be cancelled.
  • Listener ownership has two strictness levels. For the bench’s current ports an unreadable working folder counts as its own, so it never moves on a guess; for a block it does not use yet, only a listener known to run inside the bench is its own (Codex found the first version trusting every listener on the current ports, which could then be killed by the runner’s cleanup).
  • service --port-offset validates the block before remembering the bench, so a refused block never makes it the default (Codex).
  • The action rewrites Procfile.lean and the runner itself after the move: they carry the ports, and when the plan was made they may have been current and therefore not in it.
  • --port-offset checks the block for conflicts even when it is the current one, so two benches already sharing block 0 are not confirmed by asking for block 0 (Codex).
  • A newcomer also moves when a foreign process listens on its own ports, not only when a known bench has them (Codex). “Foreign” is decided by working folder: honcho starts redis, web and socketio inside the bench, and an unreadable folder counts as the bench’s own, so nothing moves on a guess. The default bench is never moved for a listener either; it gets a warning.
  • The port clash check now also reports a bench that is only configured with the same ports, since only one of them can run; benchbar up still asks only about a running one, so the app (which cannot answer) can start either bench.
  • MariaDB ranges live in the profile (mariadb_min, mariadb_max). v16 uses frappe’s own bounds, 10.6 to 11.8. v15’s code warns above 10.8, but the v15 docs and this installer use 10.11, so v15’s range ends at 10.11; the check is a warning either way, as in frappe.
  • The toolchain’s MariaDB check was named mariadb_version before 0.4 shipped, to match the brief; nothing released used the other name.

0.4: the MariaDB decision

  • Option (a), decided with Akash on 2026-09-26 (“let’s struggle and find out”): the v16 bench uses the mariadb@10.11 server the v15 bench already runs on 3306. frappe v16 accepts it without a warning, and it keeps one server, one data folder and one Keychain password. Option (b) (a second mariadb@11.8 service on 3307) stays possible later if v16 turns out to need 11.8.
  • The supported path: when a MariaDB server runs on 3306 and its version is inside the profile’s range, phases 00 and 01 and every command use that server’s formula instead of the profile’s (named from the server binary’s opt/<formula> or Cellar/<formula> path, read with ps), and the formula is stored per bench (MARIADB_FORMULA) by phase 01 or the first service, adopt or install. Daily commands read only the stored value: test-process guards that down never touches port 3306, and a detection in every command’s context broke that guarantee. So a v16 install on this Mac never installs or starts mariadb@11.8, and a machine without MariaDB still gets the profile’s formula. mariadb_version in doctor checks the range per profile.

0.4: sites and the scheduler

  • The default site is the one benchbar remembers per bench; site default also runs bench use, so currentsite.txt agrees, and rewrites the runner, whose ping embeds the site. benchup’s wait, status and the app all use it; a second site never changes it on its own.
  • site add creates a new site only, on the same MariaDB server as the bench (option (a) of the MariaDB decision), with the Keychain password; it never drops or overwrites a site, and there is no site drop in 0.4.
  • Apps for a new site come only from apps/ (--bundle or --apps); a missing app is refused with the bench get-app command. Fetching apps belongs to the 0.6 app commands.
  • A site’s ping_code is only tried when something listens on the web port, so list --json on stopped benches costs no three second timeouts per site.
  • site hosts asks once for all missing lines and uses one sudo prompt, through the existing marker block logic.
  • site list takes no lock (it is read only); add, default and hosts take the checkout’s lock like every other writing command.
  • The scheduler line is an optional template line: a line that is only a token rendering to nothing is dropped. With the scheduler off, Procfile.lean renders byte for byte as before 0.4 (hash 92548cf35913 checked against the real bench), so no existing bench sees an outdated Procfile. The file’s comment still says “no schedule” when it is on: changing it would change every existing Procfile’s hash.
  • The scheduler choice is per bench state (SCHEDULER in its state file), so repair renders the same Procfile and never undoes it; status --json reports scheduler.

0.4: found on a real Mac (v16 bench, 2026-09-26)

  • The MariaDB root password of a bench set up before 0.3 was never in the Keychain, so phase 00 stopped with exit 2 as designed; Akash reset it with the documented recipe (databases kept, the v15 bench stopped around it with benchbar down and brought back with benchbar up), and the next run verified the new password and saved it to the Keychain.
  • bench init in ~/dev/v16-bench chose 8001, 9001, 11001, 13001 (and file watcher 6788) by itself, as a sibling of ~/dev/frappe-bench; benchbar’s port check agreed and moved nothing.
  • On v16, bench --site v16dev install-app erpnext failed with “Error 61 connecting to 127.0.0.1:11001. Connection refused”: frappe v16 connects to the bench’s Redis during site setup, and a fresh bench has none running. Phase 01 and site add now start the bench’s own Redis servers from config/redis_*.conf when nothing listens on their ports, and stop only the ones they started: the queue with shutdown save, so jobs an app install enqueued wait for the first worker, the cache with nosave (Codex). Redis startup errors go to the run log, whose path is now exported to the phase scripts. A listener on the bench’s Redis ports is used only when it runs inside the bench (working folder); another process there, such as another bench on the same default ports, stops the setup with the --port-offset fix, so this bench’s cache and install jobs never land in someone else’s Redis (Codex). A running bench’s Redis is used as it is.

0.4: the v16 bench on a real Mac, verified (2026-09-26, macOS 27, Apple Silicon)

  • Installed ~/dev/v16-bench (profile v16-lts, bundle minimal, site v16dev) with benchbar install next to the running v15 bench: frappe and erpnext version-16, Python 3.14.7, Node 24.21.0, the existing mariadb@10.11 server (10.11.19, option (a)), ports 8001, 9001, 11001, 13001. The profile is now supported.
  • Both benches up at once: macdev on 8000 and v16dev on 8001 each answer /api/method/ping with 200 (Host header); benchbar list shows both, frappe-bench as the default; the shell block still puts python@3.11 and node@20 first.
  • benchbar down on v16 left v15 running (ping 200), and down on v15 left v16 running (ping 200).
  • up on v15 then crashed v16 once: the v15 bench still had runner template v2 (Akash has not run repair since 0.4’s runner v3), whose start up cleanup kills every socketio.js on the Mac; honcho in v16 exited with 241, launchd restarted it after about 20 seconds and it answered 200 again. This is the bug the working folder match fixes; benchbar repair on the v15 bench installs runner v3. The crash guard counted one start and did not trip.
  • Doctor: v15 30 ok, 2 warn (CleanMyMac, the outdated runner). v16 28 ok, 4 warn: CleanMyMac, the missing v16dev hosts line (sudo, pending Akash), Chromium not downloaded yet (expected: frappe fetches it on first chrome PDF), and the python formula check, which was wrong (next section).
  • The resumed install ran with --yes and no terminal: phase 01 installed erpnext on the existing site with the setup Redis, the service files were written and the agent loaded with the bench stopped, and the hosts step was skipped with its manual command. The run ended with the verify warnings and a misleading [FAIL] Last command failed ... brew tab from the ERR trap; the trap now stays quiet when the last command is a finished verify pass.
  • Then, with Akash: benchbar repair on the v15 bench (runner v3; doctor 31 ok, 1 warn), benchbar site add v16two (his Administrator password, the hosts line with sudo) and benchbar site hosts for v16dev; macdev:8000, v16dev:8001 and v16two:8001 all answer ping 200 by hostname, and the hosts block holds exactly one line per v16 site.
  • frappe v16’s bench new-site calls --no-mariadb-socket deprecated in favour of --mariadb-user-host-login-scope='%'; v16 profiles now pass the new flag, v15 keeps the old one, which it still needs.

0.4: python formula check

  • python_leaves asks brew list --formula --installed-on-request, not brew leaves: on the real Mac python@3.14 was installed on request (and brew tab set it again), but pipx, uv and ollama depend on it, and brew leaves hides every formula another formula depends on, so the check could never pass. What protects a formula from brew autoremove is “installed on request”, which is what the check now reads.

0.5: benchbar mcp

  • The server is stdlib only Python 3.9 (lib/frappe-local/mcp.py), run with whatever python3 is on PATH: the Command Line Tools’ 3.9 is enough, so CLI only users need nothing new. The Swift MCP SDK would tie it to the app.
  • Every tool shells out to benchbar ... --json and returns its output as text and as structuredContent; the server holds no bench logic. Actions (up, down, restart) return their text output plus a fresh status --json, so an agent sees the outcome without a second call.
  • No repair, install, service, site add or anything with sudo behind a tool: those change files or need a password, and belong in a terminal with the user. Read tools carry readOnlyHint: true, actions false.
  • stdin of every CLI call is closed, so a question (a port clash on up) is answered no instead of hanging the session; one bad message gets a JSON-RPC error and the session goes on.
  • Protocol versions 2025-06-18, 2025-03-26 and 2024-11-05 are accepted as asked; anything else gets 2025-06-18. The transport is newline delimited JSON on stdio, as the spec says for stdio.
  • logs --json exists for benchbar_logs_tail: the brief keeps the server free of logic, so the process filter (honcho’s HH:MM:SS name.N | prefix, with unprefixed lines such as tracebacks kept under their process) lives in the CLI. fl_json_escape now drops control characters, since log lines may carry terminal colors.

0.5: repair —json

  • The events go to fd 3 (the command’s stdout) and every human line to the run’s log: the engine prints with the same helpers as always, so the text and the JSON can never disagree, and the log keeps the full story for a failed step.
  • Without --yes a JSON run applies nothing: the confirmation would be invisible, so stdin is closed and the plan is answered no (plan, then done with 1). The app shows the plan from --dry-run --json, asks, and then passes --yes.
  • sudo actions are marked in the plan ("sudo": true): without a terminal they are skipped with their manual command as the message, which the app shows as “run in Terminal”.
  • A step’s message is its own last [FAIL] line, else its [WARN] line, from the log lines written during the step; the step summary line is ignored.

0.5: local review before the pull request

  • bench new-site gets both passwords on stdin, not in its arguments: the bench’s own Python reads them and swaps @secret0@, @secret1@ in its argv before handing over to frappe’s bench_helper, so the passwords never appear in ps or the run log.
  • Every git ls-remote puts -- before the repository, and a profile or lockfile entry whose repo, branch, commit or name starts with - is refused: a team file must not be able to pass git an option.
  • app list --json on a bench without sites reads its cache with grep ... || true, so pipefail does not end the run.
  • benchbar mcp answers a message that is not an object with -32600 and parameters that are not objects with -32602, and keeps the session going.
  • The app’s repair sheet applies stream events on the main actor in order, through one AsyncStream, so a late step event can never overwrite a failed run.
  • Codex on the pull request: app update refuses to run when any site’s list-apps fails, since a stale list could skip a site’s backup or migrate. pull into a running bench sets pause_scheduler in common_site_config.json before the restore and removes it once the copy has its own pause_scheduler and mute_emails, so the bench’s scheduler never sees the production copy unguarded; a bench already paused by hand is left alone, and a pull that stops halfway leaves the bench paused and says how to resume. The test runner kills a timed out test’s whole process tree, collected before anything dies.

0.5: found on the real Mac before the release

  • app add URL for an app config/apps.tsv knows follows that branch when the repository has it (Raven added by its URL came in on develop, the repository default, and doctor then warned that it was off main). A URL for an unknown app still takes the default branch; --branch always wins.
  • Raven moved to github.com/frappe/raven; the registry points there (the old URL redirects).
  • The app reads app list with the 90 second limit, not 20: at app start it runs next to status and doctor for every bench, and it makes several git calls per app.

The easy install run (v0.3)

Setup and environment

  • Work happened in a Linux cloud container, not on a Mac: there is no Xcode, launchctl, codesign or hdiutil here. Everything Mac only (the app build, the Swift tests, release-local.sh) is exercised on GitHub Actions macOS runners; the CLI is tested locally under mocks and in CI under macOS /bin/bash 3.2.
  • Branch feat/easy-install is pushed and a draft pull request opened against main: the brief for this environment allows exactly that remote write and nothing else (no tags, no releases, no settings).
  • Conflict with AGENTS.md: it says never write LaunchAgents or Procfile yourself; this run changes the code that generates them (adopt, install). The brief wins; nothing was loaded on a real machine.
  • bash 3.2 could not be built locally: the proxy blocks the GNU mirrors and only the project repository is reachable on GitHub. CI’s macOS /bin/bash 3.2 is the bash 3.2 check.
  • The mocked test suite was made to run on Linux too (GNU stat and sed, a uname mock, a fake uid): it gives fast local feedback here and a quick Linux job in CI, while the macOS job stays the one that counts.

Stage 1: harden

  • benchbar report masks by key name (password, secret, token, key, api, auth, case insensitive, anywhere in the key) rather than by value shape: values are unpredictable, key names are not. Over masking (for example encryption_key, Authorization: headers) is accepted.
  • Header style Key: value masking runs to the end of the line, or to the next quote, comma or brace inside a one line JSON document, so a bearer token after Authorization: is gone but a JSON file is not wiped.
  • The username and hostname are replaced by simple string substitution (any occurrence, no word boundaries, names shorter than three characters skipped): BSD sed has no \b, and replacing too much is safer than too little.
  • Site config files are never copied; site-config-keys.txt lists their key names so a helper can still see whether, say, developer_mode or db_host is set.
  • REDACTIONS.txt reports what was replaced per file, so the sender can check the bundle before attaching it.
  • report works without a bench (versions only) so a tester whose install failed early can still send something.
  • The CLI job in CI runs on macos-latest, the app jobs on macos-26: the project uses Xcode 26 build settings (default MainActor isolation) that the Xcode on macos-latest does not have. The old release workflow already targeted macos-26.
  • CI also runs the CLI tests on ubuntu-latest: it finishes in a minute and catches most breakage before the macOS runner is even assigned.
  • The old macos-release.yml failed on every push with “workflow file issue”: its job level env read ${{ runner.temp }}, and the runner context is not available there. The merged release.yml uses $RUNNER_TEMP in steps.
  • release-local.sh is the ad hoc path of the release and is also what CI runs on every pull request to produce a downloadable app; macos-release.sh stays the Developer ID path.
  • macos-build.sh takes BENCHBAR_VERSION and BENCHBAR_BUILD and passes them to xcodebuild instead of editing project.yml in CI, so the checkout stays clean and the tag decides the version.

Stage 2: the manual steps

  • The MariaDB root password goes into the macOS Keychain (service benchbar-mariadb, account root) through security add-generic-password -U: the only non interactive way to write a Keychain item. The password is an argument of that one call; everywhere else it travels in MYSQL_PWD, never on a command line. The mocked test asserts that no call log ever contains it.
  • Order of password sources: MARIADB_ROOT_PASSWORD, then the Keychain, then a prompt. A password that verifies against the server is always saved, whatever its source, so the next run needs no source at all.
  • The secure installation is plain SQL (DELETE FROM mysql.global_priv for anonymous and remote root, DROP DATABASE test, ALTER USER ... IDENTIFIED VIA mysql_native_password): mariadb-secure-installation is interactive and the answer table in the old README was the whole reason it was manual. Native password auth is kept on purpose: Frappe needs it.
  • Phase 00 exits 2 only when MariaDB already has a root password that no source knows and there is no terminal to ask; every other former manual step is automated or skipped with a warning, so benchbar install --yes normally finishes in one run.
  • A generated password is 24 letters and digits: safe in SQL, in a shell and on bench’s own --mariadb-root-password argument, which cannot be avoided.
  • The pinned wkhtmltopdf is 0.12.6-2 from github.com/wkhtmltopdf/packaging, sha256 81a66b77...94f8, computed from a download made during this run. The package’s binary is x86_64 only (checked by reading the Mach-O header inside the nested tarball: cputype 0x01000007, no arm64 slice), so on Apple Silicon Rosetta 2 is required and offered first; declining it or the download only costs PDFs, and the run continues.
  • Rosetta is detected with arch -x86_64 /usr/bin/true, not by looking for the daemon: it is what the binary will actually need.
  • Downloads land in .benchbar/downloads/ and are reused when their checksum matches, so a re-run after a failed installer does not fetch 50 MB again.
  • One sudo prompt per run: benchbar install looks ahead (patched wkhtmltopdf present? hosts line present?) and runs sudo -v with a keepalive before phase 00; the phase scripts inherit the session through FL_SUDO_SESSION and sudo’s own per terminal timestamp. Standalone 00-mac-system-deps.sh asks itself, only when it gets to the package.
  • The /etc/hosts line sits inside # >>> benchbar >>> markers. A missing block is appended with sudo tee -a; an existing block gets the new line inside it through a temp copy and sudo cp, after fl_backup_file. Rewriting the whole file is limited to the second case.
  • The utf8mb4 drop-in and wkhtmltopdf became doctor checks with repair actions (mariadb_utf8, wkhtmltopdf_install): the same code serves phase 00 and benchbar repair, and the tester guide can say “run doctor”.
  • benchbar adopt is benchbar service with a positional path, validation and an explicit safety statement, run through the same engine restricted to the service group: that group has no migrate, build, update or env rebuild by construction. It remembers the bench only after the plan was applied or found unchanged, so a cancelled adopt never makes that bench the default (a Codex review finding).
  • benchbar mariadb-password asks before printing unless --yes: a terminal print is a deliberate act, and the app or a script can pass --yes.
  • The mocks gained state (mariadb_root_pw, a keychain folder, wkhtml_installed, rosetta, sudo_refused, download_payload) instead of environment switches, so a test reads like a machine’s history.

Stage 3: install.sh

  • install.sh is self contained bash 3.2 with its own small output helpers: it runs before the repository exists on the machine, so it cannot source lib/.
  • Prompts read and write /dev/tty so curl | bash can ask; without a usable tty every question takes its default (--yes does the same). Defaults are “yes” for the things the person asked for (Homebrew installer, adopt) and “no” for anything destructive (removing agents or the checkout on --uninstall, where --yes flips them to yes because that is what a non interactive uninstall means).
  • The CLI is cloned into ~/.local/share/benchbar and updated with git pull --ff-only; --version pins the app only. Pinning the CLI to a tag would leave a detached checkout that the next git pull cannot update.
  • benchbar and frappe-mac are symlinks into the checkout, so git pull is the whole CLI update; a regular file at either path is left alone with a warning.
  • The PATH block in ~/.zshrc uses its own markers (# >>> benchbar-path >>>): the CLI’s # >>> benchbar >>> block is regenerated by repair from a template and only exists after install or adopt.
  • The app zip is checked against the release’s SHA256SUMS; a release without that file is refused, not installed unchecked. The installed version is read from Info.plist with awk (no plutil, so the same code runs in the Linux tests).
  • The app is unpacked with ditto -x -k and copied with ditto: the tools macOS uses for bundles. Because curl writes the download without the com.apple.quarantine attribute, Gatekeeper never assesses the app on first open. That is the reason the one liner is the recommended install for an ad hoc signed build; the DMG path gets the “Open Anyway” steps in README instead.
  • A found bench is offered to benchbar adopt, which keeps its own plan and question; install.sh only passes --yes on when it was given --yes. benchbar install is never started without a terminal: it needs the Administrator password.
  • On an Intel Mac the app is skipped with a warning (it is built for arm64 only); the CLI installs.
  • --uninstall removes only what the installer made (app, links, PATH block), offers benchbar uninstall-service per com.benchbar.* agent and the removal of the checkout, and never lists or touches a bench folder.
  • The tests drive install.sh against a fake HOME with a git mock that copies this checkout, a curl mock that serves a release JSON, zip and SHA256SUMS from the mock state, and a ditto mock on top of unzip; scenarios: no release, fresh, unchanged rerun, upgrade with a running app, version pin, tampered zip, --no-app, --app-only, --dry-run, adopt offer, Intel, --uninstall twice.

Stage 4: releases

  • One release.yml replaces macos-release.yml: a check job reads the secrets and picks the path; the build job runs scripts/macos-release.sh (Developer ID, notarized, Sparkle, cask) or scripts/release-local.sh (ad hoc). Everything after the build is shared: notes from the CHANGELOG section, a workflow artifact, a draft release, uploads with --clobber so a re-run replaces files.
  • The version comes from the tag and is passed to xcodebuild (BENCHBAR_VERSION, BENCHBAR_BUILD = commit count) instead of editing project.yml; the workflow still refuses a tag that disagrees with MARKETING_VERSION or has no CHANGELOG section, so the source of truth stays in the repo.
  • Releases are created as drafts: the person publishes after looking at the files. workflow_dispatch with a version builds the artifact without touching releases, for a rehearsal.
  • Sparkle stays out of the ad hoc path: unsigned updates would defeat its purpose, and the ad hoc app contains no update code.
  • The DMG carries an Applications symlink and is not signed on the ad hoc path (signing a DMG ad hoc buys nothing); Gatekeeper’s warning is expected and documented with the macOS 15 “Open Anyway” steps in README.

Found on the macOS runners

  • A generated password reads a bounded 4 KB of /dev/urandom: BSD tr on an endless stream never exits when head closes the pipe and SIGPIPE is ignored, as it is under GitHub Actions. The CLI job hung on that for 45 minutes before the cause was found.
  • The sudo keepalive owns no stdio and sleeps in five second slices: with the caller’s stdout inherited, every $(...) capture of a run waited for its sleep 50, and on macOS never returned. The tests capture output, so this showed up only in CI.
  • The git mock’s pass through to the real git picks the first git on PATH outside tests/mocks and refuses to exec itself: on macOS /bin/bash 3.2, command -v -p git still returned the mock, which then exec’d itself in a loop at full CPU. This was the hang behind three timed out macOS CLI jobs; the Linux job never saw it. The test runner now kills a test after ten minutes and prints its process tree, which is how this was found.

Review findings on the pull request

  • fl_wkhtmltopdf_ensure returns 2 for a deliberate skip (Rosetta or the package declined, no sudo) and 1 for an error; the repair action reports a skip as skipped, and the engine’s verify pass ignores the optional actions wkhtmltopdf_install and redis_stop, so a bench install without PDFs exits 0.

  • The report’s assignment style masking takes a quoted value whole (key="secret", key='secret'): the unquoted form stopped at the quote and left the secret in place.

  • install.sh --app-only skips the Command Line Tools and Homebrew checks: the prebuilt app needs only curl, shasum and ditto; macOS and the architecture are still checked.

  • The installer’s PATH block is generated from the configured bin folder (BENCHBAR_BIN_DIR), written as $HOME/... when it lives under the home folder.

  • A refused up-front sudo -v sets FL_SUDO_REFUSED=1, exported to the phase scripts, so an install asks for sudo exactly once whatever the answer and skips every sudo step with a message after a refusal.

  • A hosts line that cannot be written because sudo was refused is a skipped step with the manual command printed, and hosts_entry joins the optional actions, so a run the user chose to keep sudo free still ends with exit 0 and warnings.

  • The repair engine resets FL_STEP_RESULT before each action and reports it, so a skipped wkhtmltopdf or hosts step prints skipped, the documented step status, not done.

  • The report’s assignment style masking also treats ? as a key boundary, so a credential in a URL query string (?api_key=...&x=1) is masked like any other.

  • release-local.sh no longer re-signs the app: macos-build.sh already signed it ad hoc with the Hardened Runtime and the entitlements, and a plain codesign --force -s - dropped both. The script verifies the signature and the runtime flag instead.

  • Phase 01 no longer requires wkhtmltopdf: a bench without it works, only PDF printing does not, and phase 00 already said so when the package was declined.

  • The report’s quoted value patterns accept backslash escaped characters inside the quotes, so "api_key":"abc\"tail" is masked whole.

  • The utf8mb4 doctor check also requires the !includedir line in my.cnf (a missing my.cnf counts as missing). The live character_set_server query that 0.3.0 added was removed in 0.3.1: it read the root password from the Keychain, and doctor is read only and runs on a timer in the app, so it must never touch the Keychain. A test asserts that doctor makes no security call.

  • adopt --dry-run never touched state (fl_state_set is a no-op under FL_DRY_RUN); a test now proves it.

  • The report also masks Python repr mappings ('password': 'x'): worker logs print dicts that way.

  • adopt runs the engine with FL_ENGINE_SKIP_ACTIONS=honcho_install: a missing honcho is reported with pipx install honcho or benchbar repair as the fix, and nothing is ever installed into the bench’s env/ by adopt.

  • Phase 00 tells a failed wkhtmltopdf install (download, checksum, installer) apart from a deliberate skip: it prints FAILED and a manual step, but still exits 0, since PDFs are optional and the bench can be created.

  • A generated MariaDB password is written to the Keychain before it is applied to the server; when the Keychain refuses (locked), MariaDB is left unchanged and the run stops with the fix, so no password ever exists only in a dying process.

  • Phase 01 exits 2, the documented “root password unknown” code, when a fresh site needs the MariaDB password and no source has it; benchbar install reports it as a pending manual step like phase 00 does.

  • wkhtmltopdf detection prefers the package binary at /usr/local/bin/wkhtmltopdf when it is the patched build, and warns with brew uninstall wkhtmltopdf when an unpatched build earlier on PATH would shadow it: Frappe finds the binary through PATH, and the launchd PATH puts Homebrew’s bin before /usr/local/bin.

  • A missing my.cnf counts as a missing !includedir in the utf8mb4 check: without it the drop-in folder is never read.

Found on a real Mac (macOS 27, Apple Silicon)

  • Verified read only on the real bench: install.sh --dry-run and --dry-run --uninstall write nothing; doctor and report --print run clean (no home path, username, hostname or site config value in the report); adopt on an already adopted bench is a no-op that only remembers the path. release-local.sh builds, passes the 114 Swift tests, and shasum -a 256 -c dist/SHA256SUMS passes; the DMG (UDZO, read-only, Applications symlink) mounts and the unpacked app carries a valid ad hoc signature. Every macOS flag the mocks assume was checked against the man pages: security add-generic-password -U, arch -x86_64 prog, softwareupdate --install-rosetta --agree-to-license, installer -pkg X -target /, ditto -x -k and -c -k --keepParent, hdiutil create -format UDZO, sw_vers -productVersion (the man page lists --productVersion; the single dash form still works and is what macOS 12 and older knew).
  • benchbar report replaces every name the Mac goes by, not only hostname: on the test Mac hostname returned the router’s name (Mac.lan) while the Bonjour name from scutil --get LocalHostName was the identifying one. The computer name (scutil --get ComputerName) is included too, and the sed patterns built from those names, $HOME and the username are escaped, since a computer name can hold brackets or a dot.
  • install.sh --uninstall --dry-run ended with “BenchBar removed”: the uninstall summary has its own dry-run branch now, and quit_app prints what it would do instead of hiding the line behind the redirected osascript.
  • The Gatekeeper dialog for a quarantined ad hoc build reads “BenchBar” Not Opened, Apple could not verify it is free of malware, with Done and Move to Trash (Move to Bin in British English) where Move to Trash is the highlighted default. README says so. The System Settings steps were not clicked: Open Anyway changes a security setting, which the brief reserves for the user.
  • fl_password_generate and the sudo keepalive were exercised on macOS /bin/bash 3.2: the generator returns 24 characters at once, also with SIGPIPE ignored, and 4 KB of urandom never yields fewer than about 900 usable characters; a captured fl_sudo_begin returns immediately and its keepalive exits within one five second slice of the parent leaving.

CI speed

Measured on five PR runs (September 2026): 11 to 14 minutes wall clock, all of it the macOS CLI job (10.7 to 13.5 min; its test step 605 to 763s). Linux CLI took 7.5 to 8.8 min, the app job 1.4 to 1.7 min, the release bundle 1.2 to 1.8 min after it, v16 0.3 to 0.7 min, shellcheck 22 to 32s, brew install 2 to 4s (bottle cache hits), DerivedData restores from its restore key in about 5s.

  • tests/run-tests.sh runs the test files in a bash 3.2 worker pool, PARALLEL at a time (default: the CPU count), output buffered per test and printed whole in list order, the ten minute deadline kept per test (polled every 0.2s instead of 1s), and every failure reported at the end instead of stopping at the first: the files were already independent (each builds its own HOME and mock state under mktemp; the real ps fallback only reads, the mocks only read from the checkout). Local, this Mac: 512s one at a time, 233s with 3, 113s with 10.
  • The deadline’s process dump lists only the stuck test’s process tree: with tests side by side, the old ps | grep tests/ would name every running test.
  • The list is ordered longest first by measured time and a test file missing from it fails the run: the pool then never waits on a slow test started last, and a new file cannot be silently skipped.
  • SHARD=i/n takes every nth test of that list: shards are balanced by construction (local, 3 workers each: 75s, 65s, 70s).
  • The macOS CLI job is a matrix of three shards, each running its share 3 at a time (macos-latest has 3 CPUs): the 11 minute test step should drop to about 2 minutes per shard. Three shards plus the app job stay under the five concurrent macOS jobs of the free plan.
  • shellcheck runs as one more pool job in shard 1 (and on Linux), over the full file list the CI step had (scripts/*.sh and install.sh included): it overlaps the tests instead of adding 25s in front of them, and needs no extra runner.
  • The separate v16 job is gone: run-tests.sh has run test-profile-v16.sh since 4d666ee, so it ran twice per PR.
  • The release bundle is built in the app job (release-local.sh --skip-tests replaces the plain Release build step, then the Swift tests run on the warm DerivedData): its Release build is a superset of the old unsigned build (it also signs ad hoc and verifies), and it saves a second runner, Xcode selection, XcodeGen install, cache restore and a full Release build (about 1.5 min on the critical path), while still producing the bundle on every PR.
  • The Linux job is gone (Akash, 0.5 pull request): BenchBar only runs on macOS, the macOS shards already run the suite on the /bin/bash 3.2 users have, and the Linux job was the slowest one left (4.5 minutes) while only ever finding GNU tool differences no user hits. The suite still aims to run on Linux for a Linux dev container, without a CI guarantee.
  • brew install is skipped when the tool is already on the runner, and the Homebrew and DerivedData caches stay as they were: both already hit (restore key for DerivedData, which is expected when sources change).
  • Estimate after: about 3 to 4 minutes wall clock per PR, bounded by the slowest shard. First run on the 0.5 pull request: shards 2m28s to 3m50s, the app job 1m50s.