Where we are, in one slide

  • bioconductor.org has been served by the new stack since 2026-09-28. It’s a Cloudflare Worker in front of R2 object storage. BiocManager::install() works unchanged: 8 of 8 acceptance checks pass for release 3.23 and devel 3.24.
  • Why we moved: crawlers overwhelmed one EBS-backed VM, and nobody could see them.
  • Cost: about $5,000 a month of AWS is being retired. The new stack runs at roughly $240–500 a month (an estimate at list prices).
  • Packages are still built by BBS and r-universe, and copied hourly from master. The registry built on r-universe is running, but it doesn’t serve bioconductor.org yet.
  • Packages and the website are independent: each publishes on its own path, and site changes go through PRs with previews and one-line rollback.

One day on bioconductor.org

2026-09-30, UTC · from the new edge’s request logs

One day of bioconductor.org traffic, 2026-09-30 UTC: 5.8 million requests, 8.1 TB served, 215 countries, about 21,000 R installations. Requests by client type: R and package clients 33%, other automation 33%, browser-like 21%, search and AI crawlers 8%, mirrors 3%, CI and other 3%. Requests per hour peak at 09:00 UTC with 388 thousand. R installations by operating system: Linux 12,347, Windows 6,303, macOS 2,650.

One machine: 162,163 downloads of Bioconductor 3.19, 40% of R installs that day.

Build reports: 22% of all requests, the site’s other front door.

The legacy estate

Where we started

The legacy estate: git.bioconductor.org feeds staging, which rebuilds the site hourly with Nanoc on an out-of-support OS and rsyncs it to master, one EC2 VM running Apache on an EBS volume with Solr and checkResults. BBS build machines push reports directly to master; BBS tarballs and r-universe binaries reach master through core team propagation. CloudFront sits in front of master. About $5,000 a month in AWS.

The crisis: crawlers against one disk

Crawlers requesting unique URLs and large files miss the CloudFront cache and reach one VM whose EBS volume runs out of IOPS; that one disk sits behind pages, installs and build reports. The course-materials directory holds 711 MB of video against 2 MB of HTML, about 400 to 1.

Why not just fix the VM?

  • One VM, one disk, behind everything. Pages, installs, mirrors and build reports share one failure domain.
  • Nobody could see the traffic. Request logs fed statistics jobs; nobody read them request by request. The download statistics have never filtered bots: the code that would has been commented out.
  • The build box is fragile. Its OS is out of support, it does a full rebuild every hour, and its daemons run with no supervisor.
  • Most of the ~$5,000 a month is egress, and sending tarballs to the world is the whole job.
  • Nothing was written down in a place a newcomer could find it.

The decision

Move serving to object storage plus edge compute, and replace the legacy pieces one at a time behind the live site (a strangler migration), with every choice recorded as an ADR.

Serving: how a request is answered

The new stack

Clients reach Cloudflare's WAF and edge cache, then the Worker (code in bioc-edge), which applies the route table, symlink map, redirects and 404 caching and logs every request. It reads R2: the immutable site build from bioc-website at site/<sha> first, then the mirror of master. Only /packages/stats is passed through to master. Logpush writes every request to Google Cloud Storage; daily probes check health and parity with master.

Publishing: how content gets there

Two publishing paths into R2. Packages, hourly: BBS source tarballs and r-universe binaries reach master's docroot through core team propagation; bioc-sync rsyncs master, uploads changed files to R2, publishes the symlink map, regenerates the mirror manifest and purges changed URLs on four hosts; a weekly checksum reconcile checks R2. Site, on every merge: bioc-website CI runs tests and an Astro build with a Pagefind index into an immutable site/<sha> folder and moves the site/latest pointer. Planned: the registry switch replaces the hourly copy for software packages.

Packages and the website are independent: neither waits for the other; they meet only in R2.

Content management: the website as a PR workflow

Content workflow: a pull request from a branch in bioc-website runs tests and a 27-second Astro build with link and accessibility checks, publishes a preview URL at /_pr/<n>/, and is reviewed on the rendered site. A merge adds an immutable site/<sha> build to R2 and moves the site/latest pointer; rollback writes an older sha. Forks get no preview yet.

  • Every change is a PR, with a preview
  • Every build is kept; rollback is one line
  • Astro build: 27 s; upload: ~18 min
  • Prose: still from upstream, by snapshot

Restyling: bioconductor.org/next/

The current limma landing page on bioconductor.org: site header, breadcrumb, title, installation code block, and a details table.

Today: from BBS, via master’s repositories

The /next/ preview of the limma page: dark theme, version and check-status badges, downloads and dependents counts, install code with a copy button, documentation links, and per-platform build and check status.

/next/: from the bioc-registry data plane

Same package, different data path: /next/ renders from r-universe builds through the registry’s gate, with no dependency on the legacy AWS servers. New designs can be tried there in public without touching the live pages. Browse it

The cutover

Timeline: July 2026 audits of findability, performance and accessibility; August 3 recon of both boxes and R2 loaded with 5.1 TB; August 13 hourly sync, weekly reconcile and Logpush running; September 4 ADR 0011, one propagation gate; September 28 nameservers moved at 17:30 and the site flipped at 20:09 UTC; the same evening the crawler loop was cut at 20:58 and a WAF rule added at 22:34; September 29 parity probe against master with fixes the same day; October 5 the monitoring week ends and October 12 the Route53 freeze ends.

  • BiocManager::install() acceptance passed on release and devel after the flip
  • Rollback stays one step away: Route53 is kept, with records frozen, until 2026-10-12

Day one: three quarters of the traffic was one crawler

Crawler requests per minute on 2026-09-28 UTC: about 8,300 after the 20:09 flip, about 3,400 to 3,700 after the redirect loop was removed at 20:58, and 0 to 15 after a narrow WAF rule at 22:34. Rates approximate.

84,414 of the first 112,632 requests came from 1,130 addresses on two cloud networks, disguised as Mac Chrome. They fed a redirect loop: /talks/* sent everything to an index page full of relative links.

Faster, smaller downloads

Time to first byte from Denver after the cache fix: PACKAGES 447 to 85 ms, VIEWS 550 to 205 ms, a tarball 227 to 72 ms, config.yaml 232 to 58 ms. Compressed bytes on the wire: PACKAGES 944 KB to 236 KB, VIEWS 5.3 MB to 1.2 MB.

  • HTTP/2 and brotli everywhere. The old site spoke HTTP/1.1 and sent the indexes uncompressed.
  • 63% of 530,455 requests served from the edge cache (two hours on 2026-09-29)
  • Still open: CloudFront was 30–50 ms faster from this host; part of that is Worker time

Costs

Monthly cost: legacy AWS about $5,000; new stack about $500 with the crawler still running, $240 with it blocked, of which storage is about $77; about $170 projected with 404 caching. Cloudflare list prices applied to measured traffic, not a bill.

What the parity probe found

We compared 13,271 paths from real traffic against master. Fixed the same day:

  • 404s returned 9 bytes of text instead of the site’s 404 page
  • Package files skipped the edge cache
  • Double-slash links, range requests past a file’s end
  • Landing pages had lost their Windows and macOS download links

The remaining edge cases (bioc-edge #43–#47) are small, tested code changes, because the web server is now code: routing rules, redirects and directory listings live in one program, with tests, not in server config.

Where packages come from today

Packages and r-universe

Package type Built by Reaches bioconductor.org via
Software, source tarballs BBS master, then the hourly sync
Software, Windows and macOS binaries r-universe, published by core team propagation master, then the hourly sync
Annotation, experiment data, workflows legacy builders; no replacement running yet master, then the hourly sync

r-universe already ships most binaries

The core team publishes an r-universe binary only when its version and commit match what BBS built. Across 2,326 packages, r-universe’s R CMD check agrees with BBS’s nebbiolo1 96.7% of the time (counting WARNINGS as clean).

bioc-registry: one gate for every producer

bioc-registry, a Cloudflare Worker, polls r-universe every 15 minutes, takes data and workflow packages from bioc-build through POST /publish, and seeds from bioconductor.org's BBS builds. One gate function checks build status, per-platform check results on the gating R version, a strict version bump, manifest rules for bioc-build, and records BiocCheck as advisory. Artifacts are stored by sha256 with every decision in a ledger. The propagation index is re-evaluated each wave. Outputs: an installable repository at its own workers.dev host, a Parquet archive, and preview package pages. It does not serve bioconductor.org's repositories yet.

bioc-registry: serverless, with storage as the database

bioc-registry architecture: a cron trigger polls r-universe every 15 minutes and starts one Cloudflare Workflow per distinct upstream state; the workflow writes write-once objects to R2, which acts as the database: observations, check results as Parquet changes, artifacts by content hash, the propagation index, and build logs. One Worker serves the installable repository, raw data with range requests, an in-browser DuckDB SQL console, per-package history and OpenAPI docs.

Using the registry today

# A preview, not for production
reg <- "https://bioc-registry.seandavi.workers.dev/repo/bioc-release"
# devel: .../repo/bioc

install.packages("edgeR",
  repos = c(BioCsoft = reg,
            CRAN = "https://cloud.r-project.org"))

Not through BiocManager

BiocManager::install(site_repository = reg) adds the registry next to bioconductor.org, and R takes the highest version. BiocManager has no setting that points it at the registry instead.

Works (checked 2026-09-30)

  • 2,390 release software packages
  • Source, Windows and macOS indexes
  • A clean install, at the registry’s version

Not yet

  • PACKAGES.rds (R falls back to .gz, so installs still work)
  • Archive/ for older versions
  • Annotation and experiment-data packages: add those repositories yourself

The registry switch: a plan with tests

Registry switch plan: step 0 fix registry gaps, two of four done; 1 materializer to a staging prefix; 2 rehearse on next.bioconductor.org for a week with all tests green; 3 flip devel 3.24, 2% of package traffic; 4 flip release 3.23 after a clean week; 5 archive superseded tarballs.

Compared file by file: 2,374 of 2,384 release source packages are at the same version.

Blocking: no PACKAGES.rds, the index R asks for first. Some packages have hard dependencies that can’t be installed.

A decision, not a fix: 3 release and 8 devel packages lag. Their newer versions fail in r-universe, so the gate is stricter than BBS here.

Ten regression tests (T1–T10) run on every materialize and block a broken tree.

Data, experiment and workflow packages

434 experiment-data packages, 28 GiB in total

67 of them hold three quarters of the bytes

425 / 433 changed size by under 1% across the last release bump

r-universe builds software packages but has a 100 MiB limit, and many data packages are far bigger. Until something else builds them, the legacy builders can’t be turned off.

bioc-build is a proof of concept:

  • a pinned copy of r-universe’s own workflow, on GitHub Actions
  • a configurable size limit
  • packages authorized by a reviewed manifest (bioc-manifest)
  • dependencies resolved from the registry, published through the same gate

The 25 workflow packages fit in 319 MiB together.

Roadmap

What’s next

Roadmap: M0 operational readiness, mostly done, deploys still run from a laptop; M1 production cutover, flipped 2026-09-28 and closing out on parity fixes; M2 full disconnect, retiring master, staging, BBS and legacy stats, next; M3 search and redesigns, can start now.

M2 needs: parity with master clean (bioc-edge #43–48) · the registry switch · a build home for data and workflow packages · download statistics from the new logs

Where this could go: two sites

Under discussion, not decided

Vision: bioc-registry and bioc-build publish packages.bioconductor.org, with landing pages rebuilt when data changes; bioc-website publishes a lean content-only bioconductor.org; a shared UI package gives both the same header, footer and design tokens; bioconductor.org/packages stays as a permanent served alias.

  • packages.bioconductor.org for everything package-shaped; bioconductor.org becomes a lean content site
  • bioconductor.org/packages never breaks: BiocManager hard-codes it, so it stays as an alias
  • One look across both from a shared Astro package (discussion: #71)

One data layer for Bioconductor

Where this could go

One data layer: build and package data, usage, scientific impact and community activity feed one open, versioned data layer joined on the package, serving package pages, dashboards, an API, an MCP server for AI agents, reports, and events that let agents act on package updates and builds.

Status, usage, impact and community, joined on the package, open to people and agents. Explore today: bioc-intelligence dashboard · bioc-registry API docs

Acknowledgements

  • Andres Wokaty and Lori Kern: major help with the website and the r-universe builds
  • Hervé Pagès: explaining the build system and the propagation logic
  • Jeroen Ooms and rOpenSci: r-universe, which most of this stands on

And the many people who built and ran Bioconductor’s infrastructure over the last 25 years. Everything here is a migration of their work, and it only worked because that work was there to learn from.

Materials