The publications shown on the website are generated from each faculty member’s DBLP record by default. DBLP has a clean public API that works from CI, so this now runs automatically in a GitHub Action — no more manual scraping.
The pipeline is two steps:
fetch_pubs.py → writes pubs.csv, pulling each faculty member’s
publications from their chosen source (see the toggle below).pubs2json.py → processes pubs.csv into ../_data/pubs.json
(filtering, de-duplication of arXiv-vs-final versions, and subject-area
tagging). The site renders pubs.json.python scripts/fetch_pubs.py
python scripts/pubs2json.py
fetch_pubs.py needs only the standard library. pubs2json.py needs pandas
(pip install -r scripts/requirements.txt).
.github/workflows/pubs.yml runs the two steps weekly (and on demand from the
Actions tab) and commits pubs.csv / pubs.json if anything changed. It needs
Settings → Actions → General → Workflow permissions → Read and write.
Each faculty member’s source is selectable, so you can default everyone to DBLP and flip a single colleague if their DBLP list doesn’t suit them.
DEFAULT_PUB_SOURCE at the top of fetch_pubs.py
(currently "dblp")."pub_source": "scholar" (or "dblp") to that
person’s entry in ../_data/faculty.json.| Source | ID field in faculty.json | Runs in CI? | Notes |
|---|---|---|---|
dblp |
dblp_pid (e.g. 95/7687) |
✅ | Keyless, fast, curated for CS; includes arXiv (CoRR). The default. |
scholar |
scholar_id |
❌ local only | Google Scholar via scholarly. Scholar blocks datacenter IPs. |
DBLP covers CS venues thoroughly but not non-CS journals (biomed, vision
science, …). So with MERGE_OPENALEX = True (the default), every dblp author
is also queried in OpenAlex by ORCID, and any works DBLP lacks are added —
DBLP still wins on overlaps. The ORCID is read straight from the DBLP record (no
config needed); if DBLP has none, add an "orcid" field to the faculty entry.
OpenAlex is keyless and CI-friendly. Only works with a named venue are added (so
venue-less notes/datasets are skipped), and pre-1960 records are dropped as
metadata artifacts.
Namesake guard (oa_affiliations). OpenAlex’s author disambiguation can
conflate several same-named researchers under one ORCID (e.g. multiple “Chen
Sun”s — medical, transportation, ML). For a common name, add an
"oa_affiliations" allowlist to the faculty entry, e.g.
"oa_affiliations": ["Brown", "Google"]; only OpenAlex works where that author
is affiliated with one of those institutions are then kept. Distinctive names
don’t need it.
DBLP itself sometimes merges two different same-named researchers under one
PID — e.g. our graphics Daniel Ritchie, a UC-Irvine education researcher, and a
Liverpool materials chemist all ended up under 17/7188. Because the OpenAlex
guard above only filters the OpenAlex supplement, these DBLP-native strays need
their own guards. Three run, cheapest/most-automatic first:
ORCID mismatch (automatic, no config). DBLP tags many records with the
contributing author’s ORCID. When we know the person’s real ORCID (from the
"orcid" field, else the DBLP person record), any record whose matching-PID
author carries a different ORCID is dropped. Records with no per-author
ORCID — the majority — are untouched, so this only removes provably-mislinked
papers. This alone caught the three “middle-school generative AI” papers.
namesake_affiliations (per-person institution blocklist). Strays the
ORCID check can’t see — a DBLP record with no ORCID tag, or an OpenAlex
supplement paper that the oa_affiliations allowlist let through — get caught
here. Run on the combined DBLP + OpenAlex list, we look each paper up in
OpenAlex by DOI and drop it when the same-named focal author is affiliated
with a listed institution, e.g. ["Liverpool", "Irvine"] for the chemistry
and education Ritchies, or ["Calgary", "Lapland", "St Thomas", "Ophthalmology"]
for the law and ophthalmology David Laidlaws. Matched case-insensitively
against OpenAlex’s raw affiliation strings (its normalised institution
names are unreliable — it maps “Brown University” to “John Brown University”).
Both the published and any arXiv-preprint copy of a flagged paper are removed
together. This needs a DOI, so it can’t judge preprint-only or
affiliation-less records — and it depends on OpenAlex being reachable (if it
isn’t, the guard simply no-ops that run). Only authors with the field set
incur the extra lookups.
Prefer a blocklist (namesake_affiliations) over the oa_affiliations
allowlist when a person has many legitimate papers with no affiliation
metadata (e.g. Laidlaw’s older or interdisciplinary work): the allowlist drops
every affiliation-less record, whereas the blocklist only drops papers that
positively name a homonym’s institution.
pub_exclude (manual catch-all). The escape hatch for the rest — a
namesake paper with no ORCID and no affiliation metadata anywhere (e.g.
Ritchie’s "pub_exclude": ["FairytaleQA"]). Each entry is a case-insensitive
substring matched against the title and the pub/eprint URLs, so it can be a
distinctive title fragment or a DOI. This is also the offline-reliable lever:
move an entry here from namesake_affiliations if you want it dropped even
when OpenAlex is down.
How scholar behaves in CI: scholarly is deliberately not installed in the
Action, so a Scholar-sourced author’s existing rows in pubs.csv are retained
(not refetched) during scheduled runs. Their list refreshes whenever you run
python scripts/fetch_pubs.py locally (with scholarly installed). This keeps
the Action green while still letting you use Scholar for someone who needs it.
If a dblp author has no dblp_pid yet, their existing rows are likewise
retained (with a warning), so you can migrate people one at a time.
Search https://dblp.org/search/author/api?q=<name>&format=json (the PID is the
tail of the author’s url, e.g. pid/95/7687 → 95/7687). Watch for
disambiguation suffixes (-1, -2) and confirm the affiliation is Brown. For a
very common name, find the PID via one of the person’s known papers:
https://dblp.org/search/publ/api?q=<distinctive+title>&format=json.
DBLP throttles bursts (~1 request/second); fetch_pubs.py already spaces its
requests and backs off, so just let it run.
pubs2json.py tags each publication with subject areas by matching venue/title
keywords defined in ../_data/areas.json. DBLP uses terse venue abbreviations
(CVPR, 3DV, WACV, …); to improve tagging coverage, add those abbreviations
to the relevant area’s words/strings in areas.json.
Add them to ../_data/faculty.json with a dblp_pid (and optionally a
scholar_id if you ever want to flip them to scholar), then run the two steps
above.
scrapePubs_DEPRECATED.pyscrapePubs_DEPRECATED.py is the old Google-Scholar-only scraper, superseded by
fetch_pubs.py (which keeps Scholar as the scholar source). Do not run it —
it’s kept for reference only. The normal flow never needs it.
The animated background is a large vendored WebGL file (js/fluid.js), driven by
a small policy module we own (js/fluid-setup.js). The vendored engine makes no
page-level decisions; fluid-setup.js decides whether to run it (it skips under
prefers-reduced-motion) and in which mode — the subtle, paused home-page
background, or the 404 page’s full-page interactive “playground”.
Its failure mode is sneaky: when init breaks, the home page still looks fine
(the fluid is paused and hidden behind the content panel), so a regression can
ship unnoticed and only surface on the 404. scripts/fluid_smoketest.mjs guards
against that — it loads the built pages in headless Chrome and checks that the
engine initialised with no exceptions, in the right mode, on both pages.
jekyll build # produce _site/
node scripts/fluid_smoketest.mjs # exits non-zero if the fluid is broken
Node 18+ only (no npm install); it uses SwiftShader so it runs without a GPU.
Set BVC_CHROME to a Chrome/Chromium/Edge binary if it isn’t auto-detected. Run
it after any edit to fluid.js / fluid-setup.js.