How Spinneret works

Spinneret watches a small fleet of web scrapers built with Bright Data Scraper Studio. After every run it measures what came back, compares that against what the collector is contracted to deliver, and decides whether the scraper is still doing its job.

When it is not, Spinneret writes a repair prompt from its own measurements, sends that prompt through Bright Data’s self-healing API, approves the resulting patch and re-runs the collector to check the score actually improved. The whole cycle needs no human to describe what broke.

The data those collectors produce feeds a buying-signal board: a ranked list of companies where something recently changed in a way that suggests budget just moved.

Scrapers rarely fail loudly. A site ships a redesign, a CSS selector stops matching, and the extraction code keeps running perfectly. It returns the same number of rows on the same schedule. The rows are just empty.

Nothing in a normal pipeline notices this. HTTP returned 200. No exception was thrown. The job is green. Three weeks later a sales team reports that leads dried up, and someone starts bisecting a month of data to find out when the values turned into nulls.

Spinneret treats that silence as the actual bug. Every run is measured against a contract, so a field that quietly empties out is a first-class event rather than something you discover downstream.

A real example from this project

The job-board collector was created from the description extract every job listing: job title and company name. Bright Data built a scraper that returned 66 rows, and every one of them contained a single field: product_page_url. Neither contracted field was present.

A conventional pipeline would have called that a successful run of 66 rows. Spinneret scored it 20 out of 100, classified it as a schema gap, and repaired it. Nothing about that break was staged for a demo.

You need Node 20 or newer and a Bright Data account. The free tier includes 5,000 monthly credits and needs no card.

git clone https://github.com/hassaansaleem28/spinneret
cd spinneret
npm install

cp .env.example .env.local        # paste your Bright Data token
npm run spinneret -- seed         # register the collectors
npm run dev                       # dashboard on localhost:3000

.env.local is gitignored. Keep your token out of commits and out of any screen recording.

Watch a collector heal itself

npm run spinneret -- observe hiring-signals   # run, measure, diagnose
npm run spinneret -- heal hiring-signals      # compose a prompt, dispatch it
npm run spinneret -- approve hiring-signals 1 # approve, re-run, record the delta

Or run the whole loop unattended with npm run spinneret -- cycle hiring-signals, which heals only if the evidence justifies it.

Seven stages, running continuously. The first three happen on every observation. The last four only run when the diagnosis found evidence worth acting on.

When Spinneret refuses to heal

There are two cases, and both are deliberate.

A run that never completed. If the request timed out or was rate limited, nothing came back. Every field looks absent, which is indistinguishable from a site that rewrote itself completely. Healing that would burn a cycle and give the healing AI no live page to work from. Transport failure is not drift.

A collector that is merely degraded, with no specific field identified as the cause, is left alone on purpose. An unfocused prompt such as the scraper seems wrong tends to make a working scraper worse, because the healing AI has nothing specific to anchor a rewrite to. In that case Spinneret records the evidence, marks the collector degraded and waits for a person.

The same reasoning drives the cooldown. A heal that ran and did not raise the score is never retried on a timer, because re-sending a prompt already shown not to work costs credits and can degrade the scraper further. A visibly broken collector is the better failure mode.

Every run produces one number between 0 and 100, made of three weighted terms. Try the scenarios below to see how the same formula separates a schema gap from a selector drift from a pagination break.

Why a value has to carry information

Coverage counts a field as filled only if its value means something. null, an empty string, whitespace, and the literal strings "null", "N/A" and "undefined" all count as missing, as do empty arrays and empty objects. Broken scrapers emit these constantly, and treating them as real values is exactly how a hollow pipeline passes as healthy.

Two decisions about the baseline

Yield compares row count against a rolling baseline, and how that baseline is built matters more than the comparison itself.

It uses the median, not the mean. One catastrophic run returning two rows would drag a mean low enough to hide the next real break behind it.

It is built only from runs that already scored well. If degraded runs fed the baseline, the degradation would quietly redefine normal, and the following break would register as an improvement.

Drift detection turns a score into a decision. It compares the run against the baseline, classifies the severity, and produces the evidence lines that later become the repair prompt.

Severity

Thresholds

regressionDropPct:   25   // percentage-point fill-rate drop that counts
criticalScore:       55   // below this is a hard break
degradedScore:       80   // below this is degradation
minBaselineSamples:   2   // healthy runs needed before trusting comparisons

These are deliberately conservative. A false positive costs one heal cycle and an approval gate. A false negative costs silently hollow data for days, which is the failure this whole system exists to prevent.

Telling two breaks apart

When fields regress but row count holds steady, the evidence says so explicitly: pagination and the row selector are intact, so the breakage is field level. When row count collapses instead, the evidence points at the row or pagination selector. These need different repairs, and naming which one it is steers the healing AI at the right part of the code.

This is the part that makes Spinneret different from a monitoring dashboard. The instruction sent to Bright Data is assembled from measurements, not typed by a person.

Fencing off what still works

When a collector is partially broken, the prompt lists the fields still filling above 90 percent and instructs the healing AI not to touch them. Without that clause a rewrite can regress working extraction while fixing the broken part, and the verification run would show a heal that helped one field and broke two others.

Two prompt shapes

If some contracted fields arrive and others never appeared, the task is to extend the schema, and the prompt says so plainly. If fields that used to work have hollowed out, the task is to repair drifted selectors. Naming the narrower job produces a more reliable patch.

The character budget

The CLI caps prompts at 1000 characters. Spinneret budgets to 960 to leave room for shell escaping. When the budget binds it drops evidence lines from the middle, because the opening clause names the target and the closing clause states the required schema, and losing either one produces a worse patch than losing a detail.

Everything else on the dashboard is retrospective. It reports what already happened and asks you to believe it. The Contract Lab is the one place you can act, and it exists because a self-healing claim deserves to be tested rather than trusted.

Pick a collector, change what it is required to return, and the whole diagnosis re-runs in front of you: coverage moves, evidence assembles, and the repair prompt writes itself a character at a time.

None of it is simulated

The rows being scored are rows those collectors genuinely returned on their last healthy run. The scoring, the drift classification and the prompt composition are the same functions the sentinel runs in production, not a demonstration copy. Because the contract is the real definition of correct, requiring a field the scraper does not produce makes it genuinely non-compliant, and the text that appears is exactly what Spinneret would dispatch to Bright Data.

Try this

Require salary on the job board collector. The score drops, the field reads absent, and the prompt names it as the repair target while explicitly fencing off the fields that still work. That protection clause is generated, not written by hand, and it is what stops a repair regressing extraction that was already correct.

A company in a directory is not a lead. A company that just opened four revenue roles is. Spinneret scores change, and every point traces back to a stated rule shown in the interface, because nobody acts on a number they cannot argue with.

Each source type gets its own rules

A job title, a changelog entry and a directory listing say different things about intent, so each runs against its own table rather than one table stretched across all three. Within a table only the strongest matching rule applies, which stops a company with many similar rows compounding the same evidence.

Single-subject sources

A competitor changelog has no company column. Every entry belongs to the one company that publishes it. Those collectors declare a subjectCompany, so attribution comes from the collector rather than from a field that will never exist.

Four layers, with dependencies pointing one direction only.

What lives in the core

health.ts       fill rate, schema conformance, yield → composite score
drift.ts        baseline comparison → severity + evidence lines
heal-prompt.ts  evidence → natural-language repair instruction
signals.ts      rows → scored buying signals with stated rationale
cooldown.ts     whether a fresh heal is permitted yet
types.ts        the domain vocabulary everything else speaks

These modules are covered by 48 unit tests. The retry policy in cooldown.ts is pure specifically because it is the code that spends money without supervision, so it is worth testing exhaustively.

Three collectors, deliberately different in shape so they exercise different parts of the health model.

Everything the dashboard can do is also a command, and both call the same service layer rather than reimplementing each other.

seed                          register collectors from config into the database
create <url> "<description>"  build a new collector with Scraper Studio
observe <slug>                run, measure, diagnose, store
heal <slug>                   compose a prompt from telemetry and dispatch it
approve <slug> <healId>       approve the patch, re-run, record before and after
cycle <slug>                  the whole loop, healing only when justified
status                        fleet health and recent heals

Running on a schedule

npm run watch                                 # every 30 minutes
SPINNERET_CRON="*/10 * * * *" npm run watch    # or set your own

A sweep can outlast its own interval during a slow heal, so a late tick is skipped rather than queued. Overlapping runs would corrupt the shared baseline.

Thin routes over the service layer. Long operations return immediately and report progress through the event stream, because a heal can run for fifteen minutes and no browser should hold a request open that long.

Server-sent events rather than websockets: data only travels one way, and EventSource reconnects on its own, so a dropped connection during a long heal recovers with no client retry code.

Public data only. No login-walled sources, no paywalled content, and no personal data.

Signals are scored at company level and derived from public job postings, directory listings and published changelogs. Spinneret does not collect, store or score details about any individual. A job posting contributes its title and its employer, never a named person.

Every target is a source with no pre-built Bright Data scraper, chosen from the categories the brief asks for: niche job boards, regional and vertical directories, and competitor changelogs.