Halyard Speech builds speech recognition for doctors' notes. The team is twelve people, four of them researchers, and none of them want to run servers.
Before
They trained on a shared university cluster. Jobs waited in a queue for up to two days, and a failed run meant going to the back of the line.
After
They now run on a reserved block of 16 Forge 80 GPUs in fra-1, with checkpoints written to object storage in the same region. When a run fails, it restarts from the last checkpoint on its own.
- Queue time went from about 30 hours to none
- A full training run takes 3 days instead of 5
- Their monthly bill is fixed, which their funders like
We used to plan experiments around the queue. Now we plan them around the question we want answered.
Noor Adeyemi, research lead at Halyard Speech
What they'd tell other small labs
Start with on demand for a month to learn what you actually use, then reserve. And keep checkpoints in the same region as the GPUs, because moving terabytes between regions is slow and it shows up on the bill.
