Artificial intelligence is moving beyond tools that simply assist people to AI agents that can take action on their behalf—scheduling meetings, managing email, checking calendars, and executing multi-step workflows. In healthcare, that shift raises a particularly important question: where should AI autonomy end and human judgment begin?
Sarah Hirschfield is the founder of Carly, an AI assistant that began as a scheduling tool and has evolved to handle more complex tasks and custom integrations. Healthcare became an important part of Carly’s customer base organically: as Sarah explains below, within a year, 30% of Carly’s clients were in healthcare, and physicians and hospital administrators increasingly approached the company as its capabilities expanded.
Sarah’s background is equally relevant to the questions surrounding AI autonomy. She came to AI entrepreneurship through philosophy and law, earning an MPhil in Philosophy from the University of Cambridge and a law degree from Yale. Her academic work has included ethics and metaethics, and before Carly she co-founded a personality-testing company.
We wanted to speak with Sarah because the shift from AI that assists to AI that acts creates a different set of questions for healthcare organizations. When an AI agent can access systems, communicate with people, and execute workflows, efficiency is only part of the equation. Human oversight, least-privilege access, PHI, authorization, and the boundary between administrative automation and consequential decision-making become equally important.
In this conversation, Sarah explains how Carly evolved from a scheduling assistant into a broader AI agent platform, why she believes AI is better at executing human judgment than making it, and how she approaches limiting an agent’s access and autonomy in healthcare environments.
From a Scheduling Problem to Healthcare
You came into AI entrepreneurship from law and philosophy rather than medicine or technology. What led you to build Carly, and how did healthcare become an important part of its customer base?
Serving healthcare professionals was a happy accident. When I started Carly, I was focused on solving a problem I experienced: time lost to scheduling and coordinating calendars. I was not thinking about doctors or any use cases beyond my own needs. Carly started as an executive assistant who only scheduled meetings. You would copy Carly on a scheduling email and she would propose times and send calendar invitations once the parties confirmed. That was it. I finished the initial version in a day or two.
I might have stopped there had I not been enrolled in a data visualization class at the time (I was in law school but spending my days learning statistics and data) in which my final project was an analysis of ten years’ worth of my own calendar data. It was that project that made me appreciate how deep one could go with scheduling—how rich of a topic it was for analysis. I decided to see if there was a market for a scheduling assistant. Indeed, there was, and my very first customer was a healthcare tech founder. A year in, 30% of Carly’s clients were in healthcare.
I didn’t set out to specifically target healthcare professionals; in fact, I was under the impression that trying to sell to healthcare was a bad idea—we’d deal with long sales cycles, heavy regulation, and a saturated market. But doctors and hospital admins came to me, especially as Carly grew to handle more tasks and custom integrations.
Administrative work dominates healthcare. Administrative staff outnumber by more than double physicians and nurses. PCPs receive, on average, 15 EHR emails for every hour they spend seeing patients—one new message every four minutes. The majority of patients schedule appointments manually, by phone. So it is no wonder that the demand for automation was palpable. Carly quickly adapted to serve the needs of medical providers.
“AI Is Great at Executing Human Judgment”
You’ve said that AI is most effective when it executes human judgment rather than making consequential judgments itself. How does that principle shape the way you think about Carly’s autonomy and the tasks you allow it to perform?
The way I think about these questions about AI autonomy is very practical. I’m focused on delivering a quality experience to the customer, so that means listening to them and sharing my learnings from working with other providers to inform what an ideal workflow will look like, then helping them get set up and started as soon as possible.
It is hard to generalize, but one conclusion I’ve come to is that AI is great at executing human judgment (and rarely for making the judgment itself). So if I make a judgment that I want to make an appointment at 8 a.m. tomorrow, AI is really good at turning that into an appointment record and sending reminders (if I decided I want reminders). But AI is not good at deciding what I should do, for example, at 8 a.m. tomorrow. Leave that to me.
Being good at executing human judgment might not sound that exciting, but when you consider how AI can write code really fast, you can see why this is so powerful. If I decided I wanted a sequence of actions to happen (say: when someone signs up on a form, it should run a bunch of checks on the inputs, cross-reference against 10 spreadsheets, check some other people’s calendars, and write an email to request documents), I would need to either purchase some software that could do this or spend hours in a workflow automation builder writing out a thousand branches on this decision tree.
But if I ask Carly to make this workflow, Carly can make it a reality in a minute or two. That dramatically increases my efficiency, and it reduces the cost of achieving that efficiency.
Designing AI Agents Around Least Privilege
When Carly is used in a healthcare environment, how do you approach limiting an AI agent’s access to data, tools, actions, recipients, and levels of autonomy—and when does Carly enter into a Business Associate Agreement with a healthcare customer?
With AI agents, less is more. We apply the Principle of Least Privilege (give an agent only the data, tools and authority it needs to complete its specific job) and HIPAA’s Minimum Necessary Standard (limit the PHI an agent uses, requests or discloses to what is reasonably necessary for its defined purpose) when designing agents.
Carly, our flagship product and clients’ point-of-contact, comes equipped with the tools to create specialized, restricted agents to serve particular purposes. This happens in many dimensions: data (which patients, records, folders, mailboxes), tools (calendar, email, EHR, or billing systems), actions (read, draft, edit, send, or delete), recipients (anyone or only approved people), time (permanent or temporary access), and autonomy.
Clients need to be able to precisely define the scopes and permissions, and we spend a lot of time thinking about how to make that easy and intuitive for them.
When a healthcare customer’s use of Carly involves PHI, we enter into a BAA.
Autonomy Is Not a Percentage
How do you determine when a Carly action should run autonomously and when it should require human confirmation, particularly for sensitive or consequential actions?
There is no meaningful percentage because confirmation depends on the action and the customer’s configuration. Reading calendar availability is materially different from sending a sensitive message, cancelling an appointment, or changing a record. Customers determine which actions may run automatically, which require confirmation, and which the agent is not permitted to perform.
Finding the Balance Between Autonomy and Control
Where do you see the right balance between AI autonomy and deterministic workflows, particularly when AI agents are deployed in healthcare?
There’s a continuum. On one hand, you can have an LLM with a bunch of tools (or a few very powerful tools that can execute any command) and give it access to everything and then let it run wild (this would be a coding tool in so-called YOLO mode, e.g., “claude –dangerously-skip-permissions”).
On the other hand, you can have a deterministic workflow with no AI involved, or AI used only for small data classification tasks with structured outputs that are carefully routed and dealt with deterministically.
Using the former (YOLO mode) means you can get what you need done really fast. But it will also make mistakes and need careful review and monitoring and cost a lot of money—because you will need to use state-of-the-art models (so it doesn’t make so many mistakes) and run it all the time (so it can do things).
Using the latter (think of something like Zapier) is much cheaper and safer, but it requires time invested upfront to set it up and requires tweaking and changing as processes need updating.
In the art of AI and automation deployments, and in life, the goal is to strike the right balance between ease and autonomy, on the one hand, and lighting money on fire and making irreversible, highly consequential mistakes, on the other.
Discover more from Doctor Trusted
Subscribe to get the latest posts sent to your email.
