There are two bad pieces of advice about fillers. One says every "um" is a cut to your score, so police them all. The other, which I have seen a lot lately, says fillers do not matter at all, so ignore them. Both are wrong, and the second one is the one I want to correct, because I have said something close to it myself.
Here is the more accurate version. One "um" will not sink you. But a stream of them is one of the clearest warning signs I see in the data. Not because the sound itself is penalized, but because a wall of fillers tells me you are planning every phrase while you speak, and that is the thing holding your score down. Let me show you what the numbers actually say.
Quick answer
A single filler is harmless, and it is better than a silent gap. In my data, once you account for how much a person said and how much they paused, the "um" sound on its own carried almost no penalty. But heavy fillers are a different story. The more you use, the lower your ceiling: among the answers with the fewest fillers, about 22 percent reached the top band. Among the heaviest, only 2 percent did. Fillers are a symptom, not the disease. The disease is searching for words as you go, and that is what you fix.
The two things that are both true
One "um" is not the problem. When I isolate the filler sound and hold silence and answer length constant, its independent effect on the score is essentially zero. A filled pause keeps sound going and holds your turn, so it is genuinely better than freezing into silence. If you say "uh" a few times in a strong, full answer, do not lose a second of sleep over it.
A wall of "ums" is a real signal. This is where "fillers do not matter" falls apart. Look at how the top band thins out as fillers pile up.
And at the extreme it is starker. Among the heaviest ten percent of filler users, only about 2 percent reached the top band, and none scored a clean 5. Flip it around and it is just as clear: among my top scorers, the median filler rate was about half that of everyone else. You can reach a good score with some fillers. You almost never reach the top with a flood of them.
Why fillers travel with a lower score
If the sound itself is not penalized, why do heavy-filler answers score lower? Because the filler is not arriving alone. In my data, the more someone filled, the more silence they also had, the fewer words they produced, and the slower their overall pace. The "uh" is the visible tip of a fluency problem underneath.
Think about why you say "uh" in the first place. You say it because your next words are not ready yet. You are planning the phrase while you are speaking it. That constant planning is what produces the fillers, and the same planning trouble produces everything the score actually reacts to: thinner content, longer gaps, more grammar slips as you assemble sentences on the fly. The filler is the check-engine light. The engine is your fluency.
You can see this in real answers. The ones that stall out at a fair score are usually the ones where "uh" and "um" appear every few words, sitting right next to small grammar breakdowns like "it's depends upon" or "some of them doesn't." The speaker is not being marked down for the "uh." The "uh" and the errors are both coming from the same place: no phrase is ready before it is needed.
So be anti-filler, for the right reason
This is the part that matters. Do not go to war with the sound. If you try to catch every "um" as it leaves your mouth, you spend the attention you need for thinking, and you trade the harmless filler for a costly silent gap. That is anti-filler for the wrong reason, and it backfires.
Be anti-filler the right way: treat a wall of "ums" as a signal to fix what is causing them. Remove the need to search, and the fillers fall away on their own.
Three things remove the need to search. First, plan the shape of your answer so you are never inventing structure mid-sentence: answer, reason, example. Second, prepare a few real stories in advance, so when you would have stalled, you reach for something you can already tell. Third, build automaticity on your common ground, the everyday topics and phrasings that come up again and again, so the words are ready before you need them. Fillers are what a searching brain sounds like. Give it less to search for.
When to actually worry
Use this as a rough gauge. An occasional "uh" in an otherwise full, flowing answer is nothing. But if you listen back and you are filling every few words, treat that as a warning light, not because the sound is scored, but because it means your speech is not automatic yet, and that is what is capping you. That is a fluency project, and it is very fixable.
See what is really going on
The clearest way to read your own fillers is to look at them in context. Take a free practice test on My Speaking Score, answer an interview question, and look at where the "ums" cluster. If they sit next to silences, thin patches, and grammar slips, you have found the real target, and it is not the sound. It is the searching underneath it.
FAQ
Do "ums" lower your TOEFL Speaking score?Not directly. The filler sound on its own carried almost no penalty in my data once silence and length were accounted for, and one "um" beats a silent pause. But heavy filler use goes hand in hand with lower scores, because it signals a fluency problem underneath.
So do fillers matter or not?Both. A few are harmless. A constant stream is a warning sign, because among the heaviest filler users only about 2 percent reached the top band. Fillers are a symptom of searching for words as you speak, and that searching is what caps your score.
Why do I say "uh" so much when I speak English?Because your next words are not ready when you need them, so you fill the gap while you plan. It is a sign your speech is not yet automatic, not a sign you are careless.
How do I actually reduce my fillers?Do not police the sound. Remove the need to search: plan your answer shape, prepare real stories, and build automaticity on common topics. When the words are ready, the fillers fall away.
Is it better to say "um" or to go silent?A brief "um" is better than a long silence, because it keeps sound going and holds your turn. But the best answer needs neither, which comes from being ready, not from forcing yourself to stop.
Can I still get a high score if I use fillers?You can get a good score with some. You rarely reach the very top with a flood of them, because heavy fillers travel with the thinner, more halting delivery that keeps answers out of the top band.
The takeaway
Do not panic over a single "um," and do not believe anyone who tells you fillers are meaningless. The truth sits in between. The sound itself is not what costs you, but a wall of "ums" is the clearest sign that you are building every phrase as you speak, and that is what holds your score down. So do not hunt the sound. Fix what is underneath it: plan, prepare, and get your common language automatic. Do that, and the fillers disappear on their own, because you no longer need them.