The problem is spectralsword, traps are a problem too but not only in shiranui, that's a different matter.
Imagine the deck without spectralsword, no other shiranui card can combo by itself after being thrown to the gy, no other shiranui card easily banish itself to be looped with solitaire.
^^^ That probably won't be enough, they'd just ditch solitaire, they have 6 cards to mill spectralsword and samuraisaga, shogunsaga and shade to keep recovering it.
Spectralsword has to leave if you really want to stop shiranui.
Boy, if you guys are more worry about tiny swords than some 3000atk dragon that can be search, destroys 2 cards without targeting, while chain blocking your move, that still roam free -- then Shiranui is surely the best deck ever.