Refactoring is something I keep trying new approaches to. When I first started coding thirty years ago (
) I never refactored anything. Then, eventually, I had a project become so comprehensively unmanageable that there was no way I could code my way out of the hole it was in. Had it been a game I might have given up on it... but it was the software I was writing for my PhD, so that wasn't an option! On that occasion I completely rewrote it from scratch in a new language (!), but to my amazement it only took two weeks.
After that I developed a different problem: I became somewhat addicted to refactoring and my projects would overrun all the time because I was constantly redesigning them. I was mostly cured of that after spending some time in industry (not games) where constant time pressure meant refactoring was a luxury you could only occasionally afford.
Now my code architecture is generally better the first time round, but I still find myself wanting to refactor quite a bit. The rule I try to apply is: will the refactoring save time overall? (Refactoring generally makes subsequent features and debugging both easier and therefore faster.) If it won't save time, I don't do it. I usually still want
to, but I'm better about resisting the temptation.
A quick note about modularity: I agree with the sentiment that modular code requires less refactoring and is generally a good plan at all times, but in fairness I think the issue is a lot more complex. Suppose, for example, that I have a very modular architecture based on object-orientation and inheritance. I might realise halfway through development that I have an object that wants to belong to two classes. After pondering multiple inheritance, mixins, components or whatever other magic my language du jour is tempting me with I eventually just conclude that the abstraction is wrong and neither object wanted to be a class in the first place. This isn't a failure to be modular per-se. It's some other kind of code-architecture failure of a kind that's not quite as easy to avoid at the design stage!
tl;dr - Modularity is only one kind of good code design. Good code design saves refactoring.