I'll share my thoughts on the subject, since I've been working on a C++ engine for more than two years now and have made lots of attempts before that...
For starters, you're probably going to have to make a few engines/revisions before you really nail it, unless perhaps if you're working off of some strict template or your needs aren't too complex. As you make more engines, and more importantly, more games, you're going to get a better feeling for what sort of stuff usually needs to go into an engine, and you'll probably want to revise your old stuff a few times. I've also found it useful to learn existing engines, like Unity, whence I've gotten some ideas on how to solve certain problems.
One big problem I remember having with most of my first attempts at making engines is that I focused too much on the engines themselves, implementing lots of stuff that I might or mightn't need, overthinking architecture way too early on and so forth.
With my current engine, the one I've now managed to work on for more than two years, and that's really starting to become useful, I have some pretty solid philosophies going for me to keep productivity up, because the biggest problem with writing an engine is losing motivation when you don't see results soon enough or you learn more and want to rewrite everything from scratch again...
Design philosophies One is to
make a game alongside the engine, using the engine, at the same time, implementing new features as needed for the game, instead of guessing what I might need (tho some stuff is certain based on experience from previous games or future plans for the current game)
Another is to
implement things quick and dirty at first, and only clean the code up when it is known for sure that it'll work out. Saves a lot of time and results are visible sooner, which is a great motivator.
Also the classic:
avoid premature optimisations. Much for the same reasons as the last point. You get results faster and if you realise later that you need to optimise, then do it then. You might not need it. I've rewritten my rendering system a lot lately, but having a barebones one to work with before that, I've been able to work on other parts of the engine and the game in the meantime with good enough visual feedback at the time. You can always perfect things later, but get them to work first and then build up. It's also very easy to make mistakes or introduce bugs if you write these humongous systems in one go.
And perhaps obvious, or not: plan important things out on paper first before you clean things up and code up the more final systems. More often than not, I've found that I've realised issues along the way, which I might've been a lot more painful if I'd spent hours coding up the first part before realising this. Also writing pseudocode to sketch something up can be similarly helpful.
Start small, start dirty, start unoptimised, and fix and expand and optimise things later, over time!Things to bear in mind before you start Please
make the engine portable, at least to Windows, Mac and Linux (or Android and iOS if you're making an engine for mobile games)! It is 2016 and this is super easy to do right away from the start, but it'll be a pain if you have to port stuff later. Use SDL2 (which I think is amazing) or something to get a nice foundation to build upon and then choose portable libraries for things like audio and so on. There is so much portable stuff out there.
Is the engine going to be very
general (like Game Maker or Unity where you can do almost everything) or very
specific (genre bound like RPG Maker, or graphics bound like a 2.5D engine or a voxel engine)? In the latter case you might want to really think hard about how to solve things in ways that work out the best only for such specific purposes and try and throw away any overhead that would come with a generic engine.
Figure out how to deal with game entities, as these are the fundamental building block of almost any game engine. Most engines today seem to be component based and try to avoid inheritance to a great extent. How will you solve this stuff? Unity separates game objects (things that can hold components) from components and uses inheritance to derive a few related components (different types of colliders, for example), which is good since you mostly don't have to bother with complex inheritance structures giving you problems, but you can still use polymorphism when it makes sense and is useful (if you don't know if an object has a box collider or a circle collider for example and you just want to get its position). My engine is similar, but a bit different: it doesn't separate game objects from components, but everything is just nodes deriving from a parent class, and can usually be nested unless the rules for a specific type of node says that it cannot have any children. I use inheritance/polymorphism on a small scale much like Unity does (having a base class for lights and deriving point lights, spotlights and so on from that, for example.)
2D or 3D? Both? Might want to keep that in mind. Don't know yet? 2D now and maybe 3D later? That's kind of my situation, so I've half-prepared for it by making the engine always use 3D vectors instead of 2D vectors for transformations and vertex data (but I actually use the z for things like layering even in 2D), and by using OpenGL to which I could easily pass 3D data if I wanted to, but that's basically it for now; the engine still only supports 2D ATM.
Will you want to add
support for a scripting language? Better figure this out somewhat early on, because you might have to bind a whole lot of your C++ interface to the scripting language, and that's a lot nicer to do right away as you code up the C++ interfaces, instead of adding the support for scripting at a much later point and having to sit down and bind hundreds of things in one day. Or two. Or three.
C++ specific protip:
you probably want to start fresh by using C++14 now that you can! Step out of the middle ages and into a modernised C++ with lots of features that it should've had years ago. Compiler support is up to it at this point.
Another thing you can much more easily do by using modern C++ is to
add Unicode support right from the start. It might be trickier to fix as an afterthought. I personally care a lot about being able to translate my games and for that matter properly rendering the names of people in the credits, so it's important to me to support diacritics and multiple writing systems.
And finally,
organise your project somewhat well from the start. Once you start getting more than a few files, either sort them into folders or use virtual folders in an IDE of choice. I couldn't imagine making a project of this size without an IDE that keeps stuff very neatly sorted out for me. It has grown over the last two years, and there are hundreds of files.
I personally find it useful to write Doxygen comments for everything from the start so that I can generate documentation if I want to, and also to put todo comments as needed in the code so that I can safely forget about currently irrelevant things that might need fixing up later until I look through the todos occasionally. I also mark all files with the creation date as well as dates when the files were updated so that I don't get confused if I manage to lose or revert files by mistake or something.
Speaking of which,
remember to BACK UP files!!
And gee, I could go on and on and on. I have complex input scheme systems. Cutscene sequencers. Customisable rendering pipelines. Materials. Gamepad support. File loading and management. Camera utilities.
Engines, unless perhaps if they are very specific, can get
big and it's no surprise to me that I've already spent such a long time on mine and still have yet to add audio support to it! Patience. Don't try to do everything at once. Do develop that game alongside the engine so that you can see it accumulating into a more and more useful engine.
I repeat:
start small, start dirty, start unoptimised, and fix and expand and optimise things later, over time!