There is an interesting possibility of designing tabletop games that are tru hybrids of the RPG and strategy genres. But to do so, we must appreciate the kinds of rules such games use, so we can see how to carefully combine them.
In pen-and-paper or tabletop games, there are two sorts of actions for the players: using a formal rule, and making some other kind of judgment. The games themselves feature those mechanical rules, and possibly meta type guidance about when and when not to use either of the two actions; these would better be called "instructions" than "rules" though.
The structure of games changes with how often rules or other judgments are made, and in what combinations. Board and wargames are usually almost entirely formal: all actions utilize mechanical, hard and fast rules. Of course, decisions are still made, but within a tightly defined range of options. And in these formal sorts of games, there is not much choice about when to invoke a rule, or which rule to invoke: the rules themselves define this. Rules are thus so closely connected that they form a closed-in sort of space; like a net with just a few small openings, through which player decisions are allowed to enter.
This is in contrast to role-playing games, which usually offer much more open space around their mechanics: there is a lot of judgment about when and how to use rules, and a great deal is done without much reference to them at all. (The less formal instructions are still kept in mind; but the variety of responses that they permit makes play very different.)
What is interesting to me is a combination of the two; or a game that falls somewhere in between a mostly-open RPG and a mostly-closed boardgame. This would be a game where quite mechanical rules exist, and are very often used, but where there is still quite a lot of room for maneuvering around them. Some very tactical RPGs may fit in this category, but they feature an extreme duality during play: there are times when the very rigid tactical combat system operates, and other times when informality reigns. I would like something that's more consistently a mix of formal and informal.
One thing I think has got to be a prominent part of this is a really creative spirit for those informal moments: not merely adding a little color, or choosing variations on some stereotyped responses, but really inventing things which will belong to the player. That would offer the greatest contrast with hard mechanics. Some "story-telling" games like Once Upon a Time offer imaginative openness, but generally very little formality at all: the players are mostly narrating according to a few instructions.
Instead of taking storytelling and adding a few rules, or taking a boardgame and adding a little creativity, I really want a hybrid. But "stories" may not be a bad thing to keep in mind, if we can apply them to subject matter that's more board-game material.
A simple game I designed when I was young involved drawing maps of secret pirate islands, which would be explored for hidden treasure by the other players. I recall that winning was easy, but the joy came from inventing the different parts of the islands and then confronting players with traps and hazards. I think something like this could work well. Invention is important, but it is not linear storytelling involving characters. What we need, in addition, is a little more formality: rules which may be mastered, and also give helpful creative constraints.
I've been interested in the abstract histories of empires for some time: how they rise and fall, their geographies, culture, and technologies; and how these all inter-relate. I tackled one slice of these issues in a game I called Halzu, which was mostly about the geography and geometry of territory defense and expansion. But Halzu fell prey to the traps that afflict a lot of wargames: too much detail to really be handled well (a computer game might have been better) and too much abstraction, so that it was difficult to care. The game mostly became a kind of laborious illustration of an abstract concept I was interested in; kind of dry stuff. What if we take that direction but begin with a lot more creativity?
One of the distinctions between wargames and storytelling games is that the former are more focused on objects, which are changed continuously by rules, while the latter are focused much more on events themselves, or sequences of events.
It may also be a matter of scale: strategy games focus on small interactions between the defined subjects of interest, while storytelling games are able to look at much larger events that change everything simultaneously. That is, wargames build big ideas (strategy, the flow of the game) out of details, while storytelling games may also generate details from bigger ideas.
It might be interesting then to attempt building a non-storytelling game that is fundamentally driven by large events, not detailed interactions of objects.