And defining your principles at the start of your game is absolutely essential. I think there's little room to argue that most of the most successful OSR games have very clear design goals. Take the core retroclones--Swords & Wizardry, BFRPG, Dark Dungeons, Labyrinth Lord, OSRIC, For Gold & Glory, Blueholme--which set out to emulate a particular D&D ruleset (and then follow-ons from those, like Iron Falcon or B/X Essentials). With those niches largely filled, the other big OSR games have mostly focused on emulation of some broader theme, whether genre/specific authors (Mazes & Minotaurs, Crypts & Things, ASSH, Beyond the Wall) or gameplay (Low Fantasy Gaming, for low-magic games; ACKS, with its emphasis on the D&D "endgame"; DCC, with its emphasis on taking shittons of crystal meth and playing D&D). The remainder have been what I call kitchen-sink rulesets, which amount to "throw in everything cool I can think of from all the D&D games": examples include Blood & Treasure and Fantastic Heroes & Witchery.
Interestingly, the big exception I can think to all this is LotFP, which despite its mandate of "weird fantasy roleplaying" doesn't have a ruleset actively supporting that ("actively" being the key: it passively supports that kind of play just fine, i.e. it can be used for such, and the system doesn't fight you when doing so). In other words, it says it wants to do something, but other than giving you art along those lines, doesn't actually do anything about it (at least in the core book). Offering nothing especially unique as a ruleset, though with some choice bits here and there, Lamentations succeeds not because of any mechanical approach but because of the vision and considerable talent of its creator.
So What do I Want in an OSR Game?
There's a lot of little things I want to play with, but the precise penalty you assign to two-weapon use or what have you is not really important at the start. Vaguely, I know I'm going sort-of kitchen-sink, but I need to nail down the heart of the project. Of course, it helps that I'm making this for myself and no one else, so I don't have to convince anyone, but that doesn't mean I don't have reasons for X and Y, so let's go into them, because talking about this stuff (hopefully with you the reader) is the whole point of this blog.
1) Short. One of the major elements of the OSR movement that attracted me has been the succinctness of the core rules: B/X is 124 pages, for example, and crams a lot in for that amount. At the same time, OSR publishers have tended to give us some pretty hefty books: witness C&T Remastered (247 pages), Blood & Treasure (267), ACKS (269), OSRIC (402), FH&W (428), DCC (484), ASSH 2nd ed (618!). Some of that is more generous layout spacing, and some is more art, but really what's in those pages isn't the point. It's that I want the trim succinctness of B/X and perhaps even more. In some ways I can cheat, in that I can just say "creatures are someone else's problem" or the like and then I don't have to waste a ton of pages on it. But I'm still going to be keeping a close eye on page count and the options which bloat it. I have some ideas on how to make some very large trims, and a vague goal of two 48-page books (Player's and GM's Handbooks).
2) Sweet. Related to the above is mechanical ease of use. I own 500 tons of Hero Games stuff. I love Hero and will cherish the memories I have of play with that system forever. But man, nowadays I just want to dive in and go. Old D&D only achieves simplicity by virtue of not having a lot of rules; the rules that it does have are often clunky. I've heard it expressed that this kludginess is a feature rather than a bug, in that it provides a sort of charm that a more aseptic ruleset using unified mechanics and a reduced variety of dice does not. While I can't argue with such subjective feelings, my own subjective feeling is that such people are wrong and that this is a recipe for bad design. I'm always reminded of someone's statement that if D&D had used ascending AC in the first place, no one would have ever felt the need to invent THAC0; that THAC0 can be learned utterly misses the point. I think "would someone ever willingly invent this, let alone implement this, if faced with the likely alternate candidates" is a good rule of thumb for all aspects of design. Mechanical simplicity also helps keep the page count down, feeding into #1.
3) Broad Compatibility: aka "Recognizing my Limitations". I want to be able to use this ruleset with minimal adaptation with one of the OSR's greatest strengths--the host of incredible adventure modules out there for it. Adapting here and there is no problem, but it's no good if it takes me hours; best case scenario it could be something I could do on the fly during actual gameplay.
Along those lines, I want to be able to use existing bestiaries. I have no interest in writing up the SRD monsters yet again: I don't feel I could offer anything new there (or rather, I'm not inspired to try, which largely amounts to the same thing). If I'm going to be using beastiaries belonging to other games, then that's going to determine in some places how far I can stray in terms of mechanics.
4) OSR Feel: I've talked about this a bit earlier, essentially arguing that mechanics alone can't guarantee you an OSR game. At the same time, they can certainly ensure you lose it. I'll have to watch out for this, especially in light of my next goal.
5) Support for Low Player Counts: Old-school D&D assumed lots of players. It then further assumed that those lots of players often had a NPC army backing it up. It spoke of walls of spearmen blocking the dungeon halls, stabbing enemies with multiple ranks.
I'm in my 40s. I have a wife. Most of my friends are in the same boat: time is precious. Even then, finding gamers who are available is one thing: filtering based on gamers with actual social skills is another. All this means my typical gameplay group is 4-5 folks, and while I intend to support the use of retainers/hirelings, I do not want to make them mandatory to survive. Quite frankly, I think the reputation old-school D&D has for lethality is somewhat overstated, due to people playing in modules originally designed for 6, 8, or 10 players or more (OD&D groups could run into a dozen or more in the early days), plus NPC support mobs. Since I'm making this game for me first and foremost, I need to be able to expect four guys (my typical play group) to be able to take on what many other OSR games, derived from these old rules, seem to assume more will be tackling.
Of course, then the Danger Danger sign goes off. More capable characters means stronger characters, which leads to heroic gameplay, which leads to fear, anger, snowflakism, muh storyline, the Dark Side, etc, and before you know it you've blown up Alderaan. I don't actually believe that there's anything wrong with heroic play--I've always found the dichotomy between much of the original inspirational material of D&D and how the game actually played in practice rather odd (as I point out in the my earlier article linked above, so too did most other people, which is I think why the gameplay style so markedly shifted in the mid-80s). I'm even okay with plot-based play, as long as everyone is on board. BUT, ultimately I'm not going for either in this case: I believe there's a wide gulf between "more capable" and "Dragonlance/WotC D&D", and I intend to plant my design flag there. If at first level a character has 10 hit points instead of 5 and can hit a goblin 70% of the time instead of 55%, it's still going to be very easy to toss mortal challenges at him; no real extra effort is going to be required. And again, he's going to be assumed to have less help than in a lot of games, so that bit of extra is only going to put him back to square one.
What matters most along these lines, I think, is keeping the resource management and other elements of old D&D intact, so no "resting and getting half your HP back"-type mechanics. I want the tension of watching torches go out, worrying about kobolds stealing your mule full of treasure because you can't care it all on your person, and whatnot.