Picking up from my previous post, How do I know?: Descartes' Rationalism versus Hume's Empiricism.
I had a request lately for a post on the topic of "Persistence Ignorance" (I swear I'll finish it soon Ward). While trying to formulate that post, I realized that I needed to start with an examination of the first causes that might lead you to Persistence Ignorance. I thought was a nearly mandatory way to start the post because Persistence Ignorance (PI) is only a means, not a goal in and of itself. It's possible that an examination of the goals behind Persistence Ignorance might lead to a perfectly good compromise that isn't pure PI. More interesting to me is if there is really an underlying set of qualities that we can use in a concrete manner to make rationalizations about software development techniques. I'd like to use this post as the kickoff for a short set of essays to explore what I think those first causes are. Just warning you now, this is going to be long.
Debate is Good
We developers are an argumentative lot. This tool/practice/process/technique versus that one. We're always debating the best way to build software -- and it's a good thing that we do because software development is a young profession. Our basic laws of physics change underneath us as hardware abilities increase and multicore systems beckon. New ideas, techniques, tools, and processes are popping up all the time. Which to try? What to bet my career on? Can my tool really beat up his tool?
Inevitably we form emotional attachments to our tools or begin to see our identity in terms of a tool. That's unfortunate in that it blinds us to other ideas and leads to us holding retrograde postitions for a reality that might have passed us by. We need to stop and remember why it is that we liked our current positions in the first place. Both to better practice and understand those ideas, and also to know when to jump to something better.
I personally believe that debate can be healthy for us as a means to compare and contrast ideas of how software ought to be built. In any argument or discussion however, somebody will eventually pull out this old nugget -- "I just use whatever is best for the job at hand!" Which is a nice sentiment, but nothing but hot air without some substance behind it. What is best? How do we decide what is best? Is there a constant set of basic criteria that we can use to judge software techniques through time? I say yes.
And why do you believe that?
Frans Bouma in his exit from the ALT.NET list included this little nugget that made me sit up and pay attention:
In general asking 'Why' wasn't answered with: "research has shown that..." but with replies which were pointing at techniques, tools and patterns, not the reasoning behind these tools, techniques and patterns. Answering Why with pointing to techniques, tools and patterns is creating a cyclic debate: the Why question is asked to understand the reasoning why some tools/techniques/patterns/practises are used. Pointing back at tools/techniques/patterns/practises isn't going to make you any wiser, as you then only learn tricks, because you can't put any argument on the table why you use pattern X, use technique Y and practise Z.
Frans goes on to use the example of Separation of Concerns (SoC) as a concept that everybody says you should use, but nobody questions or explains. I've called Separation of Concerns the "alpha and omega" of software design. I obviously think it's important too, but why? What does it give us? To Frans' point, only by understanding those real goals can we better utilize SoC to write better software.
It's important to understand why we espouse the things that we do, both to better understand how to apply something, and to know if we're just doing something out of Cargo Cult behavior. What is the underlying first causes that make us think SoC is so important? Let's take up Frans's gauntlet and start trying to answer the "why's."
But first, before we even talk about the "why's," I want to first throw out some of...
My Choices
I've made and continue to make a certain set of choices for how I prefer to work and what I think works. You've made your own set of choices. Below are some of my choices and preferences. I'm sure some of these are purely preference, but I think many of them are directly connected to my interpretation of the first causes and underlying forces that I'll discuss in the next section.
- Evolutionary design is much more natural to me, and I strongly prefer an environment that actively supports that.
- I'm a huge proponent of Test Driven Development and Continuous Integration practices. I'd say that creating an automated build script is the very first development task in any project. My tool usage is very much impacted by my ability to do TDD with a given tool. And yes, I think TDD provides a lot of benefits above and beyond simple unit testing (more on that later).
- I design from the middle tier or user interface first, and the database last. Again, some of that is just preference, but there are some very real reasons to work first with your objects and let the database fall out secondly.
- I'm all for micro code generation in forms like ReSharper's live templates, but I don't care for fullblown code generation schemes that try to build a full application from a fully formed database. Given a choice between using a reflective approach versus code generation I'll almost always pick the reflective approach. We can't do it in a mainstream .Net language yet, but I'll take metaprogramming over code generation as well.
- I'm interested in the little mechanical advantages of software factories, but Software Factories as a methodology sounds like a complete non-starter to me (can you say CASE 2.0?)
- As far as the Domain Specific Language (DSL) experimentation goes, I'm betting on textual DSL's and language oriented programming over graphical DSL tooling. I'm much more interested in things like the Ruby acts_as_state_machine than I am in Microsoft's Workflow Foundation.
- I'll go out of my way to avoid using designers in Visual Studio for anything but laying out elements on a screen, and I'm looking hard at dynamic layout techniques for my system. I'm going down a path of using my own fluent interface DSL to express data bindings and screen behavior in WinForms because I think it's faster than using the design time support in WinForms.
- I favor explicit coding over magic. I vary rarely use custom events in my code because I think they obfuscate the code and make it harder to understand. I prefer explicit observer classes as need be. Call me strange on that one though because I'm close to a minority of one.
- I prefer to understand code by examining and reading the code. I'm not a big believer in external documentation (mostly because I've never found other people's documentation to be very useful and I know that the documents that I labored diligently over in the past were never read by anyone but me). Code readability is paramount to me (but of course we all have our different ideas of what that means).
- I generally think that most (but not all) comments in the code are an apology for bad design rather than something that is helpful. The xml documentation embedded in code makes code harder to read.
- I like clean separation of concerns in my architectures. I'm very iffy about Active Record approaches in static typed languages. I intensely dislike architectures like CSLA.Net that ignore Separation of Concerns -- even though CSLA.Net obviously has plenty of fans. I think orthogonality in a software design is a paramount concern that cannot be sacrificed for short term productivity gains.
- I absolutely detest intrusive frameworks, especially if they involve a lot of inheritance chains.
Ok, there's what *I* like. Now, let's see if I can connect some of these choices to what I think the underlying forces of software development really are. And of course, if I can't make the connection, I need to reconsider these preferences;-)
First Causes and Values
Underlying everything we choose is a set of drivers. At the very lowest level is the simple goal: "provide as much customer value as possible as efficiently as possible." You might say that your first cause is to just get it done. Great, but what does "done" really mean? I'm going to take the XP line here, "Done, done, done" means that the code can ship. The functionality is complete, it's tested well enough, and the customer has approved the functionality. It's not good enough just to write code, we also have to have ways to make sure the code we've written is correct and that we've written the correct functionality in the first place.
Most of the effort in software development is deciding what to do. The construction of software is nothing but typing. I've seen it argued that the most important, and time consuming, activity in all of software development is learning. Learning what the customer wants, what they need, how our architecture performs, and learning that our initial design wasn't that great when we actually tried to implement it. I don't think we can really know everything upfront, so if we're really setting off on a project without knowing the exact path, we better make regular course corrections with plenty of...
Feedback - I honestly think this is the single most important "First Cause." My experience is that productivity is most heavily impacted by our ability to quickly apply some sort of validation to anything we do. I've just written a requirement / designed a screen layout / coded a requirement / sketched a UML diagram, is it right / valuable / usable / not really what the user intended? I strongly think that a team's or individual's tools and practices should be chosen with a very strong regard for the quality and quantity of feedback that those tools and practices provide.
A couple weeks back I was in some discussions about the future Composite WPF guidance. One of the participants made the remark that we should be careful not to sacrifice productivity for the sake of testability. I think that's a false dichotomy. When we talk about getting to "done" we need to be careful to consider everything that it takes to get to done, and that includes the effort that we put into removing defects from the code. We need to optimize the complete lifecycle of the code, not just the initial production of the code. Being fast to code and hell to debug when anything is wrong is a net loss. So let's add...
Testability - The simple ability to test and verify a system bitwise. How easy and quick is it to validate parts of the application work? I feel like this is a specific case of Feedback, but merits it's own treatment. I've written about this quite a bit in the past, but I want to use the essay on this topic to challenge my own views a little bit and see how things fall out in the post-TypeMock world.
Software is complex, and the overall system may have hundreds and even thousands of variables. The human mind can only hold so many variables at one time. What we need to do is to be able to pick off logical parts of the system and work them in isolation. We need to eat the elephant one bite at a time. That leads to iterative development processes for the team, and in the code we need...
Orthogonality - Simply put, the ability to divide and conquer by being able to work on a single concern of the whole system at a time. Every major design technique that I'm aware of exists to provide a heuristic to take an amorphous blob of requirements and decompose it into a series of approachable coding tasks. The ability to work on one thing at a time is crucial for productivity. Designs that allow us to turn a complex system into a series of simple tasks are good. Designs that organize the code to make the assignment of responsibilities predictable are good. Designs that throw all the code into a giant DoWork() method aren't helpful.
I'd also extend "I just want to get it done" to being able to sustain ongoing efforts in the same code later. Most systems I've worked on are continuously maintained and extended until they just become too expensive to maintain and extend. Clearly, there's a huge economic value in these systems to providing for sustainable development. Along the way, I've observed (empiracism) a very large disparity between my productivity in one codebase versus another codebase. For purely economic reasons, I definitely want to build in the ability to change a system over time. If I'm going to get in and change a system over time, I want a couple of qualities in my system to make that change safe. Feedback loops, Testability, and Orthogonality all play major parts in making it safe and economical to change an existing system, but there's another major factor in a software design's ability to change over time:
Reversibility - The best example I can give for Reversibility is watching my 4 year old son paint with water soluble paint across the table from me as I write this. I'd be a lot more worried about what he's doing if I didn't know that I can easily wash it all off later. The same thing applies in software design. If I know I can reverse a design decision later without serious consequences, I don't have to be that worried about getting it perfect now. That's a hugely liberating feeling. If I can build the system's persistence scheme simple, and fit in caching later when the volumes demand it, I can build the system right now for the initial release without having to make those decisions about caching upfront. When we have Reversibility, continuous design is possible and analysis/paralysis problems can be put to bed. My design choices are directly informed by my wish for better Reversibility.
Oh yeah, and people stuff too. I wrote ad nauseum about teams and communication and collaboration last year because my client was self destructive in this regard. That's all available at: On Software Teams.
The Series
If I've tallied things up correctly, I'd like to write some deep dives on:
- Feedback
- Orthogonality (mostly links though)
- Testability
- Reversibility - Its impact and importance for doing design. What design choices give us more or less reversibility. I also want to look at how external factors can improve or diminish reversibility
- Can you "see" the flow of the code? Solubility, readability, whatever you call this quality of code. Can I tune into the "signal" in the code that I care about without getting bogged down by unrelated "noise?" I think I'm going to say that this is a specific manifestation of Orthogonality.
- Why and How TDD Works - First and foremost, TDD is a design technique, and I'm going to treat it as such. TDD is more than unit testing. TDD encourages Orthogonality and provides a fast Feedback loop.
- Persistence Ignorance (PI) - Just applying the first causes to the problem of persistence and seeing where PI applies. It's just a means for better Orthogonality and Testability.
The Fly in the Ointment
There's at least a couple problems with my arguments in this post that I'm happy to acknowledge:
- Not every system or project is the same. As I've said before, I strongly favor using a Domain Model architecture backed by an Object Relational Mapper that supports "Persistence Ignorance." I also said that I don't care for the Linq to Sql or NetTiers type of approach to system building, but there are a lot of systems where the entire goal is to attach a user interface to a database in the fastest way possible. I think data-centric approaches are perfectly applicable in those cases, even though I would never touch them in any of the systems I build that tend to be very rich in behavior and business logic. All I mean to say is that our choices of "what is good" are very much influenced by the work that we do. If you work primarily on CRUD or reporting applications that don't change much throughout their lifecycle, then you won't care about the same things that I do. Then again, you're going to get into serious trouble if you take those data-centric techniques and try to apply them to a system with rich domain logic.
- Opportunity Cost. Just because your favored solution "works," doesn't mean that there isn't a better way out there that would give you better results. If that's the case, you're actually losing time and money by using your current techniques, technologies, or processes. The first example that comes to my mind is a talk I sat in last spring. The high priced architect speaking was demonstrating his application framework. The framework required you to write a *lot* of little factory methods and classes by hand. I asked him why he didn't just use an IoC tool to avoid all of that monotonous coding. His response to me was that his way has always "worked" for him. Yes it worked, but I could have done the same thing with less code and mechanical effort. I think that TDD works for me, but maybe there's something totally different I could be doing instead or more likely, a better way to do TDD. Maybe Design by Contract could replace some parts of TDD for me at some point (I still see DbC as a small subset of TDD, not the other way around).
I can make both an empirical statement that TDD works, and I can provide the rationale for why it works, but there's always the possibility that something better is out there. Even if you "know" what's right, you better keep a look out for better things.