Posts filed under 'Software Development'
This is a long overdue announcement as I joined Pivotal Labs in 2006, after I left Nominum. But we are trying to raise our profile recently as we no longer want to be “the best kept secret in software.” (This is a quote from more than one client.)
Anyway, check out the following links. (In particular, note my personal Pivotal Labs blog as I plan to blog mostly there.)
April 9th, 2007
Most software practitioners I’ve met think the choice of technologies (programming languages, tools, libraries, etc.) has the largest influence on a project’s success. I actually think it is a minor influence and I think the project’s alignment with market demands is the single largest determinant of success.
While there are those who would explicitly claim the significance of technology, when asked directly developers may cite another factor. However, I make my claim regarding this “conventional wisdom” based on behavior; in most any project I’ve been involved in, a tremendous amount of time and energy is dedicated to selection of programming languages, technical evaluations, and debates about architecture and patterns.
By contrast, it seems the topic that gets the least attention is a very simple question: Are we solving the right problem? It amazes me how often, because of lack of interest or a strong sense of faith, developers do not question why they are building the application they spent countless hours working on.
From a separation of roles and responsibilities perspective, it seems appropriate that developers should not concern themselves with such a question; it is the job of the product manager. However, I’ve seen the most successful teams consist of a cross functional group, where each member took an active interest and responsibility for the project’s outcome. (Anyone been involved in a technical success, but it was a business failure?)
Agile software development can help a lot; the emphasis on communication, feedback and iterative development encourages the identification of real user need. It is amazing how much clarity can be achieved when developers and users sit together in the same room or you demo an application every two weeks. Alternatively, I find that traditional, specialized and compartmentalized development models, where information is “handed over the wall” or communicated with documents instead of conversation encourage a very myopic view by developers. Even if you wanted to better understand the business goals, pressures and motivations for the project, there may not been anyone accessible to ask.
Though I make a living being a proponent of agile development, giving too much significance to the process is just as misguided as emphasizing the technology. Agile development can be an effective means at understanding market needs, but it is not an end unto itself.
If I were to quantify my idea, in terms of percent influence on success, alignment with market needs accounts for 65%-75%, process, practices and team dynamics for about 20%-25% and technology for only 5%-10%. Similarly, I think the time and effort spent evaluating issues or devising tactics should be proportional to their importance. Simply put, I’d rather spend three weeks training a development team about user’s daily activities than three weeks evaluating the choice of Java vs. Ruby or SOAP vs. REST.
I’ve seen numerous examples where the development group had horrible practices or made many poor technical choices; it was amazing they ever got anything done. Yet, their products were great successes and wins for their organization. I’ve also seen cases where the architecture was beautiful and most any developer would awe at its magnificence, but it was flatly rejected by their user base. Unfortunately, I’ve even been involved in a case that was a marvel of agile adoption(well, at least the engineering practices): thousands of tests running in seconds, pair programming, and elaborate continuous builds: but it didn’t do much for the company’s bottom line.
Ultimately, you can’t win if you cross the wrong finish line.
April 20th, 2006
Every now and then someone makes a call for collective brain power; it may be someone in a blog posting or a CEO addressing his staff. The odd thing, at least to me, about such requests is that they frequently occur after some suboptimal individual attempt at solving the original problem. This seems inefficient; why wait as a last resort to call on the power of a collective conscious?
The simple answer is one of resource allocation and opportunity cost; it is inefficient to always point some collective at every issue when a subset can solve the problem. However, agile environments offer an alternative. The high visibility and communication of requirements, risks and decisions provides the opportunity for an opt-in application of collective brain power. Team members can choose to give background thinking to a topic, while not consuming everyone with it.
This isn’t necessarily a fully baked idea, but something I’ve observed with more than one agile team. It is not often there is the explicit call for ideas, yet is seems like many decisions have the benefit of input from multiple people. Besides being egalitarian and empowering it generally leads to good outcomes.
November 16th, 2005
Okay, so my title may be little exaggerated, especially since there isn’t officially an Agile 2.0. However, for the agile development community, there are many shifts, trends and new ideas that make it feel like a new generation of agile development. (More on this later.)
Anyway, I’ve been following some of the Web 2.0 developments lately; I’ve read many blog posts (http://web20workgroup.com is a good place to start for those interested), heard gossip about VC funding, and know a few people working on Web 2.0 projects. All of this information makes me think an agile approach is perfectly matched for the development of Web 2.0 applications.
Agile development emphasizes the need for feedback, discovery and adaptation. A common agile credo is “release early, release often.” Agile development is basically a genetic algorithm (for more on this, read this post); an agile approach evolves its software. Evaluating the “fitness” of the application with each iteration is critical. Real user feedback is one of the best indicators for making such an evaluation.
Most Web 2.0 projects (or at least those I have seen) are consumer oriented. Consumer oriented, service-based software markets are generally accommodating (and sometimes welcome) frequent releases, assuming each release is at least as good and as stable as the previous. Agile development favors short iterations and core practices such as automated testing and continuous integration develop the ability to release early, often and reliably as a core competency of its adopters.
Most projects in early stages are based on a vision and skeleton of a possible solution; success is largely determined by how quickly the application can evolve into what user’s need. Quick delivery is necessary to compete and not miss windows of opportunity. I think the best software process to support this is based on an agile model. One of my former clients (when I was a consultant) believed this as well and feels that their agile adoption was a key facilitator of their IPO last year. I also know some very large, high profile, Bay Area web companies who are adopting agile software development and see it as necessary to remain competitive.
I think others support the complementary nature of agile and Web 2.0. The canonical book on Ruby on Rails, one of the popular Web 2.0 enabling technologies, is Agile Web Development with Rails and it describes how Rails development embodies and accommodates an agile approach.
I noted that I believe the agile community is undergoing Agile 2.0. Last year the 2nd edition of the XP white book was published, in which Kent Beck revised and updated his description of XP. Mary Poppendieck’s keynote address at XP/AU 2004 described agile as “crossing Moore’s chasm”; it is shifting from early adopters to mainstream. While some feel this shift may water down agile, I think it can amplify its effectiveness; many in the agile community have updated and evolved their ideas to better support the software development process.
Finally, I have also adopted a more mature understanding of agile development. I’ve had many successes in my career applying agile methods, but have been challenged in my current job and our agile adoption. These challenges have caused me to reflect on many assumptions and ask many questions. Though I have more questions than resolutions (which I think is a good sign you are actually learning), my understanding and application of agile software development has evolved into its own “next generation.” I know a few others who share many of my ideas and are working in the Web 2.0 space. For these instances, I am excited to see how the combination of Agile and Web 2.0 turns out.
November 16th, 2005
Agile development is structured on iterative and adaptive development. The application evolves over time and, at least hopefully, closer matches the user’s needs. The basic process of agile development is a genetic algorithm; after each iteration, adaptations are made and the whole process repeats with each iteration.
The success of a genetic algorithm generally depends on the quality of its fitness function. A fitness function evaluates the success, or “fitness”, of a given result. Within agile development, the iteration boundary marks an opportunity to evaluate product backlogs, prioritize features, and generally tweak the process; both the product and the process evolve.
In order to have a successful agile project, it is imperative to have fitness functions for evaluating both the application being developed and the process by which it is developed. The agile community provides various guidelines for the process evaluation; retrospectives, cost of change, quality of team dynamics, test suite run time and size, shrinking code bases, etc. are all assessment tools and indicators of process health. However, there is little information about evaluating product health.
There are some obvious indicators and questions: do users like it, are bug counts low, is the product selling? I think these are symptoms of the relative health of the application, but I think you need more from a suitable fitness function, particularly when the product isn’t an instant and obvious success.
For example, prioritizing features can be a very difficult activity; users will typically assign many features a “number 1″ priority. However, ranking each feature in strict order, with no duplicate priority, is a challenge.
While the assessment of the product’s health can be seen as a marketing or product management activity, I do not think it is outside the scope of agile development. Successful projects depend on the collaboration of cross-functional teams. Thus, I think it is important to communicate the fitness criteria for an application amongst all on the team, even if one group has primary ownership of it.
Ultimately, the fitness function for an application is an answer to the question of “why do we build the products we do?” Do you know the answer to this question for your current project? Ask others and you may be surprised at the responses.
November 11th, 2005
Superficially, there are many ideas and practices that are in common with agile development and waterfall development. There is a Agile Unified Process, for example and an “XP plugin” for the Rational Unified Process. However, I think agile development is quite different than waterfall, though it is not always easy to name the differences and understand why drawing strong distinctions matters.
I think agile and waterfall are each based on a fundamental assumption; if you subscribe to this idea then you probably should adopt that process. The assumption is about how to achieve a successful software project.
For waterfall development, I think the basic assumption is that perfect planning leads to optimal results. Agile assumes is that an adaptive and evolving approach is the way to go.
I do not think anyone who adopts waterfall actually thinks perfection is attainable, but I think the entire process is aimed at the pursuit of a perfect plan. Extensive requirements, complete with test traceability, elaborate architectures, long analysis and design phases, and up-front test plans are attempts at precise, accurate plans. Usability, overcoming technical challenges and meeting deadlines are achieved (or so is the idea) by building a plan that satisfies these requirements.
By contrast, agile development tends to be lightweight on planning. There is a general pragmatism to “do as little as you can get away with” when considering planning, requirements capture, architecture and documentation, as emphasizing planning would emphasize a different base assumption. However, agile development, in particular XP, is very heavy weight when it comes to automated testing; tremendous effort is usually expended. The reason is the emphasis on adaptability and evolution. Agilists assume change is a necessary and inevitable and agile processes encourage developing your “accommodate-change muscles.” Thus, automated tests tend to be a primary mechanism to achieve efficient regression testing, which accommodates code design changes, for example.
Anyone who reads my blog knows I am very biased towards agile development. Though it may sound arrogant and obnoxious, I think many of the practices of waterfall are based on myths and thus make the process flawed. For example, the practice of software architecture is the application of learned patterns and principles in an attempt to satisfy functional and non-functional requirements. The problem is that technology changes quite fast and the software industry is too immature; it is hard to successful apply experience when the rules, constraints, and contexts are in flux. For at least the foreseeable future, I think you are better served with a process that allows for discovery and adaptation than depends on rare constants.
November 11th, 2005
I have interviewed a lot of people over the years. (I’ve hired over 50 people, interviewed hundreds, phone screened many more and read thousands of resumes.) While I don’t claim to be any sort of recruiting expert, there are two things I can intelligently (as much as I can for anything else) speak of: trends in resumes / candidates and guidelines for being stronger on both.
The single most important fact to understand as a job seeker is: A prospective employer has no attention span whatsoever.
Your goal is to get the attention of the person who will actually decide to hire you and make clear why you are different from all the other candidates she comes across. Employers must sift through many resumes (there were times I would look at 100 resumes a day) and frequently cannot afford to spend a lot of time on a resume. Assume you must communicate in 10 seconds a reason that the employer should continue reading. Recruiters, internal or external, are not your target audience. While you may need get past them to reach the actual employer, never forget that your message should be targeting the actual employer.
While this sounds obvious, you are selling yourself to an employer. However, most resumes I see are not selling but reporting. Listing the details of each job, tool, language, API, or activity from your entire experience is not effective.
A general sales adage is a product should be made easy to buy, not simply easy to sell. When faced with buying something that is a “no-brainer”, there is no selling. A candidate should always consider the perspective of the employer; what would make it a “no-brainer” to hire me?
There are many different approaches and considerations for how a candidate can position herself as an obvious choice. Ultimately, I think it can be summarized by two questions (from the point of view of the employer): What can this candidate do for me? How does this help me? The first question is about the results the candidate can achieve or deliver. The second is about the match of these skills for the employer’s context.
The following are some guidelines for candidates:
Overall
- Have a 30 second elevator pitch.
- Know what your strengths are. Know what results you have achieved. What makes you special and stand out?
- Know your professional goals.
- What are your career plans? Where do you want to be in one year? Five years?
- Have a career plan.
- The goals are a strategic concern; your plan is tactical. You should be able to describe the role you want to perform, with detailed functions and responsibilities. Be able to describe an ideal work day, month, or year.
- Know your strengths, weaknesses and what affects them.
- More importantly than what you do well or poorly are the factors that enhance your skill set and those that detract from it. These are critical to selecting the right position.
Resume
A resume makes the first impression. It must get someone’s attention immediately and should answer the two most important questions above for the employer.
- 1 page only. No exceptions.
- I don’t read 7 pages resumes; if the person can’t be brief, it implies they are disorganized and cluttered. I will note one caveat to this: your resume should be one page. It is an introduction and summary. If there are details that are important to communicate, include an addendum or appendix (which should be never more than 2-3 pages) and make it clear that this is extra, not part of the resume. You can title it “Project Details” or something, but it should have it’s own header and not be confused as part of the resume.
- Lead with the elevator pitch.
- Your 30 second elevator pitch should be communicated immediately in a “Professional Profile” or similar section.
- Focus on results and achievements
- When I read that a candidate was on a project that tripled company revenues, I am interested to continue reading.
- Highlight technical qualifications
- Do not list every three letter acronym you know, even if you are an expert. Focus on your core strengths and the technologies you want to work with or that are relevant to the employer. If you know C++ but are looking for job doing Java, don’t mention you know C++.
Almost 100% of the resumes I read look the same. They all share the same subset of technical jargon references and technologies. There are many, many J2EE, .NET, C++, etc. developers. It is almost impossible to differentiate yourself in this category. Though a technical profile is obligatory, as a technical worker, it should be brief and contain highlights. I suggest putting it at the end of the resume.
The technical profile of a resume is the source of the largest mistakes in candidate positioning, so I will babble and rant about this a bit. I think the reason for these mistakes are:
- technical people focus on technology
- many recruiters encourage listing all your technical experience
- employers may screen based on the content of technical profile
The first point is probably pretty obvious and I won’t discuss it. The second is also common, though I discourage it. The last reason is the most significant: frequently resumes are so poor (or there are so many) that employers have little choice than to screen based on technical profiles. However, I think there are a few mistakes being made here. First, candidates try to keep as many options open as possible and this actually hurts them. Few people are experts in the large lists of technologies, languages and tools commonly listed on resumes. Even if you are, simplify. For example, if you’ve worked with many Web Service / SOAP technologies, write in your technical profile something like “SOAP / Web Services expert (worked with Axis, WebMethods, BlueTitan …) instead of listing the 20 different Web Service related frameworks you know. Alternatively, if your goal is to work on web services, only list the web service frameworks you know and omit the other items such as object relational mappers, databases, etc.
Listing too many elements in a technical profile has the risk of implying indecision and insecurity. Don’t list technologies you don’t really want to work with. Also, a candidate that doesn’t care what technologies they work with (when they are listing many, unrelated technologies) seems to have little career focus. Finally, it may convey insecurity in the candidate; it can make a candidate look desperate and thus undesirable. Less is more.
Phone Screens
In over 90% of the phone screens that I do, I know if I will pursue a candidate in the first two minutes of the phone screen. A phone screen is intended to serve answer one question: Should I invest the time to formally interview this candidate? Formal interviews are generally very expensive to conduct and therefore most organizations don’t do them lightly. There are a few things to keep in mind at this point.
- Research the company
- I hope this is obvious, but it amazes me how many people do not do this. Look at the companies website, competitor websites and industry websites for the employer. Learn as much as you can.
- Be prepared
- Know your own experience well, such that you can communicate succinctly. Learn how to be brief and focus on highlights. Interviewers will ask follow-up questions when interested and will appreciate brevity. Also, give thought to typical questions and know your answers. Such questions include: Why are you looking for a job? What is the ideal job, role, organization? What is the most important thing I should know about you as a candidate? Tell me about an interesting technical or team challenge and how you resolved it. What was the best project your were on? Worst? Why?
- Interview the employer organization
- You should look as closely at the employer company as they are looking at you. Good candidates take their professional development very seriously and are engaged in knowing about the company, its culture, its business model, future, history, etc.
If you are looking to work for me…
This entire blog entry is biased about making a candidate attractive to me, as an employer, but there are a few other items of note:
- Communication Skills are Key
- Being able to communicate clearly, effectively and succinctly is critical. It is vital to working in an agile environment and poor communication is one of the core reasons I don’t pursue candidates.
- Must be a Team Player
- Agile Processes emphasize collaboration. You must thrive in a team environment.
- Be Honest
- I generally know when a candidate is weak in a particularly area, unsure or blatantly lying. I prefer and respect an honest and direct “I don’t know” rather than trying to dance around ignorance.
- Be Committed to Projects
- I am looking for people who will take ownership of the result and do what they can to achieve it. I want candidates who take the outcome as a personal reflection of their abilities and work ethic. I don’t ask for late nights or weekends but respect people who sacrifice when they need to make sure something is done.
Conclusion
Employers hire people, not cogs. A candidate should never forget that an employer is investing in a person and should do everything they can to position their value as an individual to an organization. If you are seen as just a Java developer who knows SOAP, you are easily replaced. Great employees, however, are assets to companies and rare.
September 23rd, 2005
In a recent blog post I discussed the “Market / Application Gap.” In order for a software product to succeed, it must align with the user’s needs. The faster an organization can bridge the gap between the market and the actual application, the greater its competitive advantage and its opportunity for success.
At the start of a project, some effort is given to requirements discovery and this effort varies with software methodology. From here requirements are communicated to engineers who in turn produce an application that reflects the requirement. With agile software development there is an emphasis on taking action as early as possible; waterfall tends to focus on a more complete sets of requirements, designs and test plans (read: more time). With my “Market / Application Gap” diagram in mind, the process looks something like this (order of events, with respect to information flow / deliverables, are numbered):
This cycle represents an iteration: from requirements to delivery of an application. While much has been discussed about the iterative nature of agile development, the truth is that every software process uses iterations. The only question is the length of the iteration. An agile project might have an iteration length of weeks to months; waterfall can have iterations that last years. (Remember, I am defining an iteration in this context as the delivery of an application to the end user.)
At the end of an iteration, the overall gap between the market and the application either grows or shrinks. If you have an iteration cycle of a few months (at most), you can recover if you missed the needs of your market. When you have an iteration cycle of years, you may not get a second chance if you missed the mark. This is one of the core reasons why agile development makes more sense than waterfall: from a risk management point of view it is foolish to take a “big bang” approach where you have few opportunities to satisfy your market.
If the goal is to close the gap between the related participants in the “Market / Application Gap”, the quicker you can complete an iteration, the quicker you possibly close the gap. Many successful software products simply work the way users expect; unless you’re omnipotent, this requires some trial and error and probably some mistakes. Users will tolerate some mistakes, but won’t wait around long. Long iterations can result in a lost audience, which spells doom for commercial software. In my years of software development, I’ve seen a variety of software processes, many of which claim to encourage iterations and quick feedback. Agile Software Development is the only approach I’ve seen deliver on this promise in actual practice.
September 2nd, 2005
Lowering the cost of change is a frequently cited benefit of adopting agile development and extreme programming, in particular. You’ll frequently see graphs drawn that show an exponentially growing cost curve as the cost of change for a waterfall project and graphs that show a level or even down sloping (after an initial ramp up cost) cost curve for XP projects.
I personally have been in involved in projects where the down sloping cost curve was actually achieved, so I know it is possible. I also know it is rare; on other projects a flat cost curve was all that was achieved. Both are good outcomes, but what is the cause for the differences in these projects? How does a project actually achieve low costs of change?
However, before exploring how to lower the cost of change, defining the cost of change is necessary. I frequently see discussions about the cost of change that center around the time or effort required to implement a story. While not wrong, I think this perspective is too narrow and short sighted. This definition is not affected by a team that is “going in circles”. But who cares if they can go in circles really fast?
Mary Poppendieck has discussed a Maturity Model of Software Development. (She has a presentation on this and an article published in Software Development magazine. http://poppendieck.com/maturity.htm) The basic idea is that what matters is how quickly and reliably can an organization take a user request and deliver it in the software. This is the cost of change I wish to focus on: the amount of time elapsed and resources spent on taking a user feature and having it present in the software application.
Now, returning to analyzing the cost of change itself, there are two considerations: what contributes to the cost of change and how does agile development help lower it.
Contributors to the cost of change:
- time to assess and prioritize a user request
- overhead for specifying a story
- “churn” in stories
- number of changes to code base to implement story
- cost of release
Time to Assess and Prioritize a User Request
How is feedback handled? How many people are part of the decision? The longer it takes for a sales person to communicate a user requirement to product management, have it prioritized and then worked on by engineering, the hight the cost of change. Similarly, the number of people (or worse yet, committees) that must approve or deny the request, the higher the cost of change.
Overhead for Specifying Stories
The time and number of people it takes to specify a story and complete set of acceptance tests affect the cost of change.
Story Churn
The higher the number of times a feature area is revisited, be it because of bugs or revisions due to what was delivered wasn’t quite what user’s wanted, the higher the cost of change.
Code Base Changes Required to Implement a Story
The time it takes to assess and discover the necessary code changes and to implement them impacts the cost of change.
Release Cost
What is the overhead to a release? Is there a manual QA cycle? How long does it take? Documentation done at the end?
How can Agile lower the Cost of Change?
The primary mechanism for lowering the cost of change is bridging the gaps between the user, product owner, developer and application. (For more on this, please see this blog post.)As the gaps are closed, the application tends to be in alignment with the user requirements.
Practices such as an onsite customer and acceptance tests help develop a shorthand of communication between the product owner and the developers. Thus, the overhead of specifying stories tends to go down and existing acceptance tests and infrastructure serve as templates and extension bases for new acceptance tests.
Iterations provide many benefits. They allow opportunities for injecting feature requests, shortening the time between a user’s initial request and when she sees it in the product. Iterations, combined with engaging a cross functional team in each iteration, keeps activities such as documentation and exploratory testing in step with development, shortening release cycles.
Finally, the engineering centric nature of XP encourages the application, and more importantly the underlying code base, to become ever closer to expressing the user’s intentions and needs. It is very gratifying as a developer when your code base almost anticipates new user features.
I think software teams should actively try to measure, monitor and lower the cost of change. The complete “cost chain”, from sales to development, needs to be included and managed. Agile practices, such as XP, will provide a lot of guidance for improving the elements closest to the code, while applying general agile principles, such as cross functional teams, iterations, high communication, emphasis on feedback, can help the other aspects not directly involved with development. An organization that can successfully manage and lower the cost of change will achieve significant competitive advantages such as lower cost and time to market. Organizations that can’t manage the cost of change will have trouble competing at all.
August 15th, 2005
I think in a software project there are varying notions of reality regarding the actual application being developed. They are:
- actual needs of the market or user base
- product owner’s vision of the application
- developer understand of the goals and requirements of the application
- code base and actual executable application
This is
represented in the following diagram:
The gap between these perspectives is normal, particularly at the start of a project. However, I think it is difficult for a project to succeed if the gap is not being closed; there needs to be alignment and overlap among the different viewpoints.
One of the motivations of agile software development, at least in my opinion, is to help close these gaps. The high bandwidth of communication, intense collaboration, emphasis on feedback and the importance of acceptance tests are techniques to communicate the assumptions, needs and current state of each different constituent. With each iteration, the gap should be bridged a little more.
This raises an interesting question: How can you measure or monitor the shrinking (or worse the expanding) of the gap?
I don’t think you can measure this directly. However, there are a variety of indicators that can help evaluate how well you are closing the gap.
Bugs are probably the best indicator. A bug represents a mismatch in expectations, understanding or communication. Bugs are reported when a calculation is incorrect as well as when a user doesn’t like the number clicks it takes to perform an operation. In both cases, there are one or gaps between what the user needs and what the application is providing; perhaps the product owner specified a requirement that made the user experience awkward or a developer missed a test case. If your bug rates aren’t going down it is an indicator that the gaps are not shrinking. I suggest assessing the cause for the constant (or growing) bug rate by evaluating which gap is the widest. Understanding where the communication / comprehension breakdown exists is the most important step in resolving it.
There are other indicators as well, such as the amount of churn in stories (requirements) and the overhead of specifying stories. If you are revisiting the same areas of the application frequently, whether the reason is user dissatisfaction or defects, there are gaps between the user and the product owner or between the developer, user and code base.
There are ultimately two reasons for closing the gap: delivering a product that users like (and will buy) and lowering the cost of change. One reason is a huge component of succeeding today, the other allows you to succeed tomorrow. When the gaps are small and there is alignment between the user, the application and all those in between, you will probably have a winning project.
August 9th, 2005
Previous Posts
