Something important to me as an engineer when working with product managers is that he or she can really wrap their head around what they want to build and clearly explain it using as much detail as is sufficient and necessary to start delivering functionality. Ambiguity is a major driver of frustration and resentment between these two key players within an organization and contributes to a lot of wasted time, money, and creativity. It's a skill to think through what features are key to a deliverable, capturing the details, and sticking to that list when working with engineers. It takes commitment to wanting to deliver the product with quality and good time while maintaining a positive team atmosphere. A common pitfall that I have seen over and over on teams is a product manager who comes up with an idea, throws it over the fence to engineering, and upon receiving a schedule proceeds to move on to other tasks for a month without regular engagement with the team to talk about how they can contribute to quality or timeliness. I have observed a direct logarithmic relationship between the engagement between these two parties and the success of most software projects. The key for the product manager is to find the point where the graph of interaction levels off and work with the team at that level and no more. Unfortunately it is all too common that this point is never reached.
On the other side, engineers (like myself) often have a way of looking at a problem, reducing it to a simple solution that handles the case put directly in front of them, and then offering a timeline with some padding duly added based on previous experience. We go on implementing our well designed systems, everything plays together well, perhaps we've beaten our timeline by 10% - 15%, and we are happy. What usually happens next is the product manager comes back and says "This is all great, but our users need to do X, Y, and Z with the product, and can we also add this feature before the deadline because it would be great". Welcome to the brick wall at the end of the road to elegant system design. Suddenly design decisions you thought made sense are a roadblock to delivering features and you're scrambling to rethink ideas you threw away previously.
Is there anyone here to blame? I don't think so. I think both sides laid out their plans, and then went off to work on them in isolation from each other, not considering that there was a deeper story to be told on the other side. Product managers don't often capture all the use cases because they haven't gotten their hands on anything to close their feedback loop. Engineers are implementing what they were told to, without any notion that it may not be delivering any actual value to customers because they only had a narrow, temporal view of the product.
This isn't an uncommon scenario, so does that mean it merely reflects a core tenet of software development? I don't think so. It's not difficult to come up with a reasonable solution after thinking about it a bit, but like learning to solve a Rubik's cube not everyone is willing to put in the time and effort to start climbing the learning curve.
To start, I suggest we start to turn stories (requirements) into a living contract between product managers and engineers. Contractually, they ask the product manager to commit to a specific slice of functionality. Alive because both sides accept that as deliverables are evaluated by the product manager, new functionality may emerge that displaces less important functionality. For their part, engineers should commit to two practices: edge-to-edge development and delivering entire slices of functionality.
Edge-to-edge development means when delivering a feature, how and where it's deployed, necessary infrastructure such as logging, monitoring, and instrumentation, and a way for it to be tested from the perspective of the customer are all in place. This requires more up-front effort before delivering the first feature, but the marginal cost of adding new ones falls quickly. Developing entire slices of functionality thinking about the system less as discrete parts and more as a vertical profile of all the parts working together to develop a feature to the customer. Instead of working on any particular piece (such as a backend system) in isolation and then trying to layer other clients on top of it (such as a web UI or a data feed generation system), each with their own functional requirements, survey the system as a whole and then pick a whole slice from the bottom up to a customer deliverable and build that. In doing so many of the assumptions from both the product and engineering sides can be validated or challenged as appropriate, and the deliverables can begin to converge on a the product your customers need.
