Management: Building teams...

If you’re starting with this post you may want to read this earlier post for context: Management: The job of a manager

Introduction

Over the years I’ve joined software teams as their manager that other folks have built and I’ve built some basically from scratch and then evolved, grown and shrunk those teams over years and releases. A lot of what I have to say about the experience has been said by others, but I hope to add some of my own observations that might be useful.

Inheriting or founding a team

The manager before me had been an intense micro-manager, they had required all the engineers to include them in every meeting, every discussion with any stakeholder, they cracked down on engineers and stakeholders alike when this rule was violated. They made every decision, parceled out their team’s work in dribs-and-drabs, they deliberately constrained creative and productive engineers, they rewarded loyalty above competence and ability. In other words they had set the bar low for me and I knew I was going to be very successful...

Most of the time, I’ve joined teams that other folks have hired and organized. Early in my career I took a “they must have chosen these folks for a reason so I should work with what I have”  approach. It turns out however that most teams have been formed haphazardly by people who were short-sighted and had no interest in the concerns of management or long-term team health.  After torturing myself a couple of times that way, I shifted to a model of “I need to evaluate this team and see how completely fucked they are and decide on the one or two things that I have to un-fuck immediately” approach. There is a corollary embedded in this statement: All teams are a little bit fucked and you’re never going to completely address all the issues. You do need to aim for a good operating temperature for the team that trades off performance, morale, stress, interpersonal tension, happiness, and satisfaction. It is often an impulse to try and fix everything all a once; you should try to resist the urge. The less common case is starting a team completely from scratch, all of the advice below applies, but the first concern is to recruit a good lieutenant. If possible choose someone you have worked with before, someone who compliments you in terms of strengths and weaknesses. This person will be a key partner in choosing all the rest of your team members and in establishing the team identity. You will make mistakes, hiring is hard, people are quirky, combinations of people sometimes don’t work out… so even your hand-built team will be a little fucked up… it’s a process.

Gaining the team’s trust 

The very first thing that a new manager with a new team needs to do is gain the trust of the team members. The first thing that I try to do is listen to the team, validate the problems they identify, and try to address one of the most important ones. This can come before any systematic or process changes. It can show that you as a manager are willing to work for the team, and that you are really listening because you are acting on their behalf. It doesn’t need to be the biggest thing, it can even be something that just requires some focused effort and attention. It is important that you do the work yourself. The second thing is to be transparent with the team, share as much information about what you’re discovering about the team back to them, along with your ideas about what’s important and what needs to be done. Then, you guessed it, listen… weigh the feedback of the team, adjust the priorities, adopt their suggestions if they’re better than yours. Finally, have integrity, for me this means being openly responsible for your decisions and actions. If you say you’re going to do something, do it. If you make a mistake, take responsibility for it. If you need to change a plan, say so and explain the reasoning.

The un-fucking process

As the new manager from the United States, from a just acquired company, from outside Europe, I didn’t have a lot of credibility with the team to start. I spent two weeks in-country with the team and talked to each team member individually and observed their interactions over the course of a release of the web service we were building. I didn’t change anything, I didn’t offer any opinions, I just watched them… Several things became obvious to me: their technical leader was incompetent, they had no experience building large scale web services, they weren’t competent to make the changes to the open source software we were using that they had made, they had no repeatable process to create a release and verify it’s quality, they had no logging or metrics coming from the production deployment and so ( it turned out ) they were missing the fact that 60% of requests from clients were failing, the architecture of their web service was a complete non-starter. I built all my future team and project plans on correcting these issues…

What should you choose to un-fuck first? Different teams will have different problems but first and foremost if the team is under-performing on their goals for the company, you should address that. Watching the team try and do a release is often instructive for diagnosing this sort of problem. Do they make a plan? Does everyone on the team have a role in the plan? Do they follow the plan? Is the work distributed in a reasonable way by role and by level? Are there people who hinder the planning or execution with discursive discussions, rat-holing, or introducing tasks that aren’t required? Does the amount of work in the plan make sense for the capacity of the team? Do they have any idea what their capacity as a team is? Do they just turn to you and expect you to plan everything?

What you want in a team is a group of people who take responsibility for the product or sub-system that they are tasked with owning and delivering. Based on your observations, find the real competent leaders on the team (or hire one) and empower them. If there are people who are corrosive and are breaking the team, get rid of them, even if they are “the guy that knows everything.” If the team has no process and no plan, help them to figure out what they want their process to be and to follow it. Do not dictate it, do not volunteer to operate the process. If there are people who are hanging back and using chaos to do little or no work, give them direct feedback, call them out at task assignment time, and if they still put in more effort ducking work than doing it, replace them. If there are people outside the team that are removing responsibility from the team by telling them what to do, work hard to redirect those people and get rid of them. If there is a yes-person who is signing the team up for huge amounts of work that they can’t possibly complete, stop them and reduce the expectations on the team and reset expectations about team output. Nothing will cause a team to check out faster than being repeatedly given impossible and, in the end, meaningless tasks to do and repeatedly failing. Fix these things before you even consider growing the team.

Setting the tone

I watched the two supposedly most senior technical people on the team regularly engage in yelling matches as the team sat by in stunned silence. It was probably going to get to a physical confrontation in the near future. The prior manager had not only ignored this behavior, they had often joined in… It turned out that neither of these folks were the right technical leader for the team, one of them I essentially fired and the other I sent to a team that had a complete void of leadership and technical experience. This created the space for a calm, productive, talented engineer on the team who had been doing all the real work behind the scenes to step forward and set a completely different tone.

Once you’ve got the team stabilized, you can look at team dynamics and behavior. A healthy team supports all the members of the team; nobody needs to be “assigned” to help anyone else, they just step up and do it. Even tense discussions should be civil and focused on arriving at a good solution based on facts and real requirements. The most common pathology on teams with regard to dynamics is hiring too many “big dogs,” people with lots of technical prowess and experience, and packing up the team with them. The problem with this in a group of humans is that inevitably, because there is nowhere to progress to, they tend to turn to snarking and sniping at each other and arguing over minor issues way past their actual value. Teams ideally have one or two “big dogs,” who are chosen carefully based on their ability to provide high level control and structure to others, and who are good communicators, good teachers, and good engineers. Then a mid-level of good, less experienced folks take up the majority of the team and a few talented entry-level folks. The actual proportions depend on the sort of work the team is doing.

Teams delivering software products or long-lived web services can skew towards the middle tier; inevitably, the most experienced folks will move on and ideally you will replace them from the top performers and leaders of the middle tier. Teams building custom solutions can sustain a lot more entry level folks and probably should; the work is repetitive and time-boxed. The middle- tier and top-level folks should focus on making the underlying platform efficient and resilient in the face of front-end churn.

When you observe bad interpersonal behavior (or someone comes to you about it or you hear about it in a 360 review), take action to bring the parties together in a discussion. Encourage folks to speak up and discuss interpersonal issues that they have directly with the people that are causing them. Provide mediation, where appropriate, to teach folks how to have a constructive discussion about an uncomfortable situation. In some cases, where intervention, discussion, and even direct feedback don’t solve an issue, removing one or both of the folks creating the behavior problem is appropriate. Of course this above and beyond issues like sexual harassment, which has a very clear process associated with it and which you should follow immediately and in all cases. Of course the opposite is also true: Reward and praise good behavior and encourage it whenever you have an opportunity.

Say out loud and in writing how you expect team members to behave with each other. Often, managers are afraid to talk about behavior, feelings, interpersonal matters: get past that. Everyone comes to your team with a different background and a different life experience, so they all need to be told what the expectations are. Back up these guidelines with appropriate actions, up to and including terminating people. One person can damage a team with their behavior and make everyone else on the team lose performance and fail. Nobody is worth looking the other way for, regardless of their skills or inventiveness. In order for any product to succeed, it needs a team that is going to sustain it for the long term.

As teams change, they often go through periods of dysfunction and chaos until a new team dynamic is established. It may take changes in structure or leadership or just some time for it to settle. This is why you should always be evaluating your teams as you did when you first formed or inherited them, looking for the same things, and addressing the same issues. Teams are always a work in progress. Another classic trap is the belief that your new 20 person team is going to look and sound like your first 5 person team of scrappy entrepreneurs. Ain’t.  Gonna. Happen. And it shouldn’t. What worked for 5 people with no legacy, customers, or competitors doesn’t work ten years later with all of those things and more. Examine whether they are now effective for the company, operating in a healthy sustainable way. It doesn’t matter if it looks different from the five patron gods of the company and their mythological deeds.

Getting bigger

Several of the most senior members of the team were dilettantes, they only worked on things that they wanted to, they didn’t participate in the team’s regular processes, they checked in code that wasn’t reviewed or understood by anyone else on the team. I worked to dis-empower them, it was a delicate process, like a lot of folks that have been in a position for a long time there was a mythology about them that they were indispensable among the other executives. One by one I managed them out of the team, often with them believing that they were attaining an even better position. I replaced them with a well balanced group of people who could collaborate and were willing to do all of the work of the team. The overall productivity of the team grew and morale improved… within a year the dilettantes were gone from the company.

When you’re growing a team, think carefully about what that specific team lacks and put that on the list. Of course you want to hire very qualified folks for the level you’re seeking, but you should also look at the personalities on the team and be on the lookout for folks that would help balance those personalities. For example if the team is all mercurial, less experienced team members and they often build overblown solutions very quickly (that later take hundreds of patches to stabilize and make work), adding someone who is more experienced and more methodical at a higher level can help shape their productivity more constructively. If a team has too many big dogs, look for a couple of middle tier engineers with a good record of consistent delivery and then trade a big dog to another team and backfill to balance the team. 

I recommend aiming for gender parity on your team; having a mixture of people who are socialized differently creates a strong cohesive team and helps a great deal with the atmosphere on the team. A very simple mechanism for this is to scan every batch of resumes you get for women, when you see qualified applicants that are women insert them into the process first. This works for other under represented groups too. You also need to avoid and root out systemic discrimination, questions or concerns that are only raised for women or other under represented group applicants. This often takes the form of code language like “too junior,” when the person has done and is doing the exact work you’re after, etc. The final step, you have to hire them. If you make it a goal and reinforce it as a matter of performance for yourself and your managers, you’ll be surprised how well it goes. Your interviewers should include women and minorities from your team. This reassures minority candidates that they won’t be alone on your team and it gives them someone in the interview process to ask questions about the team culture. It’s also a good social test for all candidates to see if they have issues with the women and minority members of your team. Several times we’ve passed over folks who decided it would be good to cut off and talk over our women interviewers and talk down to them when pressed.

Team morale

When I took over the team, one of the founders of the company had been “helping” what was perceived as a team in trouble. The founder’s model of how to help people was to give nebulous directions and then argue with people when they misinterpreted this direction or when they pointed out obvious conflicts in the direction. The team never received any praise, just constant criticism for not being engineers like the founder. Of course, the founder had no experience in the area that the team was working, or the technology… but, this didn’t dent his confidence that he could tell them what was wrong with them. The team was paralized, very few features could get completed because they were mired in senseless circular arguments. I started a process with the team to define who they wanted to be, what were their goals, what was their idea of quality, what were they going to deliver for our customers. Several iterations of this document engaged everyone on the team in forming a vision of themselves and how they wanted to be. At the same time I worked to manage the founder out of the process, noting that in our fast growing company there were many more vital projects that required the founder’s genius. I praised the team every time they lived up to their goals and standards and the morale on the team improved steadily.

The best team morale/cohesion-building is simply working together on meaningful problems. Morale is highest when the team feels in control, successful, and recognized. If your team doesn’t own what they work on--for example, if an overly controlling product owner insists on defining not just the business requirements but the solution, the architecture, and the technology choices--then no amount of building LEGO structures together is going to fix the disengagement that results. Consultants like these little exercises because you can basically take any arbitrary group and make them feel like they worked together and succeeded. That is mostly because: a.) the objective means nothing and failing means nothing, b.) the problem is simple and bounded, and c.) none of the decisions have a lifespan beyond the session. Apart from maybe being a game to play while you’re drinking, these exercises don’t have any impact on the real workplace. I do think that creating contexts for everyone to participate in planning, process decisions, and implementation decisions is good. By this I don’t mean everyone needs to agree or work on everything; instead, I suggest that when you are forming task groups, insert folks from other disciplines and levels into those groups so they work with team members that they wouldn’t normally and on topics that aren’t in their everyday workflow. This expands everyone’s knowledge of the product and what goes into building it, exposes them to evaluating and understanding requirements, and creates social bonds.

Don’t take engineers’ ideas about how to run the team too literally. Many engineers have naive ideas about how humans work and often devolve to trying to program them like a distributed system. Their ideas do have value as signals about what needs to be addressed in the team. All processes that the team adopts need to be humanized. Humans are not machines; the most consistent of them will vary enough and often enough to matter. They will respond to outside forces from their personal life, they get tired, they burn out, they have subjective responses to people and situations that are unique to them… Therefore team processes have to be fault tolerant and have mechanisms for fault detection and correction.

Getting to the right size

I had worked to build one of my sub-teams and fix it’s culture and staff it appropriately for it’s mission… but they were still struggling to deliver. I realized that their main stakeholder still treated them like outsiders, ignoring them until the last minute, not sharing plans and information, not considering the impact of their decisions on them. I decided to advocate that we move the team into the stakeholder’s organization, of course I had to make it seem like their idea… which I did, and the organization change worked wonders causing the team to gain focus, momentum and productivity. One could look at it as me losing talented engineers that I had recruited at great time and expense… On the other hand I now had a whole group of folks that liked me embedded in our stakeholder’s organization…

I would note that I talk about firing people a few times above, but in reality if you consider these issues as you grow your team, there should be little need. In a recent role, I grew a team from <20 people to >100 people in three years, I terminated five people for various reasons relating to performance or behavior over that time. The team had some of the best engagement scores and productivity of any that I have ever managed. Hire good people, build teams thoughtfully, and you will have a great environment filled with strong leaders. 

Reducing the size of the team is sometimes called for. Products don’t work out in the market, companies make bad investments that catch up with them, and organizations need to be restructured to deal with macro-economic events. In some cases, you are allowed to choose the folks to retain in a restructuring. Think about all the topics above as you plan your restructuring. It is tempting to always choose just the most experienced, and most proficient folks, but doing this will result in a team that is out of balance after the restructuring. Reduce the scope of the team’s responsibilities as it shrinks. I have been asked many times to just go on with everything and do it with half the people. I just gave them back a list of the things we’d be failing at, which was also a good list of things that could be cut if they decided to be reasonable. Sometimes you get asked to lead a team that is already quite large and when you do the evaluation of the team you realize that it is too big and very wasteful. It is tempting to just add the new size of team to your resume and let the waste roll on, but teams like that are often riddled with behavior and morale problems, and eventually companies detect the waste and you are as likely to be disposed of in the restructuring. It is better to cancel wasteful projects, send sub-teams to some other part of the company where they can be productive, and, in some cases, even lay off folks to get the team to a healthy size relative to its real productivity and value to the company.

Conclusion

Finally, I would counsel managers to not meddle too much in the team. Even if there is tension on the team, give it time to be worked out by the team. If you do have to intervene, try as much as possible to do it by coaching the team in a process to identify and resolve their issues. The more that you insert yourself between members of the team, the less of a team it becomes. This also means that you should think about your own transition, try not to make every process dependent on you personally, this will limit the impact if you decide to leave the team or are promoted or reassigned.

Management: Getting the most out of your team's brain matter...

( Continued from: Management: The job of a manager… )

Their division volunteered for every project and everyone there “sometimes” worked on any or all of the projects. Their manager collected teams and subordinates unrelated to any business purpose. The division grew and grew and they proudly touted how many projects they had and how many people. The company praised them and raised them up as an example to other divisions. When an economic downturn occurred, coupled with strong competition, management showed up looking for things to cut. When they looked at this “wonderful” division, they found much that the company could live without. They cut it by two-thirds and then, a couple of quarters later, disposed of it entirely. Many people on the team were shocked: They had stayed with this group for years being unproductive and, in many cases, their skillsets became unmarketable. Now they had to find a new job.

My overall model for deciding what to do with my teams is to think “am I using these folks’ skills and brain matter as effectively as possible?” This means having them help create the goals, solve the problems, make the plans, make most of the decisions, come up with the ideas, come up with the processes, and get the credit for the work. The more that I find myself doing those things for a team, the more I feel like I’m wasting their abilities. The more engaged a team is in the whole process, the more productive and efficient they become. This involves trusting your team and being prepared to back them, even when they make mistakes, but it pays off every time. If you focus on the roles I described [ see Management: The job of a manager ], and give the people in those roles good information and guidelines, coach them and give them feedback, and give them the team members they need, they will take over and drive these things better than you ever could on your own.

Always be looking for emerging leaders on your team, and work to support and empower them. They don’t have to have a leadership title; they can just be the person who provides the drive to get stuff done. Reward that behavior and feed that inclination. Then when your team grows, you already know who to turn to when you need to fill leadership roles.

The company didn’t have a process that you could name; they had a couple of people who indoctrinated new team members in “the way.” When you did something that wasn’t “the way,” one or more old timers would correct you. We sat in a circle and coded together and released the software when we thought it was ready. We released very high quality, high performance software on a regular basis...

I have deliberately not made any specific references to formal processes, because I don’t think they matter much. As long as your process provides the basic elements of a software development life cycle, and gives you as a manager the information you need to evaluate your team and give them feedback, it’s a fine process. It does need to be a process that the team buys into, documents, and follows. A process that changes every week or every release is not a process. Ownership of the process is key, for as soon as the team feels the process is imposed, you have lost a large amount of engagement and productivity. The other thing to guard against with a formal process is that it ends up replacing customer value, business value, and the product as the goal that the team is driving towards. I’ve seen scrum teams with the “perfect” burn-down chart that produced nothing of value to the business for multiple sprints. It’s easy to succeed against artificial metrics that mean nothing to the product.

The manager’s flying monkey came into my office and shut the door. “You’re making a big mistake,” they said. “It’s going to be the end of your career if you don’t get on board…” I had spoken up at a review meeting saying that I didn’t think a product that they desperately wanted to ship was ready. I had a track record of delivering excellent results and doing it fast, while bringing the rest of the team along with me. I laughed at them; it was the most ridiculous and transparent bullshit I’d ever seen. I said, “please do your worst. I can always get another job. Now get out of my office.” Three months later, they were giving me a $40,000 bonus to try to entice me to stay on their team.

I talk a little about politics in passing in order to highlight the things that give you as a manager the most political leverage. Delivering value for your company—and being able to point to it and a cadence of delivering value—can protect you against the worst of incompetent executives. It is very hard for greedy CEOs to ignore your team producing things that make money for the company and listen to a cranky executive that you’ve annoyed tell them you aren’t a “Team Player.” Focus on delivering value and motivate your team around that, and the rest will follow. Or it won’t: Some companies are so profoundly broken that they are entirely driven by ego and vanity. Deliver value while you look for another job—that’s what sounds the best in your job interviews. It does you no good to say that you spent all your time arguing with (and resisting) venal executives and that you failed to lead your team to produce results.

“Don’t manage your peers’ teams” is another corollary in this vein. Figure out which of your peers is good at their jobs, then determine whether you can trust and/or defend your team from the rest. Trying to correct your peers’ flaws, organize their people, etc., is a self-defeating process: It ends up with your team not delivering and you’ll get no thanks for interfering with another team. Sometimes, however, it’s the lesser of two evils to have your team building things or solving problems that aren’t their stated goals, because it protects them from depending on failing teams. If you are asked for feedback, give it: be honest, be factual, be constructive, but don’t volunteer to solve their problems.

“Don’t be greedy or possessive” is another guideline. If you have people on your team that can be utilized better somewhere else, offer them up. Don’t grow your team for the sake of growing it; make sure that the folks you hire will deliver for the company. If something extraordinary comes up in the business cycle and you’re called on to disrupt things to fix it, don’t complain: just do it and get back to normal cadence as soon as possible. These aren’t “your” people; their contract is with the company, and so is yours. If you remember this and act with integrity, you will gain influence and trust.

When I talk about having a contract with the company I don’t mean to suggest blind allegiance, but a recognition that you’re taking money from folks and they have a right to expect you to do what is reasonable for your role. Things like lying, committing crimes, defrauding the company, covering up mistakes, supporting the abuse of employees, enabling abusers, participating in a discriminatory process, etc. are NOT part of that contract. Depending on the company, these things may come up to a greater or lesser extent, and should be prepared to do the right thing. You cannot recross the bridge of doing something unethical regardless of the pressure from above: Once you do it, everyone knows it, and you’ve lost trust and respect. Again, you’re better off looking for another job than ever compromising on these things.

Also, companies are sociopaths. That doesn’t mean that they’re evil, they just don’t care about anyone’s emotional state or livelihood but their own. Part of your job as a manager is to discourage folks from engaging in magical thinking about the company: “We’re all a big happy family”, “we’ll take care of you”, “we’re all in this together,” for example. Employees are worth what they generate (or what they are perceived to generate) in revenue for the company, so I always encourage folks to consider their productivity and where it is being directed. When a project seems like a fantasy boondoggle that will never make money, nine out ten times it is, and the people who go to work on it end up looking for a new job. They might learn great things on that project and get great experience, but they should be cold-blooded about the likely outcome. Every company goes through ups and downs, sometimes very dramatically. Having a loving and caring culture while on the upswing is easy, but the downswing is usually where the reality of being a sociopathic business entity with detached investors becomes evident. Again, the best thing you can do as a manager is to encourage your people to be flexible, productive, and skillful; this will make them the most attractive employees to retain in any restructuring.

Finally, you’ll note that I mention incompetence. It’s part of the reality of large groups of humans doing anything together. You should always approach anyone you work with with the initial belief that they are competent at their job, and continue like that until they prove that false. Incompetence gets into organizations because people often make hiring and performance decisions based on things other than fitness for the role or actual job performance. Basing these decisions on personal loyalty, nepotism, friendship, giving titles to justify pay increases, and other motivations promotes incompetence. It often gets into management because folks become managers by default; I can’t tell you the number of manager candidates I’ve interviewed who said “nobody else wanted to be manager so they picked me.” In good organizations, it eventually gets exposed and corrected, but that process can take a long time. You should be aware of incompetence and where it exists in your organization, and keep your team and its projects from being damaged by it. Also, if you’re not competent, do better—or remove yourself and do something you are good at. Life is too short. Recognizing your own strengths and weaknesses, and being honest with yourself about them, allows you to build a team that compensates for your shortcomings, and lets you focus on your strengths.

Management: The job of a manager...

I recently retired after 38 years in the software industry. I have been a software engineer, architect, chief architect, team lead, development manager, director of engineering, and VP of engineering. I’ve worked on and led teams from <10 engineers to >100 engineers at companies like Lotus Development Corporation, IBM, Intuit, Nokia, Elastic, and a bunch you’ve never heard of. I’ve built software for everything from 8-bit microcomputers to Cray Supercomputers and shipped it in boxes, downloads, web services, cell phones and Linux appliances to customer bases ranging from hundreds of specialists to millions of consumers. I’m sharing these retrospectives just to say “here’s my perspective…” I think there are lots of ways to succeed in the complex business of software engineering, these just describe my approach..

I have been asked more than once: “What is the job of a manager?” All of my management experience has been managing software engineering teams. So, my answer is in terms of that field, but I think a lot of it is the same across fields. The HR department might have a whole bulleted list of responsibilities for managers, but I don’t often refer to those. My answer is usually a combination of saying what a manager does and what they shouldn’t do. Often the things you don’t do are very important.

Their manager had created an enormous spreadsheet, listing hundreds of product features, ranked in some arbitrary way. “Here you go, these are  your priorities” they said. There was nothing to tie the features or the ranking to any business goal that could be measured. Weeks later the manager was frustrated saying “Why did they choose to do THOSE things? Those aren’t important.” The team said “See, we ticked off these things from your list! They were easy to get done.”

I think the most vital role that a manager has is to communicate the team’s goals and the relative priority of those goals to the team. This is hard to do well because it involves understanding the reality of the business and organization that you’re working in. It is not as simple as just listening to your boss and passing down whatever they say. You should understand your team’s stakeholders needs and how high priority they are, how much business impact they have, and where your team is in terms of satisfying their requirements. The team itself is a stakeholder, often generating goals internally that need to be aligned with the wider business.

A manager needs to at some regular interval update the team’s priorities and goals and reflect back to the team where they are in terms of accomplishing their goals. It is critical to give ownership of the “how” part of accomplishing the goals to the teams almost entirely. It’s fine to apply “taste & discretion” or “guidelines” to their work, but the manager should not delve down into implementation details. The manager should ask their teams for a plan for how they will address the goals, and that plan should include what is going to happen, who is doing what, and what order things will happen; sometimes they also include time estimates. A manager should provide feedback and coaching on the plan, seek to reduce scope and risk, and question unnecessary work. When the work is in progress the manager should look at how the work is tracking to the plan and suggest re-planning if new information supports that.

All of this planning and feedback should happen in the context of the set of goals and priorities and the goals should support the team’s decisions. If you find yourself or a team going away from those goals, then it’s time to re-evaluate the goals.

A new manager comes to take over an existing team, they’re informed by their boss: “This team is all messed up, you’ve got a lot of performance problems. Here is a list of the people you should get rid of…” The new manager asks “Have you given them any feedback about their performance?” Their boss responds “They know what’s expected of them…” In other words, “no.” The new manager reviews the performance of the people in question and gives them appropriate feedback. The majority of the team improve their performance immediately. Some of them are managed out of the team. The rest of the team gets regular feedback and they steadily improve. 

This gets into the second highest priority of a manager: giving feedback. Once you’ve been clear with your teams about goals and priorities, and they have given you plans and status, which you’ve reviewed, you’ll probably have great content for feedback. First rule of feedback is: “Don’t wait!” As soon as there is an opportunity to learn something together, it is time to talk about it. The punctuated reporting schedule of HR systems is not a good structure for managing successful projects. 

Feedback should be 75% praise, 25% critical feedback. If you can’t come up with 75% praise, you probably have a real performance or morale problem. The 25% critical feedback that you give should be for the highest priority things that are impacting the effectiveness of a team or individual. These should include one to three points, with accompanying suggestions for how to take action as a prompt for the person or team to suggest their own ways to improve. The critical feedback should be something that is measurable in a goal or deliverable.

Solicit feedback regularly from the team or individual’s stakeholders and summarize the trends in that feedback, both positive and critical. Often the best language and suggestions for describing improvements come directly from stakeholders (including peers). Be careful not to overweight feedback based on the age/rank/level of the person giving feedback. Similar feedback from multiple stakeholders is much more useful than cranky feedback from an executive. Of course you should monitor that crankiness and perhaps discuss it with the executive, to try to keep that executive from developing a grudge against your team or employee.

The CTO would rigorously review every project schedule with dedicated project managers, schedules, and task lists. The same CTO would drop in on the most productive engineers and ask them to build things for them, speculative projects that didn’t appear on any plan. Projects would then miss their dates, deep concern would be expressed, and pressure applied to correct the issue… The speculative project would be sidelined and never shipped. The cycle would repeat.

The third thing is to drive a cadence in the work of your team and protect them from chaos agents. In some companies that are very early stage or prototypical startup efforts, chaos is the normal environment, but there are processes to focus and control that chaos and as a manager you should work to do that. The key thing is to allow your team to complete projects and have the satisfaction of delivering things to the market. This isn’t as simple as saying “no” to everyone; it involves working with your teams to determine a structure for how to acknowledge, prioritize, and sequence the new work that inevitably comes in. The other thing to fold smoothly into the workflow is technical debt or maintenance. This needs to be a high priority and carefully chosen, so that it doesn’t end up completely overwhelming the flow of work at random, unpredictable intervals.

Supporting your team in resisting distractions when they come in (from executives with a great idea, internal teams hacking up competition to their products, etc.) is a key part of this process. There may need to be some reserved capacity for ego maintenance of executives, but like a lot of political capital, it needs to be carefully managed because the backlash will be even worse if the team fails in its core mission for the company while delivering an executive’s pet project. Usually engaging executives in the boring process of figuring out all the trade-offs required to make room for their project can make them wander off to bother a different team.

Everyone who applied and passed the phone screen by the recruiter was sent to do an online coding test. Anyone who got past the test was interviewed by the team. People were often rejected after seven interviews, seven hours, by at least seven different people. The team’s hiring rate was very low and the people they spoke to were often not senior enough for the role and were being rejected at the final executive interview. None of the interviewers were clear on what they were interviewing for and what criteria they should use; they often passed people along figuring the next person would reject them. Sadly, the automated coding test was causing some of their best candidates to be offended and to simply opt out and take a different job at another company.

The final key is to make sure that you have great people on your team and that they are the right people for what you’re doing. This involves hiring as well as performance management and internal staffing. You should be clear on the profile of the folks the team needs, both from a “skills and experience perspective,” but also personality type and social fit. The hiring process should be as short as possible, but comprehensive in having at least a couple of people check out each attribute you’re looking for. I personally like combining an interview step with multiple goals; for example, an interactive coding session with two engineers and having the candidate work through the problem with them and interact with them in a similar way to the way the team would work. This can answer questions beyond “can you code?” and get way into team fit.

Like any process, you and your team should examine your hiring process at regular intervals and ask if it is delivering what you want. For example, it is expensive to have your team interview and then not hire people. So, the more that candidates can be screened at the resume screen or at the initial phone contact, the better. Likewise, if you have an executive approval phase in the interview process, you should have addressed all of that executive’s concerns before a candidate gets there, because that is the most expensive time to have someone exit the hiring process.

“Look, “ I said “you are very good whenever we have a support escalation, I get messages from customers praising you and marveling at how passionate you are about solving their problem. On the other hand, you struggle to keep up with even our most junior engineers, people have to step in and correct or redo your work. You are visibly unhappy whenever we talk about your job…” They had already started sobbing. I continued, “Why don’t we talk to the support organization and see if they have something…” They look up in amazement. “You’re not firing me? You want to help me?...”

Performance management is the other side of the coin, and you should act promptly when someone is under-performing. Effective 360 reviews and close examination of status vs. planned assignments will help identify team members who aren’t meeting their goals. Finally, metrics on code base productivity can be useful to support or refute a performance issue. Understandable feedback (along with clear explanation of the consequences of not improving) is crucial all along. Again, don’t wait for a review: As soon as you know something is going on, give the feedback. There are HR processes for dealing with folks who can’t listen to feedback and respond to it. Don’t be scared of them—work with your HR person and replace people quickly who are not performing well. Even in the best teams, a poorly performing person is a drag on morale and will impact the whole team.

A different problem is having the wrong person for a role. Once you recognize this, start a career path discussion with that person. Sometimes they can find a role in the company better suits them, and you can replace them with someone who is a better fit for your team. You can usually differentiate this situation from other performance issues because the person will sincerely try very hard to respond to feedback, but will swing back and forth, first getting better, then worse. They will also show symptoms of disengagement, lack of attention, lack of communication, etc., so it’s vital to suggest changing roles for folks in this situation.

I talk about some organizational context for these four parts of a manager’s job in the next post: “Management: Getting the most out of your team’s brain matter…”

Digital archaeology...

I was going through a drawer in my file cabinet and came across a bunch of 3.5 inch floppies. These are now approximately 28 years old and approaching their expected use life of around 30 years. I thought “Why don’t I get a USB floppy drive and see if I can back up any documents and such that are still on these disks?” So, I did.

I was going to just attach the USB floppy drive to my Ubuntu machine and do it all from there, but for some reason Ubuntu really didn’t like the floppy drive and it was making bad noises and not working at all… So, I dug out the emergency Acer Windows 10 micro laptop that I keep for just these “I need an actual hardware device running Windows to do this thing…” sorta moments. Windows 10 was very happy with the floppy drive and I started to inspect the disks. Before a single file showed up I was transported back 30 years, the sound of the disk slotting into the drive, the shingggg noise as the metal shield pulled back from the media, the whirring and clacking of the disk spinning and the head seeking all sounds that I haven’t heard on a regular basis in decades.

Many of the disks are backups of Dana’s archaeology work, papers, databases, and her dissertation from grad school and beyond. So, digital archaeology of actual archaeology… I immediately ran into some interesting file format issues. At the time in the 1990’s I was working for Lotus Development Corporation (best known for the popular and best spreadsheet program 1-2-3) and so Dana was using Lotus software on DOS to do her work. The first word processor was Lotus Manuscript ( gone into the depths of time, only a Wikipedia entry to mark it’s passing ) and then Lotus AmiPro followed by Lotus WordPro ( similarly gone ) when she moved to Windows later. There are no import filters to read those file formats because Lotus lost the office suite wars with Microsoft and IBM had no interest in that market post acquisition.

Thank goodness for retro-gamers, they have built great emulators for both DOS and early Windows so that they can run vintage games. Because games tend to max out all kinds of capabilities of these environments, the emulators are more than adequate to run original office software if you can get disk images of it. I happen to have original Lotus SmartSuite 97 CD-ROMs (I worked on 1-2-3 for Windows 97 so I got a box of product when we shipped it, back when software came in boxes). I installed it under PlayOnLinux (a wrapper around the WINE windows API emulator for Linux) and very shortly had the applications running. They are VERY FAST on my 16 core 3.60 GHz desktop, the fastest computer of the time was around 50 MHz, and they were written to be fast on those, so even in an emulator which must have a lot of overhead they scream.

Because Lotus was always a very customer oriented company, we actually shipped a high fidelity file import filter for Lotus Manuscript 2.0 in Lotus Wordpro 97 and it works a treat to read Dana’s oldest documents.

Next problem was a database that Dana used for cataloging all of her artifacts and creating the tables and appendices for her dissertation. It was called Q&A from Symantec and we chose it at the time because it had a really simple form builder that let us model the same recording sheets she was using in the field while excavating. We were on the cutting edge at the time… So again, you’ve never heard of Q&A and that’s because it was crushed by many other more popular databases including one from Microsoft. Fortunately we still had the disks… I just copied them all into a folder on Linux and used Dosbox ( one of many MSDOS emulators ) to run the program… and it just works… it came up and read the original files and could query them and generate tables etc… very cool.

There were some disks that contained files created by the MSDOS backup tool… These I can’t get to restore, I may figure it out eventually, but even running actual MSDOS 6.22 in a virtual machine I can’t seem to get the restore program to read them. I’ll keep digging, there are hints out there on the interwebs that folks have figured this out.

I found a couple of disks that belonged to me and that contained backups of source code for a program I had been writing from 1991-1993 in my own time. It was written originally in Borland Turbo C and some assembly language. So strange to look at code 28 years later, I immediately had all kinds of memories come back of where I was and what I was doing then. The program was building a .COM executable which modern Windows won’t even load. These programs were essentially a single 64k code segment and a 64k data segment, no fixups, no dynamic linking, just a code image that got plopped into memory and executed. Most software nowadays doesn’t even draw breath in under 64k… And the program still runs nicely in Dosbox and does it’s thing like it was yesterday.

Now these files are getting safely backed up for the next 30 years along with the software to read them. I think there are some more floppies lurking in the basement, I may dig them out and do some more digging…

Upgrading, music projects etc...

The last week was taken up with upgrading Ubuntu on all of our house/work machines. Going from 18.04 LTS to 20.04 LTS ( and in one case a laptop that was hanging around on 16.04 LTS… ). In general there weren’t any major problems, though there was a bug in Libreoffice Writer that was very bad, so bad I’m surprised they released it, essentially embedded tab characters didn’t work any more, they didn’t indent, you couldn’t enter them, they had no effect, it appeared like they were stripped on file open, though they weren’t. Libreoffice patched this bug, but their patch releases don’t end up automatically upstream in Ubuntu so I had to install the patch manually. The other wonderful quirk is that their patch release doesn’t upgrade the packages installed by the distribution, they install next to it… this forces you to make a whole bunch of changes to launch the correct version. I was not impressed.

I’ve also been working on a music project, I am trying to create some videos with words, imagery and music that I’ve captured, created. This week I was working on themes and writing some of the words down that I’d like to also communicate with the music and the images. That process is helping me with editing and “composing". I put composing in quotes because I’m a very untrained composer with a limited grasp of music theory. What I have is some training in improvisation and a loose understanding of chord progressions, scales and structure. I have been building on that to come up with the structure of the piece and I’ve been recording different sections of it and I plan to work on editing/adjusting those down to a final product.

I’m using google apps presentation software to automatically display the images annotated with words and cues so that I can play and respond to them in real time. My goal is to get to a “draft” finished product so I can understand it’s limitations and then iterate. I’m also getting in a lot more practice and that is in itself a good thing. I tried a lot of different recording/DAW software and I finally ended up going back to Band in a Box 2011.5 (the last version I bought) because it provides some excellent automatic accompaniment bass, drums etc… in multiple styles, and actually has some really nice recording features and a decent notation editor. It runs on Windows, so I have it running on my laptop in a Windows 10 VM and amazingly it keeps up just fine and is able to see the MIDI and Audio devices mounted in Linux. The accompaniment helps me a lot to follow the structure I’ve set up and to stay roughly in time ( I need all the help I can get ).

The process I come up with is probably going to be quite unconventional, but I guess that will be part of what makes it my expression.

Software Project: ped-editor v1.2.0 shipped!

A new feature release of the editor. The feature is that either when you pipe content into the editor like “find ~ | ped” or you use the F10 shell command dialog the buffer will update in the background for as long as the command produces output. You can press Ctrl-F in that buffer and toggle following, i.e. the screen will automatically show you the end of the stream and the latest content. If you want to follow a file on disk you can use “tail -f filename.log” as the command and the resulting buffer can be used to view this information. All of the regular read only buffer commands are available, copying content, searching with regular expressions, etc…

There are new tests since the underlying StreamEditor class is used in the StreamSelect component and so the existing tests caught a lot of side-effects that I had to clean up as I did this. I have to say having sensitive tests has made this whole thing a lot safer, I probably would have pushed my first cut on this in the past and then discovered the problems later. Now, it all works and is covered with regression tests.

I’ve also updated the wiki usage documentation with up to date screen shots and more information about how the editor works.

Software Project: ped the editor, v1.0.0

After all the work getting the editor ( https://github.com/jpfxgood/ped ) to Python 3 and getting the test coverage to 78% and fixing a lot of bugs, I also wanted to do some improvements. I worked hard on getting the optimal redraw to work and the terminal cursor was still being a problem so I turned it off and now render my own cursor. I also had to redo a lot of the code that managed multiple views so that they would render efficiently and also keep their cursor positions when you changed frames or current files and updates in other windows are properly reflected in all the windows. I also made it easier to see which frame is current when there are multiple frames and cleaned up a bunch of over refreshing in the dialog code as well.

I made it so that editing a new file doesn’t create the file until you save it and also the editor now warns you if the file you are editing has been modified on disk.

I optimized the code for recoloring the current file and now it runs very fast reducing or eliminating any flashes when it has to catch up.

So, given all the improvements and the testing I decided to mark that moment with a v1.0.0 release. I’ll try to do patch releases relatively frequently as I find and fix other things ( two seconds after the v1.0.0 release I found several things and fixed them…)

It’s been a lot of fun getting this code cleaned up and improving the testing, I’ve spend hours at a time in a kind of flow state just coding and debugging. I even forgot to get lunch a few times, which I never do…

Software Project Python 2to3 continued... 66% coverage, >75% in the core editor...

This has been a very healthy process for the editor code, as I’ve been writing tests I’ve also rewritten code to make it much better than when I originally hacked it up. That is the thing when you’re in a hurry, often you’ll stay on a bad implementation track because you’re “close” to getting it to work. It is nice to go back and have the time and focus to reconsider a bunch of things. The tests themselves are interesting, for example I had to figure out how to resize the terminal window to test the code that resizes all the editor panes in response. That code always annoyed me and had a lot of bugs, now it is rewritten and it is very stable.

I’ve been using the python coverage tool to measure the coverage and it has also helped point out a few modules that aren’t used anymore so I’ve been improving coverage through deletion as well. I’m going to try and get the overall coverage over 70% and I need to make another pass over the minimal redraw logic as well… It still has some odd behaviors…

Then on to the backup and sync scripts that I wrote, it’s going to be interesting to create valid tests for those…

Software Project Python 2to3 continued... 47% coverage, 70% on the core editor...

Whoo… a lot of bug fixing, a lot of test cases, and getting all the test cases to work in both non-wrapping and line-wrapped mode… The cool part is that now when I’ve made some minor improvements, the tests are showing me areas that are impacted by the changes and catching side effects. I should have done this a long time ago. I’m done with the core editor module and I’m going to do the module that manages windows and switches between files and such at a high level next. I’m hoping to get the overall coverage over 60% with this one… we’ll see…

Here’s the latest video capture with the coverage report at the end:

This gets me up to about 47% coverage but 70% coverage of the core editor.

Software Project Python 2to3 continued... Wow, such bugs...

The process of converting the editor ( https://github.com/jpfxgood/ped ) to Python3 and writing tests is going well. I am fixing a LOT of minor bugs and edge conditions that don’t really come up a lot in my normal use of the editor but are very evident doing detailed test cases. I came up with some cool ways to test features that are only available interactively by using the keyboard macro feature of the editor to feed the required interaction to the editor main processing loop. It reminds me of testing 1-2-3 for DOS where virtually all of the testing was done with keyboard macros and screen comparisons. It also tests the macro feature, and it already helped me fix a hidden bug in the macro feature where it was accumulating null characters when no key was being pressed. If one left macro record on it would have eventually sucked up all of the heap memory, though I guess it would have taken a while on modern machines. I may have to refactor the main test function since it is getting pretty long, I’ll see where it’s at when I finish.

Oh and running pytest inside VSCode is wonderful for debugging both the code under test and the test automation at the same time.

Here’s the simple launch config that I use:

{
    // Use IntelliSense to learn about possible attributes.    
    // Hover to view descriptions of existing attributes.    
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "externalTerminal"
        },
        {
            "name": "Python: Pytest",
            "type": "python",
            "request": "launch",
            "module": "pytest",
            "console": "externalTerminal"
        }
    ]
}

If you want to watch the tests run I have this handy screen recording:

Software Project Python 2 to 3 conversion for my scripts...

I’ve got a bunch of python scripts that I have written over the last eleven years or so. I have published some of them on github here: https://github.com/jpfxgood Probably the most sophisticated is the editor ped (python editor) that I wrote originally back in 2009 and have been working on ever since. Over those years I was pretty busy as a professional programmer and managing software engineers in ever larger groups. I snuck in the work on these scripts when I had the time to focus. A couple of things just didn’t get done as part of that, I never wrote proper tests, and I ignored the looming end of life for Python 2 and the migration to Python 3.

Well, now I have the time so I’ve started on both projects. First I brought the github projects up to date, truth be told I use svn locally because it is what I was used to. After this project github will be the canonical version of these scripts. Then I used the 2to3 script to do the rough conversion to Python 3 into a new temporary subdirectory “2to3out”. The script does a decent job, but it was still a surprise that division suddenly works differently… Also some of the code around reading and writing “utf-8” encoded files need to be tweaked. The differences between byte streams and character streams came up in several places. A few miscellaneous changes to the signatures of imported modules and functions had to be corrected. In general the changes beyond what the script converted were small.

The testing was the real problem. As I went through and manually tested each feature of each script I found a number of bugs, most of those bugs were unrelated to the 2to3 conversion, they were simply things I’d never really tested before. So as I got to the point that I was fixing more and more of these I decided that now was the time to bite the bullet and do penance for my past sins and write a regression test suite for these projects.

I liked pytest in the past, it is simple, doesn’t require any intrusive changes, is flexible, and I’ve used it before. So, after a few minutes reviving the brain cells that used to know about pytest I started on a tests folder and a set of test suites for each module in the scripts. I started with the editor first since it is the most complex. I’ll probably do the backup script after that since the fixtures to test it are going to be interesting to implement.

I’ve got a few cases written already and I do thank my past self for refactoring the editor multiple times over the years to make the internals pretty modular and independent. Apparently they screwed with the regular expression module as well as I’m getting Deprecation warnings for many of my uses of it, sigh, I’ll have to go through and clear those up too.

Here’s the test output so far:

View fullsize

It is a meditative process writing tests for eleven year old code, I get to appreciate the good parts of the code and laugh at the crappy parts again.

More updates as it continues…

Software Project Timelapse / Slides / Music

It has been overcast here so mostly taking pictures of the clouds illuminated by the light pollution… In any case I switched to a 28mm prime wide angle lens that is F 2.8 and that has improved a number of things including making the focus easier and getting a wider aperture. I tweaked the music script and also created additional sections in the music so the sound track is better. The ISO was 1600 and the exposure was 10 seconds, I think I can double the ISO so I’m going to just adjust that for the next experiment.

In any case here is the latest WiP:

Software Project Timelapse / Slides / Music

Well here is the first effort that produced images:

It is a bit overexposed due there being a lot of ambient light, this was 28 mm, F 2.8, 20 second exposure, ISO 3200. The focus isn’t good either so I’m going to work on that too. I’m thinking for the next attempt I will try cutting the exposure time and the ISO in half. And I’ll spend some more time in the setup phase on getting the focus right, just have to get used to working with the camera in the dark.

Software Project Timelapse / Slides / Music ...

I’ve been experimenting with controlling my Canon 6D using Linux and using it to capture sequences of photographs in a Timelapse. A while ago I wrote a script to stitch images together into a video slideshow and also generate an improvised soundtrack to play along with them. You can see a version of this project here: ( https://github.com/jpfxgood/slides-sound ). I’m using that script to make the images into a video with a soundtrack.

The camera control software comes from the gphoto2 package ( thank you Linux) and some helpful instructions here: https://www.moreno.marzolla.name/software/linux-time-lapse/

Here’s a sample of the output so far:

I’m working on improvements to the generated music and I’m going to try some night sky time lapse video… more to come… ( also I need to update my github project… )