Management: Building teams...

If you’re starting with this post you may want to read this earlier post for context: Management: The job of a manager

Introduction

Over the years I’ve joined software teams as their manager that other folks have built and I’ve built some basically from scratch and then evolved, grown and shrunk those teams over years and releases. A lot of what I have to say about the experience has been said by others, but I hope to add some of my own observations that might be useful.

Inheriting or founding a team

The manager before me had been an intense micro-manager, they had required all the engineers to include them in every meeting, every discussion with any stakeholder, they cracked down on engineers and stakeholders alike when this rule was violated. They made every decision, parceled out their team’s work in dribs-and-drabs, they deliberately constrained creative and productive engineers, they rewarded loyalty above competence and ability. In other words they had set the bar low for me and I knew I was going to be very successful...

Most of the time, I’ve joined teams that other folks have hired and organized. Early in my career I took a “they must have chosen these folks for a reason so I should work with what I have”  approach. It turns out however that most teams have been formed haphazardly by people who were short-sighted and had no interest in the concerns of management or long-term team health.  After torturing myself a couple of times that way, I shifted to a model of “I need to evaluate this team and see how completely fucked they are and decide on the one or two things that I have to un-fuck immediately” approach. There is a corollary embedded in this statement: All teams are a little bit fucked and you’re never going to completely address all the issues. You do need to aim for a good operating temperature for the team that trades off performance, morale, stress, interpersonal tension, happiness, and satisfaction. It is often an impulse to try and fix everything all a once; you should try to resist the urge. The less common case is starting a team completely from scratch, all of the advice below applies, but the first concern is to recruit a good lieutenant. If possible choose someone you have worked with before, someone who compliments you in terms of strengths and weaknesses. This person will be a key partner in choosing all the rest of your team members and in establishing the team identity. You will make mistakes, hiring is hard, people are quirky, combinations of people sometimes don’t work out… so even your hand-built team will be a little fucked up… it’s a process.

Gaining the team’s trust 

The very first thing that a new manager with a new team needs to do is gain the trust of the team members. The first thing that I try to do is listen to the team, validate the problems they identify, and try to address one of the most important ones. This can come before any systematic or process changes. It can show that you as a manager are willing to work for the team, and that you are really listening because you are acting on their behalf. It doesn’t need to be the biggest thing, it can even be something that just requires some focused effort and attention. It is important that you do the work yourself. The second thing is to be transparent with the team, share as much information about what you’re discovering about the team back to them, along with your ideas about what’s important and what needs to be done. Then, you guessed it, listen… weigh the feedback of the team, adjust the priorities, adopt their suggestions if they’re better than yours. Finally, have integrity, for me this means being openly responsible for your decisions and actions. If you say you’re going to do something, do it. If you make a mistake, take responsibility for it. If you need to change a plan, say so and explain the reasoning.

The un-fucking process

As the new manager from the United States, from a just acquired company, from outside Europe, I didn’t have a lot of credibility with the team to start. I spent two weeks in-country with the team and talked to each team member individually and observed their interactions over the course of a release of the web service we were building. I didn’t change anything, I didn’t offer any opinions, I just watched them… Several things became obvious to me: their technical leader was incompetent, they had no experience building large scale web services, they weren’t competent to make the changes to the open source software we were using that they had made, they had no repeatable process to create a release and verify it’s quality, they had no logging or metrics coming from the production deployment and so ( it turned out ) they were missing the fact that 60% of requests from clients were failing, the architecture of their web service was a complete non-starter. I built all my future team and project plans on correcting these issues…

What should you choose to un-fuck first? Different teams will have different problems but first and foremost if the team is under-performing on their goals for the company, you should address that. Watching the team try and do a release is often instructive for diagnosing this sort of problem. Do they make a plan? Does everyone on the team have a role in the plan? Do they follow the plan? Is the work distributed in a reasonable way by role and by level? Are there people who hinder the planning or execution with discursive discussions, rat-holing, or introducing tasks that aren’t required? Does the amount of work in the plan make sense for the capacity of the team? Do they have any idea what their capacity as a team is? Do they just turn to you and expect you to plan everything?

What you want in a team is a group of people who take responsibility for the product or sub-system that they are tasked with owning and delivering. Based on your observations, find the real competent leaders on the team (or hire one) and empower them. If there are people who are corrosive and are breaking the team, get rid of them, even if they are “the guy that knows everything.” If the team has no process and no plan, help them to figure out what they want their process to be and to follow it. Do not dictate it, do not volunteer to operate the process. If there are people who are hanging back and using chaos to do little or no work, give them direct feedback, call them out at task assignment time, and if they still put in more effort ducking work than doing it, replace them. If there are people outside the team that are removing responsibility from the team by telling them what to do, work hard to redirect those people and get rid of them. If there is a yes-person who is signing the team up for huge amounts of work that they can’t possibly complete, stop them and reduce the expectations on the team and reset expectations about team output. Nothing will cause a team to check out faster than being repeatedly given impossible and, in the end, meaningless tasks to do and repeatedly failing. Fix these things before you even consider growing the team.

Setting the tone

I watched the two supposedly most senior technical people on the team regularly engage in yelling matches as the team sat by in stunned silence. It was probably going to get to a physical confrontation in the near future. The prior manager had not only ignored this behavior, they had often joined in… It turned out that neither of these folks were the right technical leader for the team, one of them I essentially fired and the other I sent to a team that had a complete void of leadership and technical experience. This created the space for a calm, productive, talented engineer on the team who had been doing all the real work behind the scenes to step forward and set a completely different tone.

Once you’ve got the team stabilized, you can look at team dynamics and behavior. A healthy team supports all the members of the team; nobody needs to be “assigned” to help anyone else, they just step up and do it. Even tense discussions should be civil and focused on arriving at a good solution based on facts and real requirements. The most common pathology on teams with regard to dynamics is hiring too many “big dogs,” people with lots of technical prowess and experience, and packing up the team with them. The problem with this in a group of humans is that inevitably, because there is nowhere to progress to, they tend to turn to snarking and sniping at each other and arguing over minor issues way past their actual value. Teams ideally have one or two “big dogs,” who are chosen carefully based on their ability to provide high level control and structure to others, and who are good communicators, good teachers, and good engineers. Then a mid-level of good, less experienced folks take up the majority of the team and a few talented entry-level folks. The actual proportions depend on the sort of work the team is doing.

Teams delivering software products or long-lived web services can skew towards the middle tier; inevitably, the most experienced folks will move on and ideally you will replace them from the top performers and leaders of the middle tier. Teams building custom solutions can sustain a lot more entry level folks and probably should; the work is repetitive and time-boxed. The middle- tier and top-level folks should focus on making the underlying platform efficient and resilient in the face of front-end churn.

When you observe bad interpersonal behavior (or someone comes to you about it or you hear about it in a 360 review), take action to bring the parties together in a discussion. Encourage folks to speak up and discuss interpersonal issues that they have directly with the people that are causing them. Provide mediation, where appropriate, to teach folks how to have a constructive discussion about an uncomfortable situation. In some cases, where intervention, discussion, and even direct feedback don’t solve an issue, removing one or both of the folks creating the behavior problem is appropriate. Of course this above and beyond issues like sexual harassment, which has a very clear process associated with it and which you should follow immediately and in all cases. Of course the opposite is also true: Reward and praise good behavior and encourage it whenever you have an opportunity.

Say out loud and in writing how you expect team members to behave with each other. Often, managers are afraid to talk about behavior, feelings, interpersonal matters: get past that. Everyone comes to your team with a different background and a different life experience, so they all need to be told what the expectations are. Back up these guidelines with appropriate actions, up to and including terminating people. One person can damage a team with their behavior and make everyone else on the team lose performance and fail. Nobody is worth looking the other way for, regardless of their skills or inventiveness. In order for any product to succeed, it needs a team that is going to sustain it for the long term.

As teams change, they often go through periods of dysfunction and chaos until a new team dynamic is established. It may take changes in structure or leadership or just some time for it to settle. This is why you should always be evaluating your teams as you did when you first formed or inherited them, looking for the same things, and addressing the same issues. Teams are always a work in progress. Another classic trap is the belief that your new 20 person team is going to look and sound like your first 5 person team of scrappy entrepreneurs. Ain’t.  Gonna. Happen. And it shouldn’t. What worked for 5 people with no legacy, customers, or competitors doesn’t work ten years later with all of those things and more. Examine whether they are now effective for the company, operating in a healthy sustainable way. It doesn’t matter if it looks different from the five patron gods of the company and their mythological deeds.

Getting bigger

Several of the most senior members of the team were dilettantes, they only worked on things that they wanted to, they didn’t participate in the team’s regular processes, they checked in code that wasn’t reviewed or understood by anyone else on the team. I worked to dis-empower them, it was a delicate process, like a lot of folks that have been in a position for a long time there was a mythology about them that they were indispensable among the other executives. One by one I managed them out of the team, often with them believing that they were attaining an even better position. I replaced them with a well balanced group of people who could collaborate and were willing to do all of the work of the team. The overall productivity of the team grew and morale improved… within a year the dilettantes were gone from the company.

When you’re growing a team, think carefully about what that specific team lacks and put that on the list. Of course you want to hire very qualified folks for the level you’re seeking, but you should also look at the personalities on the team and be on the lookout for folks that would help balance those personalities. For example if the team is all mercurial, less experienced team members and they often build overblown solutions very quickly (that later take hundreds of patches to stabilize and make work), adding someone who is more experienced and more methodical at a higher level can help shape their productivity more constructively. If a team has too many big dogs, look for a couple of middle tier engineers with a good record of consistent delivery and then trade a big dog to another team and backfill to balance the team. 

I recommend aiming for gender parity on your team; having a mixture of people who are socialized differently creates a strong cohesive team and helps a great deal with the atmosphere on the team. A very simple mechanism for this is to scan every batch of resumes you get for women, when you see qualified applicants that are women insert them into the process first. This works for other under represented groups too. You also need to avoid and root out systemic discrimination, questions or concerns that are only raised for women or other under represented group applicants. This often takes the form of code language like “too junior,” when the person has done and is doing the exact work you’re after, etc. The final step, you have to hire them. If you make it a goal and reinforce it as a matter of performance for yourself and your managers, you’ll be surprised how well it goes. Your interviewers should include women and minorities from your team. This reassures minority candidates that they won’t be alone on your team and it gives them someone in the interview process to ask questions about the team culture. It’s also a good social test for all candidates to see if they have issues with the women and minority members of your team. Several times we’ve passed over folks who decided it would be good to cut off and talk over our women interviewers and talk down to them when pressed.

Team morale

When I took over the team, one of the founders of the company had been “helping” what was perceived as a team in trouble. The founder’s model of how to help people was to give nebulous directions and then argue with people when they misinterpreted this direction or when they pointed out obvious conflicts in the direction. The team never received any praise, just constant criticism for not being engineers like the founder. Of course, the founder had no experience in the area that the team was working, or the technology… but, this didn’t dent his confidence that he could tell them what was wrong with them. The team was paralized, very few features could get completed because they were mired in senseless circular arguments. I started a process with the team to define who they wanted to be, what were their goals, what was their idea of quality, what were they going to deliver for our customers. Several iterations of this document engaged everyone on the team in forming a vision of themselves and how they wanted to be. At the same time I worked to manage the founder out of the process, noting that in our fast growing company there were many more vital projects that required the founder’s genius. I praised the team every time they lived up to their goals and standards and the morale on the team improved steadily.

The best team morale/cohesion-building is simply working together on meaningful problems. Morale is highest when the team feels in control, successful, and recognized. If your team doesn’t own what they work on--for example, if an overly controlling product owner insists on defining not just the business requirements but the solution, the architecture, and the technology choices--then no amount of building LEGO structures together is going to fix the disengagement that results. Consultants like these little exercises because you can basically take any arbitrary group and make them feel like they worked together and succeeded. That is mostly because: a.) the objective means nothing and failing means nothing, b.) the problem is simple and bounded, and c.) none of the decisions have a lifespan beyond the session. Apart from maybe being a game to play while you’re drinking, these exercises don’t have any impact on the real workplace. I do think that creating contexts for everyone to participate in planning, process decisions, and implementation decisions is good. By this I don’t mean everyone needs to agree or work on everything; instead, I suggest that when you are forming task groups, insert folks from other disciplines and levels into those groups so they work with team members that they wouldn’t normally and on topics that aren’t in their everyday workflow. This expands everyone’s knowledge of the product and what goes into building it, exposes them to evaluating and understanding requirements, and creates social bonds.

Don’t take engineers’ ideas about how to run the team too literally. Many engineers have naive ideas about how humans work and often devolve to trying to program them like a distributed system. Their ideas do have value as signals about what needs to be addressed in the team. All processes that the team adopts need to be humanized. Humans are not machines; the most consistent of them will vary enough and often enough to matter. They will respond to outside forces from their personal life, they get tired, they burn out, they have subjective responses to people and situations that are unique to them… Therefore team processes have to be fault tolerant and have mechanisms for fault detection and correction.

Getting to the right size

I had worked to build one of my sub-teams and fix it’s culture and staff it appropriately for it’s mission… but they were still struggling to deliver. I realized that their main stakeholder still treated them like outsiders, ignoring them until the last minute, not sharing plans and information, not considering the impact of their decisions on them. I decided to advocate that we move the team into the stakeholder’s organization, of course I had to make it seem like their idea… which I did, and the organization change worked wonders causing the team to gain focus, momentum and productivity. One could look at it as me losing talented engineers that I had recruited at great time and expense… On the other hand I now had a whole group of folks that liked me embedded in our stakeholder’s organization…

I would note that I talk about firing people a few times above, but in reality if you consider these issues as you grow your team, there should be little need. In a recent role, I grew a team from <20 people to >100 people in three years, I terminated five people for various reasons relating to performance or behavior over that time. The team had some of the best engagement scores and productivity of any that I have ever managed. Hire good people, build teams thoughtfully, and you will have a great environment filled with strong leaders. 

Reducing the size of the team is sometimes called for. Products don’t work out in the market, companies make bad investments that catch up with them, and organizations need to be restructured to deal with macro-economic events. In some cases, you are allowed to choose the folks to retain in a restructuring. Think about all the topics above as you plan your restructuring. It is tempting to always choose just the most experienced, and most proficient folks, but doing this will result in a team that is out of balance after the restructuring. Reduce the scope of the team’s responsibilities as it shrinks. I have been asked many times to just go on with everything and do it with half the people. I just gave them back a list of the things we’d be failing at, which was also a good list of things that could be cut if they decided to be reasonable. Sometimes you get asked to lead a team that is already quite large and when you do the evaluation of the team you realize that it is too big and very wasteful. It is tempting to just add the new size of team to your resume and let the waste roll on, but teams like that are often riddled with behavior and morale problems, and eventually companies detect the waste and you are as likely to be disposed of in the restructuring. It is better to cancel wasteful projects, send sub-teams to some other part of the company where they can be productive, and, in some cases, even lay off folks to get the team to a healthy size relative to its real productivity and value to the company.

Conclusion

Finally, I would counsel managers to not meddle too much in the team. Even if there is tension on the team, give it time to be worked out by the team. If you do have to intervene, try as much as possible to do it by coaching the team in a process to identify and resolve their issues. The more that you insert yourself between members of the team, the less of a team it becomes. This also means that you should think about your own transition, try not to make every process dependent on you personally, this will limit the impact if you decide to leave the team or are promoted or reassigned.

Management: Getting the most out of your team's brain matter...

( Continued from: Management: The job of a manager… )

Their division volunteered for every project and everyone there “sometimes” worked on any or all of the projects. Their manager collected teams and subordinates unrelated to any business purpose. The division grew and grew and they proudly touted how many projects they had and how many people. The company praised them and raised them up as an example to other divisions. When an economic downturn occurred, coupled with strong competition, management showed up looking for things to cut. When they looked at this “wonderful” division, they found much that the company could live without. They cut it by two-thirds and then, a couple of quarters later, disposed of it entirely. Many people on the team were shocked: They had stayed with this group for years being unproductive and, in many cases, their skillsets became unmarketable. Now they had to find a new job.

My overall model for deciding what to do with my teams is to think “am I using these folks’ skills and brain matter as effectively as possible?” This means having them help create the goals, solve the problems, make the plans, make most of the decisions, come up with the ideas, come up with the processes, and get the credit for the work. The more that I find myself doing those things for a team, the more I feel like I’m wasting their abilities. The more engaged a team is in the whole process, the more productive and efficient they become. This involves trusting your team and being prepared to back them, even when they make mistakes, but it pays off every time. If you focus on the roles I described [ see Management: The job of a manager ], and give the people in those roles good information and guidelines, coach them and give them feedback, and give them the team members they need, they will take over and drive these things better than you ever could on your own.

Always be looking for emerging leaders on your team, and work to support and empower them. They don’t have to have a leadership title; they can just be the person who provides the drive to get stuff done. Reward that behavior and feed that inclination. Then when your team grows, you already know who to turn to when you need to fill leadership roles.

The company didn’t have a process that you could name; they had a couple of people who indoctrinated new team members in “the way.” When you did something that wasn’t “the way,” one or more old timers would correct you. We sat in a circle and coded together and released the software when we thought it was ready. We released very high quality, high performance software on a regular basis...

I have deliberately not made any specific references to formal processes, because I don’t think they matter much. As long as your process provides the basic elements of a software development life cycle, and gives you as a manager the information you need to evaluate your team and give them feedback, it’s a fine process. It does need to be a process that the team buys into, documents, and follows. A process that changes every week or every release is not a process. Ownership of the process is key, for as soon as the team feels the process is imposed, you have lost a large amount of engagement and productivity. The other thing to guard against with a formal process is that it ends up replacing customer value, business value, and the product as the goal that the team is driving towards. I’ve seen scrum teams with the “perfect” burn-down chart that produced nothing of value to the business for multiple sprints. It’s easy to succeed against artificial metrics that mean nothing to the product.

The manager’s flying monkey came into my office and shut the door. “You’re making a big mistake,” they said. “It’s going to be the end of your career if you don’t get on board…” I had spoken up at a review meeting saying that I didn’t think a product that they desperately wanted to ship was ready. I had a track record of delivering excellent results and doing it fast, while bringing the rest of the team along with me. I laughed at them; it was the most ridiculous and transparent bullshit I’d ever seen. I said, “please do your worst. I can always get another job. Now get out of my office.” Three months later, they were giving me a $40,000 bonus to try to entice me to stay on their team.

I talk a little about politics in passing in order to highlight the things that give you as a manager the most political leverage. Delivering value for your company—and being able to point to it and a cadence of delivering value—can protect you against the worst of incompetent executives. It is very hard for greedy CEOs to ignore your team producing things that make money for the company and listen to a cranky executive that you’ve annoyed tell them you aren’t a “Team Player.” Focus on delivering value and motivate your team around that, and the rest will follow. Or it won’t: Some companies are so profoundly broken that they are entirely driven by ego and vanity. Deliver value while you look for another job—that’s what sounds the best in your job interviews. It does you no good to say that you spent all your time arguing with (and resisting) venal executives and that you failed to lead your team to produce results.

“Don’t manage your peers’ teams” is another corollary in this vein. Figure out which of your peers is good at their jobs, then determine whether you can trust and/or defend your team from the rest. Trying to correct your peers’ flaws, organize their people, etc., is a self-defeating process: It ends up with your team not delivering and you’ll get no thanks for interfering with another team. Sometimes, however, it’s the lesser of two evils to have your team building things or solving problems that aren’t their stated goals, because it protects them from depending on failing teams. If you are asked for feedback, give it: be honest, be factual, be constructive, but don’t volunteer to solve their problems.

“Don’t be greedy or possessive” is another guideline. If you have people on your team that can be utilized better somewhere else, offer them up. Don’t grow your team for the sake of growing it; make sure that the folks you hire will deliver for the company. If something extraordinary comes up in the business cycle and you’re called on to disrupt things to fix it, don’t complain: just do it and get back to normal cadence as soon as possible. These aren’t “your” people; their contract is with the company, and so is yours. If you remember this and act with integrity, you will gain influence and trust.

When I talk about having a contract with the company I don’t mean to suggest blind allegiance, but a recognition that you’re taking money from folks and they have a right to expect you to do what is reasonable for your role. Things like lying, committing crimes, defrauding the company, covering up mistakes, supporting the abuse of employees, enabling abusers, participating in a discriminatory process, etc. are NOT part of that contract. Depending on the company, these things may come up to a greater or lesser extent, and should be prepared to do the right thing. You cannot recross the bridge of doing something unethical regardless of the pressure from above: Once you do it, everyone knows it, and you’ve lost trust and respect. Again, you’re better off looking for another job than ever compromising on these things.

Also, companies are sociopaths. That doesn’t mean that they’re evil, they just don’t care about anyone’s emotional state or livelihood but their own. Part of your job as a manager is to discourage folks from engaging in magical thinking about the company: “We’re all a big happy family”, “we’ll take care of you”, “we’re all in this together,” for example. Employees are worth what they generate (or what they are perceived to generate) in revenue for the company, so I always encourage folks to consider their productivity and where it is being directed. When a project seems like a fantasy boondoggle that will never make money, nine out ten times it is, and the people who go to work on it end up looking for a new job. They might learn great things on that project and get great experience, but they should be cold-blooded about the likely outcome. Every company goes through ups and downs, sometimes very dramatically. Having a loving and caring culture while on the upswing is easy, but the downswing is usually where the reality of being a sociopathic business entity with detached investors becomes evident. Again, the best thing you can do as a manager is to encourage your people to be flexible, productive, and skillful; this will make them the most attractive employees to retain in any restructuring.

Finally, you’ll note that I mention incompetence. It’s part of the reality of large groups of humans doing anything together. You should always approach anyone you work with with the initial belief that they are competent at their job, and continue like that until they prove that false. Incompetence gets into organizations because people often make hiring and performance decisions based on things other than fitness for the role or actual job performance. Basing these decisions on personal loyalty, nepotism, friendship, giving titles to justify pay increases, and other motivations promotes incompetence. It often gets into management because folks become managers by default; I can’t tell you the number of manager candidates I’ve interviewed who said “nobody else wanted to be manager so they picked me.” In good organizations, it eventually gets exposed and corrected, but that process can take a long time. You should be aware of incompetence and where it exists in your organization, and keep your team and its projects from being damaged by it. Also, if you’re not competent, do better—or remove yourself and do something you are good at. Life is too short. Recognizing your own strengths and weaknesses, and being honest with yourself about them, allows you to build a team that compensates for your shortcomings, and lets you focus on your strengths.

Management: The job of a manager...

I recently retired after 38 years in the software industry. I have been a software engineer, architect, chief architect, team lead, development manager, director of engineering, and VP of engineering. I’ve worked on and led teams from <10 engineers to >100 engineers at companies like Lotus Development Corporation, IBM, Intuit, Nokia, Elastic, and a bunch you’ve never heard of. I’ve built software for everything from 8-bit microcomputers to Cray Supercomputers and shipped it in boxes, downloads, web services, cell phones and Linux appliances to customer bases ranging from hundreds of specialists to millions of consumers. I’m sharing these retrospectives just to say “here’s my perspective…” I think there are lots of ways to succeed in the complex business of software engineering, these just describe my approach..

I have been asked more than once: “What is the job of a manager?” All of my management experience has been managing software engineering teams. So, my answer is in terms of that field, but I think a lot of it is the same across fields. The HR department might have a whole bulleted list of responsibilities for managers, but I don’t often refer to those. My answer is usually a combination of saying what a manager does and what they shouldn’t do. Often the things you don’t do are very important.

Their manager had created an enormous spreadsheet, listing hundreds of product features, ranked in some arbitrary way. “Here you go, these are  your priorities” they said. There was nothing to tie the features or the ranking to any business goal that could be measured. Weeks later the manager was frustrated saying “Why did they choose to do THOSE things? Those aren’t important.” The team said “See, we ticked off these things from your list! They were easy to get done.”

I think the most vital role that a manager has is to communicate the team’s goals and the relative priority of those goals to the team. This is hard to do well because it involves understanding the reality of the business and organization that you’re working in. It is not as simple as just listening to your boss and passing down whatever they say. You should understand your team’s stakeholders needs and how high priority they are, how much business impact they have, and where your team is in terms of satisfying their requirements. The team itself is a stakeholder, often generating goals internally that need to be aligned with the wider business.

A manager needs to at some regular interval update the team’s priorities and goals and reflect back to the team where they are in terms of accomplishing their goals. It is critical to give ownership of the “how” part of accomplishing the goals to the teams almost entirely. It’s fine to apply “taste & discretion” or “guidelines” to their work, but the manager should not delve down into implementation details. The manager should ask their teams for a plan for how they will address the goals, and that plan should include what is going to happen, who is doing what, and what order things will happen; sometimes they also include time estimates. A manager should provide feedback and coaching on the plan, seek to reduce scope and risk, and question unnecessary work. When the work is in progress the manager should look at how the work is tracking to the plan and suggest re-planning if new information supports that.

All of this planning and feedback should happen in the context of the set of goals and priorities and the goals should support the team’s decisions. If you find yourself or a team going away from those goals, then it’s time to re-evaluate the goals.

A new manager comes to take over an existing team, they’re informed by their boss: “This team is all messed up, you’ve got a lot of performance problems. Here is a list of the people you should get rid of…” The new manager asks “Have you given them any feedback about their performance?” Their boss responds “They know what’s expected of them…” In other words, “no.” The new manager reviews the performance of the people in question and gives them appropriate feedback. The majority of the team improve their performance immediately. Some of them are managed out of the team. The rest of the team gets regular feedback and they steadily improve. 

This gets into the second highest priority of a manager: giving feedback. Once you’ve been clear with your teams about goals and priorities, and they have given you plans and status, which you’ve reviewed, you’ll probably have great content for feedback. First rule of feedback is: “Don’t wait!” As soon as there is an opportunity to learn something together, it is time to talk about it. The punctuated reporting schedule of HR systems is not a good structure for managing successful projects. 

Feedback should be 75% praise, 25% critical feedback. If you can’t come up with 75% praise, you probably have a real performance or morale problem. The 25% critical feedback that you give should be for the highest priority things that are impacting the effectiveness of a team or individual. These should include one to three points, with accompanying suggestions for how to take action as a prompt for the person or team to suggest their own ways to improve. The critical feedback should be something that is measurable in a goal or deliverable.

Solicit feedback regularly from the team or individual’s stakeholders and summarize the trends in that feedback, both positive and critical. Often the best language and suggestions for describing improvements come directly from stakeholders (including peers). Be careful not to overweight feedback based on the age/rank/level of the person giving feedback. Similar feedback from multiple stakeholders is much more useful than cranky feedback from an executive. Of course you should monitor that crankiness and perhaps discuss it with the executive, to try to keep that executive from developing a grudge against your team or employee.

The CTO would rigorously review every project schedule with dedicated project managers, schedules, and task lists. The same CTO would drop in on the most productive engineers and ask them to build things for them, speculative projects that didn’t appear on any plan. Projects would then miss their dates, deep concern would be expressed, and pressure applied to correct the issue… The speculative project would be sidelined and never shipped. The cycle would repeat.

The third thing is to drive a cadence in the work of your team and protect them from chaos agents. In some companies that are very early stage or prototypical startup efforts, chaos is the normal environment, but there are processes to focus and control that chaos and as a manager you should work to do that. The key thing is to allow your team to complete projects and have the satisfaction of delivering things to the market. This isn’t as simple as saying “no” to everyone; it involves working with your teams to determine a structure for how to acknowledge, prioritize, and sequence the new work that inevitably comes in. The other thing to fold smoothly into the workflow is technical debt or maintenance. This needs to be a high priority and carefully chosen, so that it doesn’t end up completely overwhelming the flow of work at random, unpredictable intervals.

Supporting your team in resisting distractions when they come in (from executives with a great idea, internal teams hacking up competition to their products, etc.) is a key part of this process. There may need to be some reserved capacity for ego maintenance of executives, but like a lot of political capital, it needs to be carefully managed because the backlash will be even worse if the team fails in its core mission for the company while delivering an executive’s pet project. Usually engaging executives in the boring process of figuring out all the trade-offs required to make room for their project can make them wander off to bother a different team.

Everyone who applied and passed the phone screen by the recruiter was sent to do an online coding test. Anyone who got past the test was interviewed by the team. People were often rejected after seven interviews, seven hours, by at least seven different people. The team’s hiring rate was very low and the people they spoke to were often not senior enough for the role and were being rejected at the final executive interview. None of the interviewers were clear on what they were interviewing for and what criteria they should use; they often passed people along figuring the next person would reject them. Sadly, the automated coding test was causing some of their best candidates to be offended and to simply opt out and take a different job at another company.

The final key is to make sure that you have great people on your team and that they are the right people for what you’re doing. This involves hiring as well as performance management and internal staffing. You should be clear on the profile of the folks the team needs, both from a “skills and experience perspective,” but also personality type and social fit. The hiring process should be as short as possible, but comprehensive in having at least a couple of people check out each attribute you’re looking for. I personally like combining an interview step with multiple goals; for example, an interactive coding session with two engineers and having the candidate work through the problem with them and interact with them in a similar way to the way the team would work. This can answer questions beyond “can you code?” and get way into team fit.

Like any process, you and your team should examine your hiring process at regular intervals and ask if it is delivering what you want. For example, it is expensive to have your team interview and then not hire people. So, the more that candidates can be screened at the resume screen or at the initial phone contact, the better. Likewise, if you have an executive approval phase in the interview process, you should have addressed all of that executive’s concerns before a candidate gets there, because that is the most expensive time to have someone exit the hiring process.

“Look, “ I said “you are very good whenever we have a support escalation, I get messages from customers praising you and marveling at how passionate you are about solving their problem. On the other hand, you struggle to keep up with even our most junior engineers, people have to step in and correct or redo your work. You are visibly unhappy whenever we talk about your job…” They had already started sobbing. I continued, “Why don’t we talk to the support organization and see if they have something…” They look up in amazement. “You’re not firing me? You want to help me?...”

Performance management is the other side of the coin, and you should act promptly when someone is under-performing. Effective 360 reviews and close examination of status vs. planned assignments will help identify team members who aren’t meeting their goals. Finally, metrics on code base productivity can be useful to support or refute a performance issue. Understandable feedback (along with clear explanation of the consequences of not improving) is crucial all along. Again, don’t wait for a review: As soon as you know something is going on, give the feedback. There are HR processes for dealing with folks who can’t listen to feedback and respond to it. Don’t be scared of them—work with your HR person and replace people quickly who are not performing well. Even in the best teams, a poorly performing person is a drag on morale and will impact the whole team.

A different problem is having the wrong person for a role. Once you recognize this, start a career path discussion with that person. Sometimes they can find a role in the company better suits them, and you can replace them with someone who is a better fit for your team. You can usually differentiate this situation from other performance issues because the person will sincerely try very hard to respond to feedback, but will swing back and forth, first getting better, then worse. They will also show symptoms of disengagement, lack of attention, lack of communication, etc., so it’s vital to suggest changing roles for folks in this situation.

I talk about some organizational context for these four parts of a manager’s job in the next post: “Management: Getting the most out of your team’s brain matter…”

Digital archaeology...

I was going through a drawer in my file cabinet and came across a bunch of 3.5 inch floppies. These are now approximately 28 years old and approaching their expected use life of around 30 years. I thought “Why don’t I get a USB floppy drive and see if I can back up any documents and such that are still on these disks?” So, I did.

I was going to just attach the USB floppy drive to my Ubuntu machine and do it all from there, but for some reason Ubuntu really didn’t like the floppy drive and it was making bad noises and not working at all… So, I dug out the emergency Acer Windows 10 micro laptop that I keep for just these “I need an actual hardware device running Windows to do this thing…” sorta moments. Windows 10 was very happy with the floppy drive and I started to inspect the disks. Before a single file showed up I was transported back 30 years, the sound of the disk slotting into the drive, the shingggg noise as the metal shield pulled back from the media, the whirring and clacking of the disk spinning and the head seeking all sounds that I haven’t heard on a regular basis in decades.

Many of the disks are backups of Dana’s archaeology work, papers, databases, and her dissertation from grad school and beyond. So, digital archaeology of actual archaeology… I immediately ran into some interesting file format issues. At the time in the 1990’s I was working for Lotus Development Corporation (best known for the popular and best spreadsheet program 1-2-3) and so Dana was using Lotus software on DOS to do her work. The first word processor was Lotus Manuscript ( gone into the depths of time, only a Wikipedia entry to mark it’s passing ) and then Lotus AmiPro followed by Lotus WordPro ( similarly gone ) when she moved to Windows later. There are no import filters to read those file formats because Lotus lost the office suite wars with Microsoft and IBM had no interest in that market post acquisition.

Thank goodness for retro-gamers, they have built great emulators for both DOS and early Windows so that they can run vintage games. Because games tend to max out all kinds of capabilities of these environments, the emulators are more than adequate to run original office software if you can get disk images of it. I happen to have original Lotus SmartSuite 97 CD-ROMs (I worked on 1-2-3 for Windows 97 so I got a box of product when we shipped it, back when software came in boxes). I installed it under PlayOnLinux (a wrapper around the WINE windows API emulator for Linux) and very shortly had the applications running. They are VERY FAST on my 16 core 3.60 GHz desktop, the fastest computer of the time was around 50 MHz, and they were written to be fast on those, so even in an emulator which must have a lot of overhead they scream.

Because Lotus was always a very customer oriented company, we actually shipped a high fidelity file import filter for Lotus Manuscript 2.0 in Lotus Wordpro 97 and it works a treat to read Dana’s oldest documents.

Next problem was a database that Dana used for cataloging all of her artifacts and creating the tables and appendices for her dissertation. It was called Q&A from Symantec and we chose it at the time because it had a really simple form builder that let us model the same recording sheets she was using in the field while excavating. We were on the cutting edge at the time… So again, you’ve never heard of Q&A and that’s because it was crushed by many other more popular databases including one from Microsoft. Fortunately we still had the disks… I just copied them all into a folder on Linux and used Dosbox ( one of many MSDOS emulators ) to run the program… and it just works… it came up and read the original files and could query them and generate tables etc… very cool.

There were some disks that contained files created by the MSDOS backup tool… These I can’t get to restore, I may figure it out eventually, but even running actual MSDOS 6.22 in a virtual machine I can’t seem to get the restore program to read them. I’ll keep digging, there are hints out there on the interwebs that folks have figured this out.

I found a couple of disks that belonged to me and that contained backups of source code for a program I had been writing from 1991-1993 in my own time. It was written originally in Borland Turbo C and some assembly language. So strange to look at code 28 years later, I immediately had all kinds of memories come back of where I was and what I was doing then. The program was building a .COM executable which modern Windows won’t even load. These programs were essentially a single 64k code segment and a 64k data segment, no fixups, no dynamic linking, just a code image that got plopped into memory and executed. Most software nowadays doesn’t even draw breath in under 64k… And the program still runs nicely in Dosbox and does it’s thing like it was yesterday.

Now these files are getting safely backed up for the next 30 years along with the software to read them. I think there are some more floppies lurking in the basement, I may dig them out and do some more digging…

Upgrading, music projects etc...

The last week was taken up with upgrading Ubuntu on all of our house/work machines. Going from 18.04 LTS to 20.04 LTS ( and in one case a laptop that was hanging around on 16.04 LTS… ). In general there weren’t any major problems, though there was a bug in Libreoffice Writer that was very bad, so bad I’m surprised they released it, essentially embedded tab characters didn’t work any more, they didn’t indent, you couldn’t enter them, they had no effect, it appeared like they were stripped on file open, though they weren’t. Libreoffice patched this bug, but their patch releases don’t end up automatically upstream in Ubuntu so I had to install the patch manually. The other wonderful quirk is that their patch release doesn’t upgrade the packages installed by the distribution, they install next to it… this forces you to make a whole bunch of changes to launch the correct version. I was not impressed.

I’ve also been working on a music project, I am trying to create some videos with words, imagery and music that I’ve captured, created. This week I was working on themes and writing some of the words down that I’d like to also communicate with the music and the images. That process is helping me with editing and “composing". I put composing in quotes because I’m a very untrained composer with a limited grasp of music theory. What I have is some training in improvisation and a loose understanding of chord progressions, scales and structure. I have been building on that to come up with the structure of the piece and I’ve been recording different sections of it and I plan to work on editing/adjusting those down to a final product.

I’m using google apps presentation software to automatically display the images annotated with words and cues so that I can play and respond to them in real time. My goal is to get to a “draft” finished product so I can understand it’s limitations and then iterate. I’m also getting in a lot more practice and that is in itself a good thing. I tried a lot of different recording/DAW software and I finally ended up going back to Band in a Box 2011.5 (the last version I bought) because it provides some excellent automatic accompaniment bass, drums etc… in multiple styles, and actually has some really nice recording features and a decent notation editor. It runs on Windows, so I have it running on my laptop in a Windows 10 VM and amazingly it keeps up just fine and is able to see the MIDI and Audio devices mounted in Linux. The accompaniment helps me a lot to follow the structure I’ve set up and to stay roughly in time ( I need all the help I can get ).

The process I come up with is probably going to be quite unconventional, but I guess that will be part of what makes it my expression.

terminal-dashboard v0.0.5

I’ve made a lot of progress on the testing for the dashboard tool, I’m at about 80% coverage across the whole codebase. It was interesting getting the fixtures written for each of the different data sources so that they could be tested. I also added some features for testing and also some refactoring to enable testing. I moved most of the code for loading and processing the configuration file out of the main script and into a separate module. This allows those functions to be tested in isolation but also they now represent a usable API in their own right. I didn’t want to include the config code directly in the dashboard module since it is coupled to all of the different types of data sources and their dependencies. This way the dashboard code could be used with totally alternative graph objects or data table objects without any direct coupling to those specific implementations or their dependencies.

I used a model of snapshot comparison to verify the graphics rendering code, this involved creating static data sets and controlling the screen size carefully. It also requires some honesty on the part of the tester to carefully verify the renderings and not accept bugs. I have at least one bug in my example dashboard that I so-far haven’t been able to reproduce in the test automation… but it is on my list to track down.

My plan is to finish getting through the CLI and config testing and then on to the next round of features and enhancements.

Breaks...

Part of the fun in being retired is that I get to decide when to take a break and it is now free from second thoughts about “What if <name of executive> has a freak out when I’m taking a break?” I realized that I’d been pretty focused for a bunch of weeks in a row on various projects and I needed to step back and get outside and look at some distance and give my brain a rest. So I have been… I’ve been reading some Chester Himes (recommended), getting back to my walks which had been interrupted by having contractors in the house every day, and even just taking a drive and looking at stuff on the rest of Cape Ann. Also, giving in to the urge to nap is also good… the cats are reaching an age where they’ll nap with me instead of just trying to pull my nose off…

I’ve been getting better at resting and recharging over the last three decades, I reached a bad spot in my twenties where after having an episode where I was sent home for yelling at a colleague in a build lab at three in the morning… I realized that I had gotten too wrapped up in work and was just teetering on the edge of burn out… Shortly after that I started taking longer and more restful vacations… one of the best was to a small island next to Tahiti called Moorea… very remote, very beautiful, and our accommodations were a hut over a reef… no clock, no computers, no air-conditioning, no TV… the first few days were hard and then we broke through and gave in to island time and pace and it was an amazing fifteen days… After that we tried to add something like that into our year as a time to just disconnect ( and islands for whatever reason are great for disconnecting ).

I’m not worried about finishing my projects, I honestly don’t care if I do or not… I probably will given my track record… I mostly worry about compulsively turning them into a job because that’s what I’m used to… Ok, back to my break, it’s Friday, never start anything new on a Friday…

Cooking: Fermenting and glass followers...

Since we’ve been getting a farm share from our local CSA at Appleton Farms ( https://thetrustees.org/place/appleton-farms/ ) , I’ve been learning how to ferment things to preserve them ( you can only eat so many beets in one week ). I’ve gotten pretty good at various varieties of pickles and sauerkraut variants based on this very good book Fermented Vegetables: Creative Recipes for Fermenting 64 Vegetables & Herbs in Krauts, Kimchis, Brined Pickles, Chutneys, Relishes & Pastes by Amazon.com Services LLC ( https://www.amazon.com/dp/B00KLNAJ0C/ref=cm_sw_em_r_mt_dp_HS-lFb6ZBKYVZ )

I had initially gotten a ceramic crock for fermenting, but I found that after a while the salt from the brine would leech into the body of the ceramic because it had un-glazed edges and would crack the glaze from inside. This made it so that you couldn’t really sanitize the crock and mold would get into it and would infect your product. I ended up having to discard the crock.

Now I use glass jars, quart wide mouth jars, and this works great and additionally everything is already in a jar and you can just apply lids and refrigerate when the fermentation is done. The only challenge is having a proper seal and keeping the fermenting vegetables submerged in the brine. I had been using plastic bags filled with water which provided both functions, however it wastes plastic bags… So I found two options for glass “followers” (that is what these weights are called) one is a set of solid glass disks made to fit inside wide mouth jars, and the other is using the jam size ball jars which also fit inside filled with water. Both can be adequately cleaned and are reusable. I use a primary follower of parchment paper ( not having fresh grape leaves handy ) then the weight on top, cover everything with a towel to keep the light out.

I have a batch of beet, carrot, onion, cabbage sauerkraut and some dill pickles going now using both methods. I have high hopes, they were both a lot easier to use than water bags…. I’ll report back…

Software Project: terminal-dashboard, progress...

I’ve made a lot of progress on the terminal based dashboard tool that I’ve been working on. It is out there on github now ( https://github.com/jpfxgood/dashboard ) and I’ve made some pre-releases. It is on pypi as well as terminal-dashboard.

The feature set has increased a lot, it handles getting data for graphs from Elasticsearch, ODBC databases, CSV files, JSON files, generic log files, remote systems via SSH. It is integrated with the system keyring so no credentials need to be stored in the config. I’ve improved the rendering and scaling of the graphs a lot, smart shortening labels, shifting stuff slightly to make axis and labels clear, reducing the number of pie slices for clarity, etc… It has a plugin interface for both graphs and data sources, as yet untested… but I’ll get it working…

Next up is writing a test suite for the system and improving the API and user documentation. There are also some minor features like better y-axis scale choices, better format trade offs as the graphs get compressed, performance improvements, check-pointing some of the log readers so that they don’t have to completely reread everything on startup if shut down for a little while, and JSON record format logs…

In the meantime, I’m running endurance tests as well to see if it’s leaking or has any long running bugs…. I still really like the model and architecture, I haven’t had any major refactors and the number of additions and tweaks to the API are rapidly declining… I think the sub-modules for character graphics, graphs, and data tables and data sources are all going to be useful by themselves and I’ve been careful to keep them very decoupled…

I’ll probably do another pre-release next week…

Remodeling...

We just had the worlds longest bathroom remodeling project. It started last November when we decided to do the project and we went to the bathroom and plumbing store and picked out all the products we needed and bought them in a Black Friday sale… Then, it took until January to get the attention of a couple of contractors and get bids… Then we signed a contract with a contractor who turned out to be a dirt bag that lied to us and gave us the run around for about two months… We fired him… We signed up the more expensive runner up in the bidding… Then COVID-19 lock down…

A couple of months with a new bathroom filling up my garage… then finally actual work starts with the additional stress of having folks coming and going from our home during a pandemic… The contractor was great, they took all the precautions, masks, temperature checks, their own washroom on site, limited the number of folks on site, etc… Then some asbestos in a wall that the plumber needed to cut into… Asbestos mitigation and removal… also a whole wall we hand’t planned on removing… More plumbing work to bring the illegal drains and vents that were revealed into compliance essentially re-plumbing the entire bathroom… also not in the plan…

Inspections are interesting now, some are very rapid on-site inspections and others are being done using photographs remotely… after the inspections there was building out the 1929 wall studs so that the new plumbing would fit and losing an inch and a half of floor space… new sub-floor… new drywall and mud… paint the whole room… New shower base and surround, then the new floor….

Finally in about the last week, the installation of all the finished products finally exiting the garage… paint touch ups… [ Oh and during the winter one of our radiators had leaked and damaged the dining room ceiling, so they were good enough to fix the leaking valve and patch the dining room ceiling ] and then…

The first shower in over a month, we had been washing up in a tiny half bath that also contains our laundry equipment… so nice to have a shower again…

It is pretty much done… we’re waiting for some custom glass sliding doors for the shower to get manufactured and installed… any week now… I’ll post some pictures when it is completely done…

Software Project: terminal-dashboard v0.0.1

Back in 2017 when I was working at Elastic and leading the Kibana team ( Kibana if you don’t know is a web based tool for querying, analyzing, and visualizing data that is stored in elasticsearch, which has as one of it’s core use cases rendering dashboards for collections of operational monitoring data in data centers ) and being the contrarian that I am, I thought: “What about the green screen folks who manage lots of systems, only use the command line and a terminal, and who would balk at installing a whole server just to run Kibana? Why can’t we have a dashboard tool that presents graphs of the same data in a terminal window?” Probably an insane idea, but you’re reading the words of someone who wrote graphical games for the TRS-80 Model I so it didn’t seem insane to me… So I started out using curses and block graphic characters to create a drawing package with all the essential graphics primitives. Then I built a data table abstraction to allow flexible access to the data without too much coupling… Then things got really busy at work and my attention span disappeared and I put the project aside.

Three intense years later I’m retired and I’m spending some time converting old projects to Python 3, moving old svn repos to git, and I come across this project again… I have the time now so I decided to push the project forward a bit more. I’ve managed to implement some basic chart types, line, bar, pie and table. I’ve created data sources for reading the syslog and summarizing it and for getting the machine’s process, memory, and disk statistics. I’ve written a driver script that loads a JSON configuration file that sets up the pages, panels and graphs of the dashboard and displays them and allows interaction as well as an unattended cycling kiosk mode. At this point I think this thing could actually be useful. Here’s a short demo of it running:

There’s a few things that I want to add in the near future:

• Ability to monitor other systems via ssh

• A flexible data source for querying elasticsearch so folks could use it with an existing Elastic Stack deployment

• Management of credentials using the system keyring so it can auto connect and self deploy through ssh to multiple systems

The sub-modules I think might be useful for other applications since they are pretty general and not coupled to this particular application.

It’ll be fun to show the next steps…

Cooking...

I’ve been getting back into cooking now that I feel like I have more time and mental bandwidth. I used to experiment a lot more and try new recipes and ingredients, but when my work life got to a certain level of intensity I fell back to more of a rotation of greatest hits and familiar things. Without a work life anymore, I find that it’s easier to take my time and look at a bunch of the cook books I’ve accumulated or search the interwebs for something interesting to cook.

It is also farm share season and the constant influx of a box of vegetables from Appleton Farm helps to drive cooking activities and planning for the week. Making dishes or food products that can be frozen or preserved is a part of this too, we can’t eat all those vegetables in one week, but over the rest of the year we do. Pickles and sauerkraut are also great things to make and enjoy later. I also have a fish share from Cape Ann Fresh Catch and this means that I'll be cooking at least one fish dish every week.

I also like classic dishes done in a classic way, they became classics for a reason, they became cliches due to overuse and losing focus on what was great about them. For example I made a classic Caesar Salad a little while ago from scratch, it was amazing especially with ultra fresh romaine lettuce from the farm. It was nothing like the watery bland stuff that you get in most restaurants nowadays… The umami from the anchovies and egg yolks against the bite of the garlic and the acid from the lemon is just amazing.

Cooking, when you can do it in a relaxed way, is extremely satisfying, the process is engaging, the result happens in a finite time, and it is something you can share and so it is also gratifying beyond just consuming the food. When I was much younger I cooked in restaurants as a short order cook, turning and burning as it were, and I learned how to relate to food, how to prep, how to be confident about the process, and how to multi-task to get things out at the same time. I remember closing the restaurant and feeling completely drained and yet calm after an especially intense shift. It also formed bonds with the rest of the team in the kitchen even if you were very different people, the intensity that you shared drove you together for that time and when you were flowing together and it worked it was a great experience.

Software Project: slides-sound v1.0.0 shipped...

So, the final project that I’m going to convert to Python 3 has shipped. I restructured it to a standard python project, converted it to Python 3, fixed a bunch of bugs and things that just weren’t good… added packaging and did some manual testing. It is here on github (https://github.com/jpfxgood/slides-sound) and from pypi by doing python3 -m pip install slides-sound.

This is really an experiment if anything and I intend to keep iterating on it, so possibly massive changes will come. I’m not doing formal tests for this one because honestly it doesn’t warrant it yet. If it ever stabilizes then I’ll dig in and do that.

There were more changes than I expected because some of the packages I used had changed in unexpected ways, some of the way I was doing things was completely inefficient, and stuff was broken because it was just hacked together ;-). So now it is a lot cleaner and less hacked…

You can check out the readme and the doc for further information…

This is the end of the Python 2 to Python 3 saga for now, there are a few other things that I haven’t published that I may convert in the future, but these are the ones I use most often and they’re published.

Here’s a parting video generated using the slides and music scripts:

Pictures from my several years commuting to Berlin.

Software Project: ped-editor v1.2.0 shipped!

A new feature release of the editor. The feature is that either when you pipe content into the editor like “find ~ | ped” or you use the F10 shell command dialog the buffer will update in the background for as long as the command produces output. You can press Ctrl-F in that buffer and toggle following, i.e. the screen will automatically show you the end of the stream and the latest content. If you want to follow a file on disk you can use “tail -f filename.log” as the command and the resulting buffer can be used to view this information. All of the regular read only buffer commands are available, copying content, searching with regular expressions, etc…

There are new tests since the underlying StreamEditor class is used in the StreamSelect component and so the existing tests caught a lot of side-effects that I had to clean up as I did this. I have to say having sensitive tests has made this whole thing a lot safer, I probably would have pushed my first cut on this in the past and then discovered the problems later. Now, it all works and is covered with regression tests.

I’ve also updated the wiki usage documentation with up to date screen shots and more information about how the editor works.

Software Project: bkp-sync v1.0.0 shipped!

The suite of backup, restore and sync tools that I use to back up and sync all of our home systems is now officially a v1.0.0. The repository is here https://github.com/jpfxgood/bkp and the package is in pypy as bkp-sync so you can just do python3 -m pip install bkp-sync. The scripts will get installed on your ~/.local/bin…

The transition from the original code base to this one was a massive change, I converted to Python 3, restructured the project to be a standard Python project format, wrote pytest automation for all of the modules and the command line scripts.

I just converted my main system over to using the packaged version in “production.”

In the process of doing this I cleaned up the code a lot, the original had lots of globals and module level state which made the modules sorta useless as an API. I fixed all that, encapsulating everything into classes so now the modules form a very powerful API for moving files between file systems, sftp sites, and Amazon s3. In the process the command line tools gained new capabilities that they didn’t have before, “sync” for example was never intended to work with s3 and now it does, so you could set up a sync between a local folder or folders and an s3 bucket and changes would just get automatically pushed and pulled.

The backup api creates versioned sets of changed files which is very cool as a concept to build other things besides backup software. The controls over what gets backed up are very flexible and powerful.

The whole process went faster start to finish than the editor project partly because it was a smaller codebase with less “features” but also because I had already climbed the learning curve of the tools and such.

So, I think I’ll get https://github.com/jpfxgood/slides-sound converted to Python 3 but without the rigorous testing since I really don’t use those tools for anything but experimenting. The https://github.com/jpfxgood/my_radio project is probably dead, it wouldn’t be worth converting it to Python 3 so I’ll just leave that there as an archive.

Software Project: ped v1.1.2, pypi and packaging

I finally built a standard python package for ped and released it to pypi which means it can be installed now by doing: pip install ped-editor ( just make sure that ~/.local/bin is on your path ). It is a sign of my brain recovering from years of being a manager ( and by definition very distracted all the time ) that I figured out the packaging stuff in a few hours. I have to say I looked at it many times over the years and I just didn’t get it so I put it off.

I’ve even started following a software development process, running and adding tests, building documentation, pushing releases. I’ve been the one making other folks do those things, but I haven’t had to do them myself for a while. There was considerable grinding and screeching as those old wheels started to turn again in my head.

Now that I have this project in shape I’m going to shift my attention to the backup software ( https://github.com/jpfxgood/bkp ) and do all these things there. Hopefully it’ll go a bit quicker since it won’t include the learning/relearning that I did on this project.

Also, I need to rehab a bike for my wife, but that is a different thread…

Software Project: ped the editor, v1.0.0

After all the work getting the editor ( https://github.com/jpfxgood/ped ) to Python 3 and getting the test coverage to 78% and fixing a lot of bugs, I also wanted to do some improvements. I worked hard on getting the optimal redraw to work and the terminal cursor was still being a problem so I turned it off and now render my own cursor. I also had to redo a lot of the code that managed multiple views so that they would render efficiently and also keep their cursor positions when you changed frames or current files and updates in other windows are properly reflected in all the windows. I also made it easier to see which frame is current when there are multiple frames and cleaned up a bunch of over refreshing in the dialog code as well.

I made it so that editing a new file doesn’t create the file until you save it and also the editor now warns you if the file you are editing has been modified on disk.

I optimized the code for recoloring the current file and now it runs very fast reducing or eliminating any flashes when it has to catch up.

So, given all the improvements and the testing I decided to mark that moment with a v1.0.0 release. I’ll try to do patch releases relatively frequently as I find and fix other things ( two seconds after the v1.0.0 release I found several things and fixed them…)

It’s been a lot of fun getting this code cleaned up and improving the testing, I’ve spend hours at a time in a kind of flow state just coding and debugging. I even forgot to get lunch a few times, which I never do…

Software Project Python 2to3 continued... 78 percent coverage!

So about 20 days worth of work but the editor ( https://github.com/jpfxgood/ped ) is much better for it. A lot of bugs have been fixed and the general quality of the code has been improved in many places. There are a bunch of cool examples in the pytest code including sort of extending one of their fixtures. Discovered some subtle bugs that have been there since day one which shows that the tests are pretty sensitive. I also matured some functionality liked wrapped mode for example, it is now stable and tested and useful.

I’m going to move on to the backup/sync program ( https://github.com/jpfxgood/bkp ) and creating tests for the Python 3 version of that.

Here’s the final video of test suite running:

This is the final set of tests generating about 78% coverage of the editor code.

Software Project Python 2to3 continued... 66% coverage, >75% in the core editor...

This has been a very healthy process for the editor code, as I’ve been writing tests I’ve also rewritten code to make it much better than when I originally hacked it up. That is the thing when you’re in a hurry, often you’ll stay on a bad implementation track because you’re “close” to getting it to work. It is nice to go back and have the time and focus to reconsider a bunch of things. The tests themselves are interesting, for example I had to figure out how to resize the terminal window to test the code that resizes all the editor panes in response. That code always annoyed me and had a lot of bugs, now it is rewritten and it is very stable.

I’ve been using the python coverage tool to measure the coverage and it has also helped point out a few modules that aren’t used anymore so I’ve been improving coverage through deletion as well. I’m going to try and get the overall coverage over 70% and I need to make another pass over the minimal redraw logic as well… It still has some odd behaviors…

Then on to the backup and sync scripts that I wrote, it’s going to be interesting to create valid tests for those…

Software Project Python 2to3 continued... 47% coverage, 70% on the core editor...

Whoo… a lot of bug fixing, a lot of test cases, and getting all the test cases to work in both non-wrapping and line-wrapped mode… The cool part is that now when I’ve made some minor improvements, the tests are showing me areas that are impacted by the changes and catching side effects. I should have done this a long time ago. I’m done with the core editor module and I’m going to do the module that manages windows and switches between files and such at a high level next. I’m hoping to get the overall coverage over 60% with this one… we’ll see…

Here’s the latest video capture with the coverage report at the end:

This gets me up to about 47% coverage but 70% coverage of the core editor.