Software Project Python 2to3 continued... 66% coverage, >75% in the core editor...

This has been a very healthy process for the editor code, as I’ve been writing tests I’ve also rewritten code to make it much better than when I originally hacked it up. That is the thing when you’re in a hurry, often you’ll stay on a bad implementation track because you’re “close” to getting it to work. It is nice to go back and have the time and focus to reconsider a bunch of things. The tests themselves are interesting, for example I had to figure out how to resize the terminal window to test the code that resizes all the editor panes in response. That code always annoyed me and had a lot of bugs, now it is rewritten and it is very stable.

I’ve been using the python coverage tool to measure the coverage and it has also helped point out a few modules that aren’t used anymore so I’ve been improving coverage through deletion as well. I’m going to try and get the overall coverage over 70% and I need to make another pass over the minimal redraw logic as well… It still has some odd behaviors…

Then on to the backup and sync scripts that I wrote, it’s going to be interesting to create valid tests for those…

Software Project Python 2to3 continued... 47% coverage, 70% on the core editor...

Whoo… a lot of bug fixing, a lot of test cases, and getting all the test cases to work in both non-wrapping and line-wrapped mode… The cool part is that now when I’ve made some minor improvements, the tests are showing me areas that are impacted by the changes and catching side effects. I should have done this a long time ago. I’m done with the core editor module and I’m going to do the module that manages windows and switches between files and such at a high level next. I’m hoping to get the overall coverage over 60% with this one… we’ll see…

Here’s the latest video capture with the coverage report at the end:

This gets me up to about 47% coverage but 70% coverage of the core editor.

Software Project Python 2to3 continued... Wow, such bugs...

The process of converting the editor ( https://github.com/jpfxgood/ped ) to Python3 and writing tests is going well. I am fixing a LOT of minor bugs and edge conditions that don’t really come up a lot in my normal use of the editor but are very evident doing detailed test cases. I came up with some cool ways to test features that are only available interactively by using the keyboard macro feature of the editor to feed the required interaction to the editor main processing loop. It reminds me of testing 1-2-3 for DOS where virtually all of the testing was done with keyboard macros and screen comparisons. It also tests the macro feature, and it already helped me fix a hidden bug in the macro feature where it was accumulating null characters when no key was being pressed. If one left macro record on it would have eventually sucked up all of the heap memory, though I guess it would have taken a while on modern machines. I may have to refactor the main test function since it is getting pretty long, I’ll see where it’s at when I finish.

Oh and running pytest inside VSCode is wonderful for debugging both the code under test and the test automation at the same time.

Here’s the simple launch config that I use:

{
    // Use IntelliSense to learn about possible attributes.    
    // Hover to view descriptions of existing attributes.    
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "externalTerminal"
        },
        {
            "name": "Python: Pytest",
            "type": "python",
            "request": "launch",
            "module": "pytest",
            "console": "externalTerminal"
        }
    ]
}

If you want to watch the tests run I have this handy screen recording:

Software Project Python 2 to 3 continued... so many bugs...

Well, it has been good writing tests for the converted editor code ( wip is here: https://github.com/jpfxgood/ped/tree/master/2to3out ) as it has also shown up a bunch of existing bugs and nits and stuff that was just appearing to work but wasn’t… Plus I got to figure out a nice way to test both the internal editor logic and the rendering at the same time. I’ve been using Microsoft’s VSCode to debug things because it can do it in a detached terminal, very handy for a curses based application like the editor.

I’ve also made some decisions about removing some features that I don’t use anymore ( and I don’t think anyone else would use either, an AIM IM integration, nntp client, svn browser… see… you don’t want those ) and I don’t really want to test them or get them working. So those came out of the code as well, the editor will be much lighter.

pytest is pretty robust I must say, it is handling test code with nested functions without batting an eye, it even shows the nesting in the test failure output… very nice. Anyway onward to hopefully finish the core editor classes in the next couple of days.

Software Project Python 2 to 3 conversion for my scripts...

I’ve got a bunch of python scripts that I have written over the last eleven years or so. I have published some of them on github here: https://github.com/jpfxgood Probably the most sophisticated is the editor ped (python editor) that I wrote originally back in 2009 and have been working on ever since. Over those years I was pretty busy as a professional programmer and managing software engineers in ever larger groups. I snuck in the work on these scripts when I had the time to focus. A couple of things just didn’t get done as part of that, I never wrote proper tests, and I ignored the looming end of life for Python 2 and the migration to Python 3.

Well, now I have the time so I’ve started on both projects. First I brought the github projects up to date, truth be told I use svn locally because it is what I was used to. After this project github will be the canonical version of these scripts. Then I used the 2to3 script to do the rough conversion to Python 3 into a new temporary subdirectory “2to3out”. The script does a decent job, but it was still a surprise that division suddenly works differently… Also some of the code around reading and writing “utf-8” encoded files need to be tweaked. The differences between byte streams and character streams came up in several places. A few miscellaneous changes to the signatures of imported modules and functions had to be corrected. In general the changes beyond what the script converted were small.

The testing was the real problem. As I went through and manually tested each feature of each script I found a number of bugs, most of those bugs were unrelated to the 2to3 conversion, they were simply things I’d never really tested before. So as I got to the point that I was fixing more and more of these I decided that now was the time to bite the bullet and do penance for my past sins and write a regression test suite for these projects.

I liked pytest in the past, it is simple, doesn’t require any intrusive changes, is flexible, and I’ve used it before. So, after a few minutes reviving the brain cells that used to know about pytest I started on a tests folder and a set of test suites for each module in the scripts. I started with the editor first since it is the most complex. I’ll probably do the backup script after that since the fixtures to test it are going to be interesting to implement.

I’ve got a few cases written already and I do thank my past self for refactoring the editor multiple times over the years to make the internals pretty modular and independent. Apparently they screwed with the regular expression module as well as I’m getting Deprecation warnings for many of my uses of it, sigh, I’ll have to go through and clear those up too.

Here’s the test output so far:

View fullsize

It is a meditative process writing tests for eleven year old code, I get to appreciate the good parts of the code and laugh at the crappy parts again.

More updates as it continues…