RPG and Programming

I have been thinking a bit about how our backgrounds, educational and otherwise, may determine at least to some extent the style we use in our programming and indeed perhaps the language we use.

One of my fellow programmers (about my age) graduated from college with a background in computers. Almost immediately he was able to find a position programming in COBOL, then in ALGOL, which is a block-structured language with a relatively free format. He continued to do well at various positions; ultimately, he came to where I work, with many years of experience in programming, but none in RPG. The style here could be described as a blend of RPGII, RPGIII, and RPG/400; fixed format to the core, loaded with indicators. He learned RPG from the ground up in that environment. Sometime after I came along, he was shown freeform RPG.

I, on the other hand learned RPGII from a book and practical experience. I had no computer background when I started to learn. I was of an inquisitive spirit, though, and I advanced in my use of RPG as the language itself improved. I too ultimately became acquainted with freeform RPG.

Now, who do you think uses freeform RPG whenever he can, especially on new programs? You might have guessed it was my partner- if you did, you’d be right. RPG was actually stifling to him; given the freedom to use free format RPG along with the BIFs that work well with it, he has been liberated. On the other hand, as you may judge from this blog, I just have not seen the light or the point. When I write code for new programs, it is in fixed format RPGIV.

We learned to love the way we learned programming, and so our preference now is to do it the way have always learned it, just better.

Illustrating the point that the way we learn to program is greatly influenced by our programming origins was a discussion I just read on an RPG forum about the significance of certain characters when writing CL (Control Language on the AS/400). Instead of indicating a “not equal” test by the common expression *NE, the code used the combination ^=, where the ^ is really a character I can’t find on my keyboard, but looks like the upper-right-hand corner of a rectangle with the top leg longer than the right. The character, as it turns out, is the “logical not”, a character often used in Boolean algebra.

Now, I don’t know who originated that code. Some experienced programmers, though, just did not know what it meant. Therein may lie a tale.

During the time I learned, many if not most of my fellow programmers were not products of any kind of computer science curriculum. They did not learn BASIC or Fortran first, and usually not COBOL, though this was a bit more common. They learned RPG as their first language. They did not learn “computer science concepts”; many, like me, either never attended college or did so only briefly. They may have started as a computer operator, or maybe at another position in their company. Usually it was a small company; it had the smallest capacity IBM computer available, the System/3, or System/32, or System/34. Something about the programming process intrigued or excited them; they persevered, and they advanced.

As time went on, though, and RPG increased in importance, people with a background in computers and computer science began to program RPG. For a long time, they would prove to be stifled if they expected to program like they originally learned to program in other languages. And where the original programmers may have simply accepted, for example, the RPG cycle, and comprehended it entirely, by the time the System/38 came around, programmers who started learning RPG as RPGIII on the System/38 evidently found it mind-boggling.

I can’t count the number of times I have found a test for LR (last record) in the RPG detail cycle in code (usually written by programmers who came from the 38) using an input or update primary file. Unless explicitly set on with a SETON instruction, the LR indicator will never be on during the detail cycle. A programmer who started with RPG II would automatically know this – it would be a part of his upbringing. A programmer who started with a background in computers later would not necessarily know this. This would cause problems if the programmer intended to base output on this indicator test. Usually it would mean that last record calculations and total output would be missing.

Another difference in outlook comes with file processing. A programmer raised on RPGII (usually in a small shop) doesn’t mind the system reading files (primary and secondary) without him telling it to. But I have noticed in discussion forums that people raised on other programming languages (usually learned in college) seem to get bent out of shape if they don’t explicitly get to do the file reading themselves. They don’t really comprehend that this really is quite an advanced feature, much closer to SQL than Pascal.

It would be interesting to find a survey of programmers that told what level of education they had when they first started programming professionally. I would suspect that as a group, RPG programmers would have a slightly lower level of educational advancement than, say, a C or Java programmer. This would especially be true if they surveyed the most experienced RPG programmers. But then, it is results that matter. RPG programmers do not write compilers – we process information so that humans may comprehend it.

The rather slow acceptance of freeform RPG since its inception has been intriguing to me. For that reason, a recent discussion on  the RPGIV forum has been interesting. Bob Cozzi, a well-known RPG expert, posed a question about how long it took to actually code the implementation of freeform to Hans Boldt, who evidently was on the team at IBM that developed the original implementation.
Hans said that it only took about 1 PM (person month?) to actually code it, but considerably more time was spent in planning. I can believe this. The planning discussions must have been particularly interesting.
One of the biggest things that have slowed acceptance of freeform was the non-implementation of MOVE and MOVEL. This is also the opinion of Cozzi and others in the field. Mr. Boldt touched on this and other issues.
Boldt’s first design included implementation of all opcodes. Others in the group, one in particular, argued vociferously against it. Boldt eventually came around to their way of thinking. As he said, “That design point meant we didn’t have to worry about those goofy multi-part factors.”  (I’m not clear on what that means, exactly. But anyway…..)
Once that was established it was just a matter of deciding which to implement and which not. GOTO was thrown out because, as stated in Nicklaus Wirth’s famous title for Edsger Dijkstra’s article on GOTO, “Go To Statement Considered Harmful”. I could quibble about that, but since it’s been probably 4 or 5 years since I last put a GOTO in a program, there wouldn’t be much point in arguing.
With MOVEL and MOVE however, Mr. Boldt doesn’t understand why people are so attached to those opcodes. He says, “It’s hard to see how these opcodes can be missed given the other options available by free-form opcodes.”
To a certain extent, I agree with his comments. I myself have used MOVE/MOVEL very seldom in the last several years. New opcodes have for the most part been very workable. Using EVAL and many BIFs, I get along very nicely. When EVAL gets a little tedious, I often create data structures to put together data in the desired patterns.
The problem is, I am talking about new code. Old code from the 70’s, 80’s and early 90’s did not have all this neat stuff available. About all you had for data manipulation was MOVE and MOVEL, along with their cousin MOVEA, along with perhaps data structures. From my observations over the years, I would say most programmers did not avail themselves of the help even data structures could provide; they did their work with MOVE(L). We are not talking about an easily disposable opcode like FORCE; we are talking about opcodes that appeared in almost every non-trivial RPG program before RPGIV, and most of the trivial ones. They used the opcodes to change data types (from alphanumeric to numeric and back). Above all, they used MOVE to manipulate strings, piecing together long strings from five or six (or more) smaller ones. One can complain that this is a very kludgy way of doing things, but this is all we had to work with. I would suspect that there are millions, if not tens of millions, of MOVEs in the legacy code sitting on today’s AS/400s (or whatever we’re supposed to be calling them today).
So what do we do in freeform once MOVE is taken away from us? As pointed out in the post I referenced yesterday or the day before, we must choose among 10 or 15 alternatives, depending on what we want to do - which in a sense is a demonstration of the power of the opcode. There is no easy conversion; it is not the same thing as changing  KEY   CHAIN    MASTER  to CHAIN KEY MASTER; . I don’t think even artificial intelligence could do even close to a good job of determining the best conversion; Linoma tried, but some MOVEs are just untranslatable without human eyes and brains to analyze them.
Therein lies the problem. If you rule out MOVE in freeform, a human must determine the best way to translate it to freeform. It simply isn’t financially practical to spend resources changing massive amounts of fixed-format code to freeform by hand, and the utilities available for automatic conversion do not do a clean job. You can use Linoma’s software; you can use Websphere’s conversion facility, which does not handle the MOVE and does not even attempt to do the indentation; there may be other utilities around. In the end, to convert legacy code, you can convert it to a degree automatically, but you will still be left with MOVEs to contend with by hand. You end up with ugly flipflops between free and fixed format that may even be harder to read and deal with than the original fixed format code.
It is very nice to say, that when we modify a program, we will also change it to freeform. But that is a slow way of converting what might be hundreds, or even thousands, of programs on your system. Remember, converting a MOVE to something else is not the same as reformatting a CHAIN; it must be carefully tested.
So, while freeform may ultimately win out, it could have been done a lot more readily if MOVE had been implemented in freeform. The more I think about it, I wonder if someone’s ego got in the way, someone who said, “I am going to teach these plebeians how programming SHOULD be done, instead of using these operations that no other reputable programming language uses.”  (Down, XFOOT, down!!!  Don’t bite him!)
And to think it could have been avoided by allowing the conversion of  MOVE  STRING1   STRING2  to the freeform equivalent:  move   string1    string2.

I didn’t think I’d be considering free-format RPG so soon as a final solution for the current RPG programs on the system at work (some of which date back to the 70’s), but a number of interesting events have taken place.
Our IBM representative made us aware of a software package that accomplishes the conversion of RPGII, RPGIII, and RPG400 programs into true RPGIV. This means (among other things) ditching the left-hand indicators and attempting to really use more modern operation codes. If you are at all familiar with what would be necessary, you can appreciate how difficult this task would be without some kind of automation. The painstaking code changes and testing would be incredible on a code base of any size.
This package is called RPG Toolbox, from Linoma Software.
So we downloaded the trial version, which allows 10 source conversions from another RPG dialect - or even RPGIV that has simply been converted from an older format using IBM’s CVTRPGSRC utility. CVTRPGSRC simply reformats old code into RPGIV syntax; it does not attempt to modernize it. We converted a source member with a lot of left-hand indicators, and we were impressed how it converted into a reasonably neat format without those indicators. What really impressed me was that it would attempt to convert a series of MOVE statements, which was commonly the method used to piece together larger fields, into EVAL statements. This could be a dangerous technique, but the RPG Toolbox utility carefully analyzes the code to make sure it is safe. In the manual, it describes how it makes decisions about code conversion; if one is interested in this product, that person would do well to download and read the manual before making decisions about what the program should do in the course of conversion.
I also had an inspiration- why not use it to convert RPG IV to freeform? In earlier posts, I have declared my reservations about how valuable freeform is. But, if it’s easy, why not live dangerously? So I tried converting one of the RPGIV programs in my file cross-reference system to freeform. The results were impressive. It looked VERY good. The more I looked at it, the more I liked it. So I said to myself, why not convert ALL our code to freeform?
But then my younger cohort brought me back down to earth. He wasn’t so sure that would be a good idea. So we converted the old-format code referred to above into freeform. The results were messy. There was much code that simply could not be converted, often involving the MOVE and MOVEL opcodes. (I have also commented on this previously.) When this happens, the free-format code is enclosed in a /FREE - /END-FREE set of compiler directives and the code drops back into fixed format. Switching back and forth between free and fixed can be maddeningly ugly.
So I have backed off the idea of a total conversion to freeform RPG. But we still feel that converting our code to a clean RPGIV that uses as much as possible of RPGIV’s modern syntax and opcodes would certainly be a good thing; and it appears that it would be relatively safe. We have ordered the software. Where the conversion of a program to freeform would not create too messy a result, we can still convert it.
I was also surprised to learn how quickly the most experienced one on our crew would take to freeform. He first started programming in ALGOL; so freeform is not really that much of a leap for him. He has in the past several weeks written new programs in freeform RPG, of his own free will. My other partner has a Pascal/Java background; so freeform would come “naturally” to him. I am the one who will take some getting used to the idea.

I work on code that, when I came to work for my client, was essentially RPGIII/RPG400 or older. Some of the code had been implemented way back in the 1970’s and was migrated forward as newer machines were installed. And the code showed it. (I have found interesting the large number of 80 and 96 byte files in some of the systems, remnants of old punchcard processing programs.) You haven’t lived until you’ve attempted to update a program that has grown over the years to be a 10000+ line, indicator-laden monster. If you’ve been away from the program awhile and you need to make a non-trivial change, you need to spend part of a day, at least, reviewing the program’s processing.

I am normally conservative about program changes. I generally do not rip apart code and rewrite it, especially if it is not broken. But if a program is to be revised, I do generally at least convert it to RPGIV before making even a minor change. As a phrase used by one of my favorite literary characters (Anne of Green Gables) says, doing this allows “more scope for imagination”.

I was making code changes for a conversion project I was working on when I came across a particular program I had seen before and shook my head at. But for some reason, this time something snapped. I just HAD to change a particular chunk of code, even though the code in fact worked. No errors. But how far to change it? I will explain what I did. In the end, what I do will likely be seen as good by some, too intrusive by others, and  not radical enough by still others.

(more…)