RPG and Programming

It has been a long time since my last post. My blog has obviously not been the biggest blip on my radar, but things keep happening, and I say, “I ought to write about that.” So here goes:

The biggest thing was my attempt to write a system to handle financial transactions using sockets over the Internet. RPG was not the language finally chosen; you can do sockets programming in RPG, but much of the problem involved continuing to handle processing while we did nightly batch processing and system saves.

The language I finally chose was Object Pascal. Not a bad choice, but it finally became apparent that I was much too used to doing file processing in RPG; sometimes you have to jump through hoops in other languages that you take for granted in RPG.

The project itself contains extremely intricate data processing. After a while (over a year) of hammering at it, and having a few minor successes and many major roadblocks, I finally threw in the towel. I suggested using ASNA’s visual RPG, but it seems that the process is being passed off to another programmer more familiar with network programming. I continue to believe that the process could done more cleanly by buying a package, but for some reason there is an aversion at our company to that - they prefer home-grown. At any rate, I am glad it is out of my hands. I never felt more inadequate as a programmer than when I was trying to make this process work. I was willing to keep on trying, but I would not have been able to make any promise as to when it would be done.

As always, I promise to try to write more. Hopefully, this will break the ice.

Sometime in your programming life you will get a reminder that you are human – unless, of course your ego is such that you will never admit to error. One of those times happened to me a few weeks ago.

I had finished a substantial system a short time before, a system with only one screen program, but with a number of batch processes to update data, print reports and notification letters (converting them to PDFs) and email them to users to send out, at their discretion, to customers. I created four or five new files and used Aaron Bartell’s excellent RPGMAIL program to produce and send the letters. It was all original programming, very much out of my largely maintenance/upgrade routine at a longtime IBM Midrange employer.

I had developed a very possessive feeling about “my” code. I tested and retested my code heavily, so I had a great deal of confidence in my code, and I felt there would be little more than trivial revisions or corrections from then on.

So it bothered me very much that a user had submitted a trouble ticket telling me that he was seeing instances of dates being pre-filled onto my display screens. The user is supposed to put in the date he notifies a customer about the issue being dealt with by the letter; but the user was telling me that the dates were being pre-filled by the system, and that they were in fact usually old dates, before the letter was even generated.

THIS could not be. The program was not designed to pre-fill dates. So it was not happening. It must be user error; he must be putting in erroneous dates and forgetting about it. MY program was not pre-filling dates. I double-checked my screen program code, then (fortunately, not out loud or in print so anyone else would know) dismissed the issue as user error and went on to other projects that were really more important.

But time went on and the problem persisted. It was brought to my attention again. It finally dawned on me that this was happening too much, and the pattern was too persistent, for it to be user error. So I decided to dig a little deeper and look at the file update programs.

And I found the problem. When I was building history records, I read the history file itself first to prepare for further processing. Then, when appropriate, I wrote a new history record. What I forgot to do was initialize the record format before writing a new record, or at least make sure I explicitly filled every field in the format before writing it. I missed a few fields – and one of those was the date displayed in the screen. The date needed to be initialized to zero, but I was writing the date from a previous read of the customer I was working with, or even another user entirely. So I fixed it, tested it, and notified the user to keep an eye on it.

No matter how skilled you are in programming, you must realize (hopefully, without being forced to do so by an irate user) that you are going to make stupid mistakes. When I started learning to program back in the late 1970’s, I heard and read a number of stories about programmers with massive ego problems. Usually, the story ended with an example of a big problem created by the programmer who thought he was a genius. Supposedly you have to be smart to learn how to program, but often smart people must learn that they are not as smart as they think they are.

So I am glad that I was reminded so gently that it is a smart programmer who learns from his mistakes and realizes how stupid he really is.

In my favorite novel, “Pride and Prejudice”, Elizabeth Bennet says archly to Darcy, “I have always seen a great similarity in the turn of our minds. We are each of an unsocial, taciturn disposition, unwilling to speak, unless we expect to say something that will amaze the whole room, and be handed down to posterity with all the eclat of a proverb.”

Perhaps a reason similar to this is why I have not been as faithful to this blog as I ought to be. I feel like I need to say something IMPORTANT. And INTERESTING. Sometimes, no topic seems important enough to grip my interest until I get home and go through all the other stuff a head of the house does after work, eat dinner and rest in front of the TV for a short time. (Hmm.. I’m beginning to sense a pattern here.) A little topic may come to mind at work, but by the time I get home, it’s gone. Even if I write it down, I then forget to look at the piece of paper.

The day-to-day life of a programmer is not really that interesting, unless he is absorbed in a project or problem.

The life of a maintenance programmer can be tedious when his mind is not fully occupied. I have a number of projects in process, but nearly all of them are at a stopping point. Not done, but at a point where my next step waits on the actions of another, usually a user.

There are no issues of cosmic importance to be resolved. I am writing new code in freeform RPG. I am not messing with the structure of old programs just to improve them. I convert them to RPGIV when I make modifications to them, but usually I do not attempt to get them all the way to freeform; with all the indicators dancing around in them, freeform usually ends up being an even worse mess than what I start with.

There are old fashioned date routines in many of the programs. I do not touch them unless the modifications have to do directly with date calculation. The basic calculations can be easily replaced with my date functions based on IBM APIs; but they don’t always plug in neatly, which means intensive testing that I don’t want to do when I only want to change a heading or tweak one or two lines of code.

There are two RPG programmers in our shop. We both code in freeform, so there are no areas of conflict there that would make life in our shop exciting.

Somehow, the areas of contention with other programmers (inhouse or online) like control break processing seem rather dull. I do what I want and no one is around to argue with me. (My wife gets after me because it sometimes seems like I like to argue..)

Of course, as I press on into my mid-60‘s, I suppose that decreased conflict is probably not a bad thing. Anything can happen, of course, but I see little likelihood of my job situation changing before I retire in another 10 or 15 years.

But maybe if I picked up on the little ideas that pop up during the day and run with them at night, I could be more productive. Maybe if I had a better way of holding onto them. Maybe if I could get a little spiral notebook.. or a netbook… or a tablet…

A problem I have had to deal with at home has caused me to meditate on the mentality a programmer must have. Of course, my wife suspects that I may have a mild case of Asperger’s syndrome. If that is true, I would venture to say that there are some aspects of the ideal programmer’s mentality that may resemble autism- and when I say that I do not mean to make light of a serious medical disorder.

(more…)

I know it’s been quite a while since I’ve made an entry here, but sometimes real life intrudes on the fantasy world of the Internet, so I just became too distracted by the real world. Either that, or I’ve just gotten lazy.

I think of things to put into this every day while I’m at work. Unfortunately, I’m at work when I get these ideas, I don’t make a note of it, or I get too tired when I get home, and the idea evaporates.

One idea that has managed to sustain itself until this evening was an observation on the state of programming in the real world – that is, outside my insulated IBM-i world. Of necessity, even my client must run outside programs and communicate with the outside world, and that is when the sadly amusing state of the computing world presents itself.

I listen as others talk about having to get new systems working on the new Windows 7 boxes. I listen to them go through the song and dance about installing this driver or that, and how it becomes difficult to keep the existing programs running when you get new hardware or install new software or both. After a while, you notice that it’s a new rendition of an old song with new singers and a new group of orchestras.

In the IBM-i world, of course, this stuff is nonsense. We are currently running programs that were written back in the 70’s and 80’s. With this IBM minicomputer (as it used to be called) family, it made no difference. The programs might need to be recompiled when you changed machines, but you didn’t need to create a new version of it- you only did it if you wanted to. No wasted effort. Nothing about the programs had to be changed if the operating system changed versions. But in the Windows world (and I believe in the Linux world too), there are no such guarantees. This always puzzled me when I looked out at my Windows.

And it has touched my world. I played a little bit with VisualAge RPG, and one of my coworkers ran with the idea and wrote an entire system, albeit a relatively small one, using VARPG to talk to our AS/400. (I know, I know. It’s not what it’s called now, but it’s the name most people in the industry identify it with.) We thought it was a very slick piece of work, even though we were aware that IBM wasn’t exactly pushing it. All we knew is that it was a very easy way to leverage our RPG skills into the Windows world. The users loved it.

But now, as time goes on and we get Windows 7 machines in, we have to make sure that if the system goes on a Windows 7 machine, it must be set up to run in XP-compatibility mode. But how long will that last? How long will it be before another “advancement” in Windows takes that option away from us? How long before it no longer becomes “cost effective” for Microsoft to have an environment in which our programs can be run? Then we either have to rewrite them in another language, or we encase them in a virtual machine where it no longer matters what the outside world (operating system) is. I suppose I should be thankful that we at least have that option.

And you can tell that the mindset of the computing world in general is that somehow this is a good thing. Throwaway software. They worry about 32-bit and 64-bit software and the incompatibilities inherent in them. But it’s “progress”. I call it wasteful – but who am I? Just a grunt programmer in an increasingly less popular computing ecosystem.

Sorry, but I don’t have any solutions. I hope somebody does.

I have been reading and thinking about the future of my favored programming language and the computer platform upon which I use it. The latter has been the subject of an extended discussion in LinkedIn. Some feel that the naming of the machine is of little consequence, unless you use the wrong one. It is felt that to call it the AS/400 is inaccurate, since the AS/400 is not called that anymore- which is, of course, true. It is felt that RPG will not survive for an extended period of time because of its intimate connection via DDS (Data Description Specifications) with the 24 X 80 character display green-screen terminal. Others feel that newer innovations may allow it to survive, but only if it dispenses as quickly as possible with the fixed-format (“punch-card”) calculation specifications and move quickly to free-format. There are not many defenders of fixed-format RPG around, and those that use it are probably expected to turn in their coding sheets soon and retire. And, as always, proponents of each view have their own facts and anecdotes to support their view.

I could not hope to make much of an impression on that forum and accomplish anything more than stir up the pot of contention. It is much better to make a one-sided argument here. Anyone who cares to argue may feel free to do so. (more…)

It has been a long time since my last entry. Much has happened since then. I made two 2100 mile round trips by car back to Michigan, both of them in connection with my mother, who passed away in June at the age of 98. Other personal situations have cropped up too.

At my place of work, I continue attempting to gradually upgrade the system. In addition to my assigned projects and troubleshooting, I am attempting to use more modern techniques; but I find it interesting to look back and see what things I have tried and not tried to do.

One thing that comes to mind is that I am no longer determined to find a way to gradually move code to free-format. If I write new code, it will probably be in RPG-free; but I am no longer attempting to translate fixed format to free. The job is just too daunting; so many of the old programs are just hideously loaded with indicators, and to do a direct conversion, using Linoma’s RPG Toolbox, creates code that would, especially to a neophyte, be even more confusing than its fixed format parent. With proper use of the toolbox we can eliminate left-hand indicators while leaving it in fixed format, and the tool does a very nice job of cleaning things up.

In another context, I am making changes that effectively are advancing beyond RPGIV as presently constituted. Using IBM routines CEESCEN, CEEDAYS, and CEEDATE, I am trying to free the system from its self-imposed reliance upon a century that runs from 1961-2060 for 2-digit years, as well as the ILE RPG range of 1940-2039 for 2-digit years. The boldest move, of course, would be to change all date references in the data files to at least an 8-digit year, if not actual date formats; but in ancient code from multiple systems, that would take a prohibitive amount of time to convert hundreds of files and programs to do this. What I have done instead is create a number of programs based upon the IBM programs above to quickly do date conversions. In doing this, I am implementing a sliding century, such that the two-digit dates will always reference a century that references the current year as the 40^th year of the century. Currently, this means that my floating century is 1971-2070. In ten years, it will be 1981-2080.

A typical call would be: To convert Gregorian (MMDDYY) to Julian, we would call:

CALL ‘GTOJ’

PARM ING 6 0

PARM JUL 5 0

You may note that I am using the RPGIII/RPG400 style of calling the program rather than using the one-line form: CALLP(ING:JUL), which involves the creation of appropriate prototypes. The program GTOJ, for instance, does consist of a module DCONV, which sets up and runs the IBM routines, and module GTOJ, which sets up the input and output for DCONV. They are then combined into program GTOJ. All the routines ultimately use DCONV.

However, I could not see the point of the extensive housekeeping involved in making the programs that call GTOJ and the other programs modularized. You, of course, have to create prototypes for each of the programs (about 15 of them, at last count). Of course, I could put them all in one service program, which to my mind creates another layer of unnecessary complexity. The chief justifications for this seem to be performance and the ability to catch inconsistencies between parameters of the called and calling programs. To me, these are not sufficient reasons. In this installation, performance will never be an issue. Trust me. Our new machine (stupid word processor! I can’t put our favorite machine’s name here – OpenOffice keeps trying to capitalize it) runs at least 10 times as fast as the one it replaced. As for parameter checking, any errors last until the first test is run, after which they are corrected.

Another thing that I have cooled on is attempting to get away from MOVE, etc , in existing code. New code that I write contains very few MOVEs, except for enabling date arithmetic. I just don’t see the point in changing existing code to remove MOVE, unless by doing so I can make the code clearer. IBM has not implemented MOVE in freeform, so I don’t see a meaningful amount of existing RPG code being moved to freeform. This means that any RPG neophyte is going to have to learn to deal with both freeform and fixed RPG formats in his maintenance programming.

So time marches on. We continue to watch the progress of IBM with its initiatives regarding RPG, and we wait to see what impact it will have on our day-to-day work. In a small installation like ours, I don’t think we will ever be cutting edge.

My recent experience on the forum on Bob Cozzi’s RPGIV website underlined the fact that I am not in contention for the title of “World’s Best RPG Programmer”. I posed a question that involved the use of APIs and prototypes, and I submitted my sample code. I got my question answered, but not before it was made very clear to me that my abilities as to APIs and prototyping were considerably below cutting edge.

How do you get to be a good programmer? You have to be intelligent, which can mean you have to know when to be as stupid as the computer. You have to be intolerant of errors. And you have to be willing to learn from your mistakes, as you will make thousands of them.

But beyond these basic qualities and attitudes, other circumstances may determine how deeply you will get into the more arcane aspects of your chosen programming language that will allow you to be among the “best”.

For one thing, your circumstances have to be such that you are exposed to advanced programming techniques. This will likely also be a function of your intellectual curiosity. But perhaps just as importantly, your circumstances have to be such that you will actually have a need for these techniques. You may never be in a position where you will need to access user spaces. While you may see some benefit to variable length fields, you may not see any burning need to start using them. And while much of the benefit of ILE is built around things like APIs and prototypes, it may well occur that your site simply does not need a wholesale conversion of code to make use of called procedures.

As it happened, my experience above came as a result of a need for them. Two-digit years are still used on my system. In calculating the maturity date of a 30-year loan written in 2010, I bumped up against the ILE default for two-digit date fields (1940 to 2039) and got an “invalid date” error when I attempted to generate 3/15/40 to represent 2040. To get around this, I decide to use an API I was aware of, CEESCEN, which allows you to float the 100 year range you want to use for the 100-year period in question. This is used in association with APIs CEEDAYS and CEEDATE to allow you to format dates in numerous different formats. (Google the API names for details.)

However, I was not familiar with how to set up the prototypes needed, and as a result I made some stupid mistakes that leaped out at the knowledgeable participants in the forum, especially Bob. But anyway, with some help, I got the prototypes and program calls to work.

But my learning things like these are in response to a specific need. I don’t have a burning need to learn all or even a substantial portion of the APIs available. I simply don’t need them. The fact is that I am getting paid to write production code, not specifically to “learn new things” . And the fact is also that I am not in a position where I can spend a lot of time outside the workplace learning new stuff, since family and other personal needs and circumstances have first claim on my time.

Other people, by reason of education, career choices, and the employers they happened to have, may have had open to them early in their careers opportunities to work on truly advanced concepts that I never was exposed to and likely never will be. Of course, their being smarter than me, not to mention better educated early on, would also make this more likely My task being to maintain and upgrade code that was from the 1980’s and 1990’s in style, if not always in vintage, I will likely be kept busy doing that for the balance of my career. I like to think I’m good, but I’ll never be the best.

I was going to discourse on another topic, but a response to Buck (to whom I apologize, along with Bill, for being derelict in checking my site’s e-mail), who commented on my previous post, demands a reply. He feels that lack of formal education, while being a contributing factor, is only a contributing factor; rather, it is the midrange habit of being exposed to one programming language.

Actually, I am inclined to agree. The isolation in RPG is real. But we also do well to consider the background of the situation. The earliest IBM minicomputers, the System/3, had as its programming language RPG II. That was it. Small businesses who wanted that computer worked in that language- at least, I’m not aware of others used on the machine, besides assembler.

When the System/34 came out in 1977, the choice of compilers widened. BASIC, COBOL, and Fortran became available. But still, RPG dominated. I’m not surprised that Fortran and BASIC popularity did not rise. But COBOL was big at the time. However, RPG was already a business-oriented language, so COBOL was not really needed from that standpoint, and RPG was perhaps felt to be more terse and easier to learn. Then too, COBOL programmers were already being kept busy keeping the IBM mainframes humming, so there were probably not an excess of them floating around. COBOL has never been much more than a peripheral language on the IBM midrange; useful probably only to those whose business already depended in some way on COBOL, perhaps having COBOL talent already available.

In addition, and perhaps most important of all, The System/34 had no need to talk to the outside world unless it were to another IBM minicomputer or mainframe. There was no need to handle HTML; there was no HTML, because there was no Internet.

This is not to say that RPG was a perfect language. String manipulation beyond the realm of MOVE and MOVEL was very difficult. The two things the designers never got right until RPGIV came along were string manipulation and date arithmetic- being able to get the date 30 days after today, for instance. (One thing I will never comprehend is why they have never been able to enable multiple dimension arrays. In earlier days in a different environment, trying to emulate multiple dimension arrays was a real pain.)

But since the programmers had no other languages to work with, they bent RPG to their will. They created these plug-ugly array- manipulation routines to piece together strings. And they created complex routines to do date addition and date format conversion. We must not forget the famous “magic numbers” - to convert a date in month-day-year for mat to year-month-day, for instance:

MMDDYY MULT 10000.01 YYMMDD. I am still reminded of the RPG programmer of my acquaintance who, with the help of engineers, created sines and cosines and other functions in RPGII.

Now, there may have been elegant functions created in C for such purposes as string manipulation and mathematical functions. But once RPG routines were developed, the C functions were not needed. It really didn’t matter. They couldn’t use C anyway. If you knew what you were doing, you could write assembler subroutines – but that was it. I don’t think RPG couldn’t even call COBOL programs; at least the techniques were not well known for doing so.

So when ILE came around, much of the motivation for calling the now-available C routines was missing. We didn’t need them. RPG routines were already doing the work. Unless there were some performance considerations, programming managers would probably take a dim view of programmers who ripped out the ugly-but-tried-and-true RPG routines just for their own amusement. One could make a valid argument that those routines were a maintenance nightmare. But unless management had a good deal of foresight, this might be a difficult thing to prove to management’s satisfaction.

And yes, programmer inertia was also a factor. My shop has had IBM midrange machines since the 1970’s, and AS/400s since the 90’s. But the first RPGIV program was written on this system in 2007, by me after I arrived.

But now, the IBM midrange does have to deal with the outside world. My company already has an Internet presence, and we quite frankly don’t have to know PHP or Java or the other languages in vogue in order to make the business work. All we have to do is make sure our Internet provider gets the necessary data. And RPGIV provides the necessary tools to alleviate some of the issues that plagued us with earlier versions of RPG. If we need outside routines like APIs or C or Java or Rexx, they are easy enough to call. I have expressed myself in earlier posts about my ambivalent feelings about freeform RPG.

But mentioning freeform brings us to the topic of: How do our employers find replacements for us fixed-format loving old codgers when we retire? That is a topic for another time.

I have been thinking a bit about how our backgrounds, educational and otherwise, may determine at least to some extent the style we use in our programming and indeed perhaps the language we use.

One of my fellow programmers (about my age) graduated from college with a background in computers. Almost immediately he was able to find a position programming in COBOL, then in ALGOL, which is a block-structured language with a relatively free format. He continued to do well at various positions; ultimately, he came to where I work, with many years of experience in programming, but none in RPG. The style here could be described as a blend of RPGII, RPGIII, and RPG/400; fixed format to the core, loaded with indicators. He learned RPG from the ground up in that environment. Sometime after I came along, he was shown freeform RPG.

I, on the other hand learned RPGII from a book and practical experience. I had no computer background when I started to learn. I was of an inquisitive spirit, though, and I advanced in my use of RPG as the language itself improved. I too ultimately became acquainted with freeform RPG.

Now, who do you think uses freeform RPG whenever he can, especially on new programs? You might have guessed it was my partner- if you did, you’d be right. RPG was actually stifling to him; given the freedom to use free format RPG along with the BIFs that work well with it, he has been liberated. On the other hand, as you may judge from this blog, I just have not seen the light or the point. When I write code for new programs, it is in fixed format RPGIV.

We learned to love the way we learned programming, and so our preference now is to do it the way have always learned it, just better.

Illustrating the point that the way we learn to program is greatly influenced by our programming origins was a discussion I just read on an RPG forum about the significance of certain characters when writing CL (Control Language on the AS/400). Instead of indicating a “not equal” test by the common expression *NE, the code used the combination ^=, where the ^ is really a character I can’t find on my keyboard, but looks like the upper-right-hand corner of a rectangle with the top leg longer than the right. The character, as it turns out, is the “logical not”, a character often used in Boolean algebra.

Now, I don’t know who originated that code. Some experienced programmers, though, just did not know what it meant. Therein may lie a tale.

During the time I learned, many if not most of my fellow programmers were not products of any kind of computer science curriculum. They did not learn BASIC or Fortran first, and usually not COBOL, though this was a bit more common. They learned RPG as their first language. They did not learn “computer science concepts”; many, like me, either never attended college or did so only briefly. They may have started as a computer operator, or maybe at another position in their company. Usually it was a small company; it had the smallest capacity IBM computer available, the System/3, or System/32, or System/34. Something about the programming process intrigued or excited them; they persevered, and they advanced.

As time went on, though, and RPG increased in importance, people with a background in computers and computer science began to program RPG. For a long time, they would prove to be stifled if they expected to program like they originally learned to program in other languages. And where the original programmers may have simply accepted, for example, the RPG cycle, and comprehended it entirely, by the time the System/38 came around, programmers who started learning RPG as RPGIII on the System/38 evidently found it mind-boggling.

I can’t count the number of times I have found a test for LR (last record) in the RPG detail cycle in code (usually written by programmers who came from the 38) using an input or update primary file. Unless explicitly set on with a SETON instruction, the LR indicator will never be on during the detail cycle. A programmer who started with RPG II would automatically know this – it would be a part of his upbringing. A programmer who started with a background in computers later would not necessarily know this. This would cause problems if the programmer intended to base output on this indicator test. Usually it would mean that last record calculations and total output would be missing.

Another difference in outlook comes with file processing. A programmer raised on RPGII (usually in a small shop) doesn’t mind the system reading files (primary and secondary) without him telling it to. But I have noticed in discussion forums that people raised on other programming languages (usually learned in college) seem to get bent out of shape if they don’t explicitly get to do the file reading themselves. They don’t really comprehend that this really is quite an advanced feature, much closer to SQL than Pascal.

It would be interesting to find a survey of programmers that told what level of education they had when they first started programming professionally. I would suspect that as a group, RPG programmers would have a slightly lower level of educational advancement than, say, a C or Java programmer. This would especially be true if they surveyed the most experienced RPG programmers. But then, it is results that matter. RPG programmers do not write compilers – we process information so that humans may comprehend it.