Layers.library V45 on the aminet

[olsen]

The problem goes deeper. The problem is that it's very likely that you do not even realize that you have an architecture problem somewhere in the code if you write everything in assembler. If you take enough time, you'll probably get working code if you work hard. But you're still lost irrelevant details. It's likely that you picked an algorithm because it looked on the microlevel pretty ideal. But whether that's relevant for the big picture is another question, and you'll easily loose the big picture in assembler - you're not forced to organize your code, and you don't have a compiler that helps you at the detail level.

If somebody is deeply committed to using the language of his choice (say, assembly language, Perl, Visual Basic, Delphi, you name it), and has a clear idea of the limitations of the language, then there is literally no problem he cannot solve using that language, even if that means having to put in extra work to solve it.

You can mention how much more leverage a different language provides, tell war stories to illustrate the time and effort you saved by switching tools, but it won't leave any impression whatsoever. If somebody grew up learning one programming language, and found that it is fit to solve all the problems he ever encountered, never saw the need to look beyond it, you won't be able to talk him out of it.

Which is fine, until you have to collaborate with such a savant and find common ground to work with him. I trust you've been in this kind of situation, and so have I.

[olsen]

Thoram: code an IPv6 capable TCP stack for AmigaOS in asm, it is very much needed, is perfectly doable and worth while on 68k, and complex enough to drive you insane.

Don't tempt me

But seriously, implementing an IPv6 TCP/IP stack in plain 68k assembly language should be doable. You'd have to start off small with the 1983 TCP/IP implementation, as documented and used in 4.2BSD, upgrade it to the 1988 implementation (with congestion control support) and then you're basically in business: this can be upgraded for IPv6 support.

The hard part is in fitting this into the bsdsocket.library framework which already exists, so that it can be used for existing client software on AmigaOS (and not just exist as a proof that anything is possible in 68k assembly language, even if you have to forgo shaving, bathing and seeing your family for an extended period of time). There is a blueprint in AmiTCP, but it will only get you so far. The only IPv6 API support which I am aware of exists in Miami Deluxe, and it has never been replicated; the number of IPv6 clients on the Amiga always was very small, too, which would make testing difficult.

Anyway, before somebody even thinks about it: don't write IPsec code in plain 68k assembly language, because this is going to end in tears. As soon as you're having to do serious heavy lifting in cryptography you're best advised not to write the code in a language which is difficult to audit and review for errors.

[olsen]

@Olsen @Thor,

Since it's too late to go back to school, could you please give a list of good books to go from a "coder" to a "software engineer". Being about C, C++, Algorithms, Design patterns whatever else suit you.

Thanks
Kamelito

Well, you asked for it

The difference between coder and software engineer, as I'd like to frame it for the purpose of pointing you to the books which completely changed how I work, is that as a software engineer you are going to find yourself doing more challenging and complex work, and if you persist in practicing what you learned as a coder, you will start hating your work. Something has to change between "coder" and "software engineer", and it's *you*.

If you are writing software, you will make mistakes and will need to correct them. This inevitably means returning to old code, rereading, understanding and reworking it. You will be surprised by how long-lived even tiny throw-away snippets will be. For this reason you need to write code in such a way that returning to modify it will not be painful. In so many words, the way you "construct" your programs matters.

There are three books which profoundly changed how I work. These are "Debugging the development process", "Writing solid code" and "Code complete". The first two were written by Steve Maguire and are probably out of print (but can be bought used rather cheaply). The last was written by Steve McConnell and is still in print. If you are going to read just one of these books, make it "Code complete". The drawback of "Code complete" is that it's rather long, though, while the other two books are shorter. However, I found that reading "Code complete" is exceptionally rewarding. Every chapter you read will change how you work and build software, if you are willing to question the way how you work today.

As for books on algorithms, I wish I could point to a set that was helpful to me, but the fact is that most algorithm collections are either very mathematically inclined, or used as teaching tools in computer science education. This means that if you need a solution for a practical problem you'll either have to wade through pages and pages with greek lettering and proofs that, yes, the algorithm actually does work (never mind how it works), or you'll get useful information on implementing algorithms from which one crucial piece is missing because the book uses that missing piece in a course exercise (you now know how to correctly construct a B-tree, but you don't know how to remove nodes from it, and you'll probably make big honking mistakes building an implementation of that, and it's three in the morning and your project is due by noon tomorrow, and you just ran out of coffee, and you are wearing sunglasses because your eyes hurt, and what could possibly go wrong now).

So... the next best thing I can recommend is a book that explains the basic algorithms which you are going to find are used pretty much everywhere. There are highly specialized algorithms, such as for cryptography, graphs or numerics, but you'll probably meet sorting and searching in most of the work you do. These basics are decently explained in books such as "Algorithms" by Robert Sedgewick. This book is still affordably priced, and the older used versions aren't so bad either, and cheaper.

The big problem with algorithms in general is that you need to know where to look when you are working on a problem which requires a better optimized solution than the basic stuff you knocked together. Not everything can be tackled as an application of sorting or searching algorithms, or whatever tricks you learned so far. But before you can look for a solution, you need to know where to look, except you don't even know where to start looking. Sound familiar? I found help in a book called "The algorithm design manual" by Steven Skiena. It contains a big catalog of algorithmic problem categories. Flip through the pages, read the introductions, consider if they generally match your problem, and if they do, look up the material listed on the page. There is an online version to complement the printed book at http://www3.cs.stonybrook.edu/~algorith/, don't miss it.

[olsen]

For me, learning from books is a mistake : the better and best way is to learn by yourself.

I have to disagree with you on that.

The best way to learn is to experiment/explore, make mistakes and find out why you made them, then avoid making them again if possible. The problem with making mistakes is knowing when you are making them.

If you are lucky, you will, after a while, realize by yourself that you made a mistake or left an alternative option to solve a problem unexplored.

You don't always get lucky. It's by far more effective to know somebody who understands your work, can give you his opinion on it and offer insights.

A third person may be able to tell you when you are on the wrong track, drawing the wrong conclusions or have falsely convinced yourself of having explored a problem to the fullest. All by yourself you are far less likely to arrive at the same conclusion.

There's a term for that: confirmation bias. Don't believe that you are immune, you're human like the rest of us.

[olsen]

Wikipedia seems to have some good articles about software engineering.

Software engineer: http://en.wikipedia.org/wiki/Software_engineer
Software engineering: http://en.wikipedia.org/wiki/Software_engineering

These articles are, frankly, a bit scary. Back when I started out programming computers in my spare time, as a hobby, I didn't realize how long the road ahead would be before I became a halfway decent programmer.

Part of the journey took me through practicing the craft (which everybody can do in his own home, without any heavy lifting involved, except for flipping through those heavy 'C' compiler reference manuals and operating system documentation), part of it took me through a professional education.

If you didn't know with all your heart that programming was fun, interesting and worth the slog, you probably wouldn't want to follow through the professional education, as it is too often tedious, impractical and unfairly challenging. This can be pretty dry and boring (I hope that I'll never have to sit through another lecture on stochastics or information theory; luckily, I got my degree and probably never will have to sit through any of these again!), and these two Wikipedia articles are far from making this profession appear to be something you might want to spend the rest of your life practicing.

As for the books I mentioned, I think they are helpful for practicing the craft even if you don't consider going the route of getting a professional education in the field. What you will find in them metaphorically makes the difference between chopping down a tree using nail clippers or, well, an actual axe, when it comes to programming.

[olsen]

No, no, sorry : the authors of your books will "programm" the readers beginner to follow a way...

This is what an education does, doesn't it? Same thing about religion...

Programming and the different practices of programming have been invented and reinvented multiple times, starting with the 19th century, and have finally been reduced to theory in the 20th century.

You are likely following the invisible footsteps of somebody who came before you. Your journey and the discoveries you made will be your own, but there is a wide world out there which already shares these ideas and has been expanding upon them for more than 70 years.

The guys outside your room who practice the same craft know a thing or two which you are missing out on.

For me, the best is to learn by yourself, to "build" YOUR own way with using (and developping) your imagination and understanding...

Anyway, do what you feel better for you, I just give here my opinion...

Best of luck to you! If this is what works for you, who am I to criticize you?

[olsen]

@Olsen
>Well, you asked for it

Thanks. About Code complete AFAIR the first edition is about C and the other about OO languages. Is the first edition enough?

I only read the first edition, always was planning to give the second a look, though.

I read good comments about Programming Pearls any thoughts about this book?
Regards
Kamelito

I wouldn't have recommended it, because it covers so much ground. There is programming practice, the matter of verifying that you got your implementation right, understanding how algorithms work, how optimizations can be made, and when they are useful (estimation methods, why optimization matters). As a book on getting to know your craft, and when to try picking a better algorithm than you already have, and why it may not be a good idea, it's as good as it gets.

The chapter on algorithm design techniques which has an example of how an algorithm in interpreted BASIC on a TRS-80 using a 1.7 MHz Z-80 CPU can beat a different implementation of that algorithm, written in 'C' and running on a DEC workstation using a 300 MHz Alpha CPU is worth the price of admission alone!

Enjoy

[olsen]

I'm wondering if OS 4.x is tested against tools like purify, valgrind or PVS Studio or any others tools like that to improve code quality and if automated tests cases are run before production.

Kamelito

As far as I know none of these tools exist for the OS4 target platform. For Purify and Valgrind that's a real dealbreaker (Valgrind needs an x86 platform, Purify is available for IBM Power architecture, but still seems to need a POSIX operating system to work its magic). As for PVS Studio, it seems that it could be used for static analysis even if the code produced was never intended to run on the PowerPC. I have no idea how good the static analysis capabilities of PVS Studio are compared to, say, LLVM-based Xcode, but I've used Xcode extensively for a small Amiga project of mine, and it certainly did help to shake a large number of bugs out of the code.

As for OS4 development work, if such code quality tests were available for the development environment, and properly integrated, it certainly would be very useful. Back in the day when Commodore switched from using a cross compiler to native development, employing the more modern Lattice and Aztec 'C' compilers, code quality did improve.

[olsen]

That's an interesting question - what defines a coder vs. a software engineer. Hadn't thought of it in those terms until now.

My personal revelation from my own growth experience is I feel software engineering is more about code development process than knowledge of design patterns/languages, etc. Of course those are important too but I think without good process you are going to have far more problems than you should with your resulting code.

My personal take on the difference between what these terms stand for is that the software engineer has a significantly bigger toolbox to draw upon than the coder. For the software engineer, the toolbox contains such things as process, a common language (terminology), and a big collection of methods which have been tried and tested.

None of this is the exclusive domain of the software engineer, the coder can use these tools, too. The thing is, if you practice software development professionally and enjoyed a professional education, you acquire this toolbox and it becomes part of your daily work. The coder has to work harder on acquiring it, and chances are that he won't be exposed to tools which a software engineer winds up finding in his toolbox as part of his education that at the time when they show don't seem to matter much. Only lateron some of their value may become apparent, and then it's often of crucial importance. A coder may not see the immediate value and discard that tool.

This sounds pretty abstract, so I'd like to give an example of such a tool that might show up one day and prove its value lateron: software configuration management (SCM), using tools such as SCCS, RCS, CVS, SVN, git, you name it. One can make backups of the source code as one maintains and modifies it, and that may be sufficient if the code base is small and the changes are sufficiently small as to be manageable. But one might cross the threshold when this is no longer the case, and if one hasn't picked up the habit of using an SCM, one may not realize how much work and effort it saves in the long run.

A might think a coder writes code without really caring too much how they get something done as long as they get their immediate problem/feature working. They have their own way of working that they feel works for them.

A software engineer employs a large number of long standing good practices that help in creating large and complex programs that contain few bugs.

Somebody might argue that a coder can accomplish the same thing through grit and determination alone

The big difference here may be in that you learn to use tools to do the work for you which the coder still has to do all by himself, using his brains alone. If a task can be performed through automation, it is reasonable to assume that it can be done more quickly in this form than a human could, and it's probably quite boring to begin with. It will save you time and effort which you can spend elsewhere. Instead of combing your code for problems, testing it one problem fix at a time, you can focus on different and more complex aspects of the software.

I would say for the first 15 years of my career I was more of a coder. In the last 10 years I feel I'm more of a software engineer. I evolved because people that were smarter/more experienced than me were brought in to make a concerted effort to increase code quality in our large development studio.

While there are many processes I have learned and employed to write better code, the main differences that saw me evolve were moving to test driven development and automated testing. Hackers always avoid this as they feel it is too much work and will slow down development (they are usually too busy fighting fires due to their undiscovered bugs to have "time" to make the switch).

I know all too well to what you refer to  What always gets me is how programmers can be so confident and effective in making their lives not just harder, but really miserable by working without the assistance of simple tools which others use regularly.

You certainly can create code with a simple text editor, without using an SCM, test the code manually without even using a debugger, crank out code which makes testing hard, makes the code hard to read, etc. but it's not going to make you happier or make it easier to collaborate with your collleagues. You'll probably just end up spreading the misery.

Yet this is par for the course for many programmers you are going to meet. It's not just that they work like they do, by rejecting just about every tool you have been using to great effect in the past 20 years, they tend to insist that you have to work like them because they find it difficult to understand why anybody would even bother using different tools and methods than they do. Things tend to get difficult at this point

I'm tempted to write all sorts of stuff describing this but I don't want to waste time, Google it if interested. All I can say is since adopting these practices, only maybe one or two minor bugs a year are found by my customers in the code that I write. It's not that I am super smart or write perfect code, it's just that I find all my bugs before I release my code by writing thorough tests (using code coverage tools, etc.) - and I only have to press one key to run thousands of tests across all sorts of platforms and configurations - something that is always done before code is checked into source control. It gets to the point where this becomes a safety net you cannot live without. If I need to change some existing code I don't even need to think the ramifications through much - my main question will be, does it pass my existing tests? Oh it does? Code is probably good then. To think of trying that without test scares the crap out of me now.

Now when I see other people just coding something up, running some quick manual test that covers maybe 20% of the code paths, and just "figuring the rest will work because it should because I am smart" I have to shake my head. Sadly I work on a team where the other coders still do this. They are constantly chasing bugs, and frustratingly my tests in my code area run into all sorts of their bugs in their areas (these are large complex projects), and it is unbelievably expensive to have to continually communicate the problems and schedule fixing when they could have just found the bug while writing their code so that I didn't have to waste hours dealing with something they have to fix anyway.

This sounds painfully familiar

My personal impression is that if you are going to have a life and a job writing code, then you'd better make writing the code a task which does not impinge on your life. That means having a plan for dealing with the task at hand (a process) and a big enough tool box to draw upon when facing the unexpected. If you are going to slip and fall, you ought to be able to stand up and try again.

[olsen]

I've never called myself a software engineer because I'm not that pretentious, I preferred the term software developer. If I was arrogant I'd call myself a software architect, which trumps your made up software engineer status.

As far as I know the term "software engineer" is not a "standardized" professional term.

It has been used as shorthand for applying the professional standards which engineers are held to in the field of software development. Not a bad idea, considering what a mixed bag the professional standards in software development tend to be. If I remember correctly, the term "software engineering" goes back at least thirty years, and look how far we've come since then :-/

The term has also been used to denote differences in status between the lowly programmer, the programmer who has leveled up twice, and so forth, up to the apex of the imaginary pyramid, where presumably the "principal architect" sits (what do you give a programmer if you can't pay him a higher salary? a fancy job title) and looks downward, smiling benevolently and drawing the kind of salary which allows him to smile, if not smirk, and hardly lift a finger actually doing any programming work

The term "software engineer" can be anything to anybody. Which is why I mentioned the context in which I used it. Because Thomas used this term to make his point, it does seem prudent to me to keep using it.

As for attaching status to titles, I'd personally be content to be known as a reliable programmer who tries to solve interesting problems and most of the time gets results

[olsen]

@Olsen @Thor,

Since it's too late to go back to school, could you please give a list of good books to go from a "coder" to a "software engineer". Being about C, C++, Algorithms, Design patterns whatever else suit you.

Thanks
Kamelito

I missed one book which might be helpful, and which is also very readable: "The practice of programming", by Brian Kernighan and Robert Pike. It covers some of the same ground as the "Programming pearls" collection, but complements it well with material absent from it, such as testing and debugging. I'd say that had this book been available to me some 25 years ago, it would have opened doors which at the time I did not even know to exist.

[olsen]

Hm, my mind boggles - in my book, SCM is a tad more than just a revision system, Software Configuration Management is more a tool for system administration, rather than software development. SCM also include automated building and testing of software (hudson, jenkins, codeship, cruisecontrol...), change control with peer reviewing before deployment (puppet, chef, cfengine ...) Of course, using such tools can also be usefull for developers too, and these days I dont know any coders who don't at least use a revision control system.

Modern times indeed. You'd be surprised how many people don't even know or care about SCM, even in its most simplistic form. Even talking them about it you're met with the kind of gaze I supposed is reserved for looking at cranky children, axe murderers and the remains of a dead rat which turned out to be hidden in a piece of sponge cake.

You are correct that the purpose of SCM goes beyond the simple stuff, yet it's the simple stuff which is underestimated, and the more complex stuff is ignored because you can do even the simple stuff manually, so why care?

Part of the job of developing software, performing system administration, etc. is not to become complacent and at least have an eye on where the field is moving. You can adopt what you find useful, after you have evaluated it. But in some cases that bit never happens, because things work well as they do. Until they don't, and when they suddenly don't work any more, you're in bound to be in a profoundly unhappy place

[olsen]

I don't use one, it doesn't make much sense for solo projects...

It can make good sense for solo projects, though modern SCMs make it look as if the technology is primarily enabling collaboration.

I started using RCS some twenty years ago in my small projects for two reasons.

The first reason is that the more you change your code, the harder it becomes to keep track of the changes. Instead of a single .bak file for each file you edited, or a collection of backup files which are tagged with a date in their names, you can keep all the changes in a single SCM file. There's also one common interface for storing the changes and retrieving them. Compare making individual backup files with just committing a change with a comment, and then later using the diff command integrated into your SCM tool to look at them, to comparing individual backup files manually.

The second reason is that a project can change scale and scope, and you may not notice how the daily changes gradually push it into a direction which make it harder to manage. The SCM acts as a safety-net, so that if your project crosses a certainly complexity threshold, you can still roll back changes, or review older revisions. Using the SCM regularly isn't that much of an effort. Getting back at the change history you cannot easily recover (because you don't have one, or because it only goes back 1-2 edits) puts a bigger burden on your work.

You don't have much to lose if you use an SCM in solo projects

[olsen]

I keep copies of each older release, and also a changelog of what was done for each new release. Plus, anything which is non-obvious as to its purpose (eg. a change to my emulator which is needed for compatibility with some obscure guest program) has a comment to that effect in the source. Those three measures have proven adequate for me. Plus, last time I checked, these revision control systems seem to require use of a makefile. YMMV; I'm not saying that revision control is useless for everyone, but if your coding is done in a disciplined way it's not necessary.

That certainly can work, but it may not always work. Some 25 years go I could literally remember everything that went on in a program, its configuration files, it data files, what all the constants meant and how they were used.

Nowadays there's not so much of the old zing around (maybe the programs became bigger, maybe the tools got better), and I have become too lazy to do manually what a script, a program, a tool could do automatically for me.

Maintaining a change log is necessary, and it is very convenient to maintain it through subversion (SVN) commit messages, which lateron get folded into a release log file. The subversion repository keeps track of my file changes, the change log data that describes the changes, and it contains the release log file, too.

If need be, I can use the subversion repository to roll back changes I made (undo), and I can go back to old releases for review. All of that can be done by reviewing archived changes, stored in individual files (which clutter up the directory), but with a revision control system you just tell it to retrieve the file change history and you're able to review each small change you made and committed.

If you change one character in a file you can commit the change and it won't eat up a lot of disk space (revision control systems store differences between subsequent revisions), but you probably wouldn't store a full backup copy of that file for later reference.

The subversion command does work which I need to do anyway, and it allows me to spend far less time on it than doing it manually would. There are positive side-effects as well.

Revision control systems such as the old RCS, CVS and SVN do not need a makefile. It's possible that the makefile contains rules for invoking these programs, but these are just icing on the cake. You could invoke these commands manually and it would not change a thing.

What is needed is a place to store the revision information, etc. Both the old RCS, CVS and even subversion allow you to keep the data in a local directory on your computer. Storing that data on a separate system, where it can be safe from data corruption or loss may be a more appropriate solution though.

Once you commit to using a revision control system and clear the first hurdle in setting up a repository, it can become second nature to use it for every project you work on, no matter how small. If you consider each small project in isolation, then setting up the repository may look like unnecessary, unwarranted effort. But if you do start using it, then every project, no matter how small, can benefit from having it around.