show all comments

DSM

Linux, Git, MetaEdit+: how three Finns brought versioning, models & code together

September 20, 2018 13:32:21 +0300 (EEST)

Coders are often accused of being allergic to modeling, but I'm not sure that's true. I think we're just allergic to wasting time, to going through the motions to satisfy some meaningless rule, without actually producing anything worthwhile. And most of all we hate having to do the same thing twice. So management-mandated post hoc UML for documentation was never going to fly.

We love our language, our frameworks and our tools, and it takes something pretty big to get us to break away from them. Modeling may be great for non-programmers, who like the visual format, but it's no substitute for code. Usually.

The most common case I've seen that persuades programmers to use modeling is state machines. If...elseif or switch...case just cannot make things clear, and neither do textual DSLs, whereas graphics work well. Even Linus's kernel GitHub contains a state machine diagram — albeit in ASCII art!

 *             |
 *             V
 *    +---> STARTUP  ----+
 *    |        |         |
 *    |        V         |
 *    |      DRAIN   ----+
 *    |        |         |
 *    |        V         |
 *    +---> PROBE_BW ----+
 *    |      ^    |      |
 *    |      |    |      |
 *    |      +----+      |
 *    |                  |
 *    +---- PROBE_RTT <--+

The horror and time wasted trying to use ASCII leads most to an actual graphical tool. And rather than draw then code or vice versa, with all the usual duplicate effort and things getting out of sync, we should be able to keep the state model as the primary source for this part — just like we have bits of our projects in other languages, when that's the best format. Providing it all gets automatically built and integrated, and versioning works as expected, that's fine. Except, it generally doesn't: hence the ASCII art.

So can we change that: make it possible to sensibly integrate code and models, and to version both? And without horrible amounts of extra work to set up, or (particularly) when using it? And are you sick of the rhetorical questions and want to actually see the darned thing?

Three Finnish amigos: Linux, Git and MetaEdit+

Since the gold standard for coding is Linux, and the gold standard for versioning is Git, and both are Finnish, let's see how adding a third Finn into the equation can help. MetaEdit+ is a modeling tool, and also a language workbench (a tool that makes it easy for you to create your own modeling languages and code generation). Conveniently, it also supports Linux, Git, and integration with other tools like IDEs. And it has a nice little 'digital watch' state machine example with full code generation.

Full code generation?! Is this going to be some horrible flow chart with IFs, GOTOs, and inline code snippets with no IDE support? Or bulky, unreadable, inefficient blub code? Fortunately not. Both the graphical language and its code generator are domain-specific, i.e. fine-tuned for this task (making digital watch apps). So there will be things anyone will recognize, and things that may be new if you've not worked in this kind of domain, but it should be understandable on both model and code levels.

Diagram watch

Here's a diagram of a Stopwatch app. The watch can be in various states, and you can press buttons to cause actions and move to a new state. Actions are things like setting time variables or turning on an icon on the display. In short, the language lets you specify things from the point of view of an end-user of the watch, with the code generation and a little state machine framework taking care of the innards.

You can grab MetaEdit+ for Linux (and its Eclipse plugin if that's your IDE) to follow along. See the Watch Example for more details. 

Models to Code

Let's take a look at the code, then. Select the top-level WatchModels graph from the Graph Browser pane in Eclipse, and from its pop-up menu choose to edit the Graph's Properties... and set the Generation target platform to 'Java: Linux'. Select and open the pop-up menu again and choose Run Autobuild. (If you're not on Eclipse, select WatchModels in the top MetaEdit+ window's Graph Browser tab, from its pop-up menu those pop-up menu choices are Properties... then Generate... | Autobuild.) The code will be generated (and either added to an Eclipse project if using the plugin or opened in a Generated Files window if not), compiled and run.

Generated Files

Java WatchModels Java Stopwatch

We can start the Stopwatch by choosing the Sporty WatchModel, then pressing Mode. As we can see from our model, pressing Up starts it running, and Up again stops it. Pressing Down will then reset it.

Models to Git

OK, so we have models, we have (generated) code, but what about versioning and collaboration? MetaEdit+ has a multi-user version that automatically keeps everyone in sync, without any merges or locking people out while others edit. Nice, and overcomes the nightmare of merging model changes, but still doesn't give us versioning. For that, we can use your normal Git or SVN. Let's assume you want a local Git repo that will also be pushed to GitHub or BitBucket.

First, fork the https://github.com/MetaCase/watchdemo repository into your own GitHub (or BitBucket). In your ~/metaedit directory, mkdir git. Add this new directory and your Git account URL (without /watchdemo suffix) to a ~/metaedit/.vcsPaths file like this:

gitBaseDir=/home/myuser/metaedit/git
gitBaseURL=https://github.com/myuser

Exit MetaEdit+, and ensure you have xterm installed, as the Git and SVN integrations use it. Then run the following command to get a local clone of your online repo:

metaedit textForMERL: "_vcsInitClone('watchdemo')" logoutAndExit

You'll be prompted for your GitHub/BitBucket password in an xterm, and assuming all goes well you can close the xterm at the end. Using the Eclipse Graph Browser toolbar buttons, Refresh it and change its Settings to point to watchdemo as the Repository name. Restart MetaEdit+ from the Graph Browser button. To see what changes were made to the original, see "Extending the modeling language" in the previous blog entry.

Freedom through code

Open the Stopwatch graph by double-clicking it in the Graph Browser (Refresh if necessary, then WatchModels->TST->Stopwatch). If we look at the model for the Down button's action at the top, we can see it's a bit ugly: resetting startTime to zero by setting it to stopTime - stopTime. The 'official' Watch Example tutorial uses that as an excuse to improve the modeling language by adding constants, but here let's use it as an excuse to integrate some 'real' code. Let's have the model just specify a function to call, and we can then implement it in code to do what we want.

Select and delete the startTime, startTime and stopTime connected to the Down button's Action. Click Function on the toolbar and click on the diagram to add a Function called 'reset', then click the Run relationship's ! icon on the end of the toolbar and drag from the Action to 'reset' to connect them (just OK the dialog: there's only one action here, so we don't need to set an order).

Diagram reset

Trying out the integration of hand-written code

Now we can generate the code for our model, build it and run it. Select the top-level WatchModels graph in the Graph Browser and Autobuild again. Choose Sporty, press Mode to get to the Stopwatch application, and press Up a couple of times to start and stop the stopwatch. You'll then be back in the Stopped state, seeing the time you measured. For now, pressing Down in the Stopped state will not change the displayed time: it calls the reset method, but that has no body yet, so does nothing.

In the generated Stopwatch.java file, edit the reset method to add a line between the leading and trailing MD5 comments, to set stopTime to zero, then Save:

		setStopTime(new METime());

(Note that we're editing a generated file: normally this is a no-no, but here the MD5 protected region will take care of things, maintaining our manual code but regenerating the rest from the model.)

Eclipse reset

Close the running Sporty and WatchModels. Rebuild the WatchModels with Autobuild, and try Sporty again with the Mode, Up, Up, Down steps, and see that Down now does reset the stopwatch time to zero.

Combining models and code in Git

So now we have our models in Git, our standard code generated, our custom code added by hand and maintained automatically. The last thing to do is extend the Git integration so that the manual code also gets included when versioning models. (Where possible it would generally be better to keep the manual code more separate from the generated code, but we deliberately chose the harder path here.)

The MetaEdit+ Git integration is handled by generators, so you can tweak it to your needs. In the main MetaEdit+ window, choose Repository | Changes & Versions Tool, choose VCS Settings | Paths, and add the following lines after the settings for $dbName and $dbBaseDir:

/* $srcDir is for source code you also want versioned; '' for none */
$srcDir = __(subreport '_default directory' run 'WatchModels' sep 'src')

Save that and select the _vcsCheckIn generator further up in the list, then add these lines about 2/3 down before its _osCd($vcsWorkingDir) line:

if $srcDir then
   _osCopy($srcDir sep '.', $vcsWorkingDir sep 'src') ' || ' _osPause() newline
endif

That will copy the contents of the $srcDir to our Git directory when we check in with Save Version. Likewise, if we ever check out we want to copy the source out of Git, so save that, edit _vcsRestoreDB and add before the last line (pushd):

if $srcDir then
   _osMkdir($srcDir) ' || ' _osPause() newline
   _osCopy($vcsWorkingDir sep 'src' sep '.', $srcDir) ' || ' _osPause() newline
endif

As you can see, we create $srcDir to be on the safe side. Those _osMkdir and _osCopy calls are simply subgenerators that produce the right batch / shell commands on the current platform, so we can write these versioning scripts once rather than duplicate the logic for each platform. ' || ' _osPause() will pause if the command gives an error, so we can see what the problem was.

Finally, if we're using the Eclipse plugin then the source code won't be stored in the normal MetaEdit+ location that we set in $srcDir, so we can override that on this machine by adding a setting for srcDir in the ~/metaedit/.vcsPaths file we made earlier. Enter the absolute literal path from the Eclipse Properties of the src directory, something like:

srcDir=/home/myuser/eclipse-workspace/WatchModels/src

Versioning changes

Go back to the Changes & Versions tool and press Refresh (or open it again from the main MetaEdit+ window's toolbar or Repository menu). That will show you the changes since the previous version, i.e. changing the Stopwatch zeroing to use the reset object. You can see the changes in the tree, or from the pop-up menu of Stopwatch: Open shows them graphically, and Compare shows them textually.

Select the top Working Version, and enable Show All Versions to see previous versions. Enter a new version number and version comment, e.g. 9 and 'Added reset function and srcDir'. Try a Save Version, both for testing and to commit the changes we made to the model and the version control generators (if you type your password wrong or something similarly simple, you can rerun from your ~/metaedit directory with metarun gitCheckIn.bat).

You can see from GitHub/BitBucket that as well as changes in the repository (versionedDB/) and the addition of the source code files (src/), we also save the generators as text files (metamodel/Graph/) to make it easy to see what's changed.

Add some new functionality to the model itself, e.g. a Laptime state. Autobuild again and see that the hand-coded line in reset() is maintained. Save Version again and see that all is as expected in the remote Git.

Summary

So, we took an existing modeling language and its generators, and added a couple of nice things:

  1. Set it up to version locally and to GitHub/BitBucket, with just two settings values
  2. Extended the Git integration to also version the manually edited code. Edit freely in MetaEdit+ and Eclipse, then version everything together with Save Version.

It's interesting to note that none of the things we changed here had been specifically set up with these use cases in mind. By building tools with the parts that might need changing accessible, understandable and open source, even quite extensive changes can be made easily. That was true on all three levels:

  • model: adding 'reset' and its manual code gives easy freedom to the modeler
  • metamodel: extending the modeling language and its generators for hand-coded Functions: see previous blog
  • tooling integration: extending versioning to generated and hand-edited source code files.

Oh, and while MetaEdit+ is as Finnish as Linux or Git, we're pretty cosmopolitan too: the versioning integration above works for Windows, SVN etc. as well.