[Exophase]

I've been waiting to make a post about this... it's late so this will probably be pretty brief and I'll clean it up later.

It has been my intention for the last month or so to move Robotic - the way the language is stored, the way we edit it, and the syntax/grammar/functional properties of the language itself - in a certain direction. I've broken up plans into several rough phases that I'll outline here.

Phase 1: Bytecode decoupling, general restructuring   Status - Internal testing
  - Make Robotic code stored as textual source inside .mzx files, instead of assembled bytecode.
	- Code is never transformed, it always looks like how you wrote it.
	- Compiled when the robot is ran, when the board is loaded.
	- Save files save bytecode still, as do "save MZMs" - neither can be loaded in the editor.
	  - Necessitated new MZM3 format which is versioned and contains textual robots.
	  - MZM3s saved at runtime and loaded at "static" time have robots dummied out to customblocks.
	- save_world, save_bc, and save_robot are removed for the above reasons.
  - New parser and Robotic compiler:
	- Variables and other names (labels, robots, boards, etc) use backticks instead of quotes (double for command level, single
		  for expressions)
	- Parser recognizes valid/invalid expressions and will visually show you when one is invalid and won't allow it to compile.
	- Expressions don't have enclosing quotes, don't require quotes around simple names
	- Commands can be split over lines arbitrarily, including inbetween expressions.
	- Optional words (like the "to" in "set x to y") now only work where they make sense.
	- The matching is loose and will fit user-defined names where the command doesn't call for something else. This allows
		   more flexibility using names that are used for other things in Robotic.
	- There's a distinction between names and strings.
  - Modified expression and name evaluation runtime:
	- &, |, and ^ instead of a, o, and x respectively, allowing the latter to be normal characters in names.
	- ** instead of ^ (for exponentiation) to accommodate the above.
	- Interpolated strings are now done with < > instead of & & (the latter can't be used for counters anymore, use expressions).
	  - Supports multiple strings in concatenation, ie < $str1 " and " $str2 >
	  - &+...& and &#...& converted to special form (+...) and (#...) respectively, can only be used in strings/names, not
			standalone expressions.
  - New color coding engine, which supports more syntax (expressions, distinction between counter names, labels, strings, etc)
  - Converter which converts current MZX robots to this format and stores as source.
  - Currently, although it breaks convention, for compatibility purposes string literals used with user-defined strings look like names.
  - load_robot won't work as it's supposed to with legacy files (will be fixed later too)

Phase 2: Program/robot separation   Status - planning
  - The notion of "robot" and "program" will be separate. There will be lists of programs, and each robot will choose which program
	 it has.
	 - The benefits of this are threefold: less space will be needed for saving MZX worlds (and eventually, SAV files) and in memory
	   for duplicates, less time will be needed for compiling programs that have already been compiled, and most importantly, it will
	   make it much easier to reuse a program across many robots.
	 - Eventually this will make it possible to "import" and/or "inherit" robots to allow for code reuse.
  - The editor GUI will make it easy to create a new program along with a new robot, or choose an existing one.
  - Current MZX worlds will have programs checksummed and entered into lists such that duplicates are removed and unique names
	are generated when collisions arise from using the robot's name.

Phase 3: Language cleanup, more syntax provisions   Status - planning
  - Expressions will be parsed into compound elements with color coding per expression token instead of the whole token. This will
	make it easier to compile them more efficiently as well.
  - Code blocks will be recognized as another compound parse element, and will be useful later.
  - Builtin counters will be converted into builtin functions that look like this:
	function(arg1, arg2), ie:
	set "vch&x&,&y&" 1 -> set vlayer_char(x, y) 1
	Note that this is unambiguous from a normal counter name with an expression, because that must be in backticks.
	- It will be possible to "set" a builtin-function, which is like passing the set value as the last argument.
	- There will be a compatibility break for any games that used strings to indirectly refer to builtin counters. This will probably
	  be rare and will be fairly easy to fix.
	- Some builtin counters will be changed to new commands instead.
	- String conventions will be changed to builtin-functions as well.
	- See this in-progress list for details: http://exophase.devzero.co.uk/counter_convert.txt
  - Several "special" versions of current commands will be changed to new commands.
	- Like with counters, if these were determined at runtime there will be compatibility problems. Again, should be rare.
	- See this in-progress list for details: http://exophase.devzero.co.uk/command_convert.txt

Phase 4: Macros   Status - planning
  - Will allow for essentially "custom commands" which take parameters and expand to one or more commands/macros that use
	those parameters.
  - More on syntax to come - I have a good idea in mind but I don't want to go indepth yet.
  - Not quite determined where/how they will be stored - macros defined in programs will probably be local to those programs,
	and macros defined in some global context (global programs, or dedicated global macro lists) will be usable by all.
  - Because macro parameters will fit combinations of valid command parameters and because the macro blocks will be parsed
	like command blocks under special context, all invocations of macros will be guaranteed to generate correct code.
	- However, any editors of global macros will not allow you to change the parameters of a macro that existing programs
	  use. This is to keep with MZX's model of never letting you exit an editor while any code may be invalid.
  - You will be allowed to overload macro names, just like commands are now. This will also be a good way to get around
	the above problem when necessary.

Phase 5: New bytecode and VM   Status - planning
  - Robotic bytecode will be stored in a more compact and efficient manner that makes it easier to quickly execute programs
	and takes out some of the bulk.
	- The bytecode format will be designed such that often used instructions are executed most quickly, allowing for
	   specialized/non-portable implementations of faster VMs (even compilers to native code) without having to carry with
	   it all the legacy compatibility that Robotic entails.
  - Commands in general will typically consist of "pushing" arguments onto an argument stack using whatever push command
	 is appropriate for the argument type, then running a command that's an argument-less instruction.
	- This will allow for a variety of argument types and optimizations.
  - Expressions will be compiled, such that they also use the argument stack and arithmetic/logical commands to evaluate
	sub-expressions.
	- This will allow for optimization to be performed on expressions, not burdening the programmer with this.
  - Now "simple" names that don't contain expressions will have pointers linked directly to the whereever the entity is,
	allowing for much faster counter lookup, gotos, etc for most cases.
	- Interpolation (for both names and strings) will be broken down and compiled in the same sense expressions are.
  - The core sections of the VM will be largely rewritten with performance in mind.
	
Phase 6: User-defined functions   Status - planning
  - Syntactic definitions similar to macros, but will allow for code to be easily reused without expanding new versions of
	that code. Like subroutines now, but will allow parameters and returns. I'll cover this more indepth later.

This doesn't really cover everything that I want, but it's a good idea of what I'm most intent on adding.

I'll keep things updated here as it moves along. I hope that at each stage things will basically be good enough that if
I stopped right there it'd still be a worthwhile, usable result. So none of the stages should outright rely on future
stages, although currently I'd have to add a couple things to phase 1 to comply with this.

(one of the admins can feel free to tidy up the line breaks and formatting on this ;P)

[hayashi]

Just had a very brief look over it and it looks pretty good (I just hope that it isn't too painful a leap from normal robotic to this new robotic, both in terms of understanding and (machine?) translation). The separation of robot and program thing reminds me of my GCS attempt, though. Which I should be working on >.>

[ajs]

Exo's converter code is robust and comprehensive. In many cases the converted code looks more readable, and in the corner cases it's generally no worse than the original. Anybody who's worried about the changes can rest assured that any robots they write in 2.82b or earlier should convert 100% correctly. So if you're not sure about the syntax changes you can just write something in 2.82b and then load the world up in the new version and it'll be converted automatically. Hopefully, of course, the more fundamental changes will encourage you to rework robots in places to take advantage of them. Also, I'm sure Exo and Terryn will document the changes thoroughly when they're finalised.

--ajs.

[Risu2112]

Among other things, I'm very excited for the new color coding. Sea of yellow has been getting very frustrating.

[Aussie Evil]

Phase 2 sounds really good. I am correct in assuming if you update the program, all robots using said program will use the updated one?

[Lachesis]

Backticks? As in, "I've gotta reach akwardly up to the top-left of my keyboard" backticks? Otherwise, I'm impressed, and in anticipation.

[Wervyn]

In general practice, you'll rarely have to type a backtick. It's only necessary when you have to clarify a name that would not normally be interpreted as such. So, any name containing only alphanumeric characters or underscore, and optionally beginning with $ and #, can be left unquoted as long as it isn't a reserved word (i.e. a command keyword or command fragment). Ultimately I think you'll find this a better system than before, at least as far as clarity of intent with respect to names. You would not believe the difference context-sensitive color-coding makes to the code appearance.

"Is keeping it simple the same thing as keeping it believable?"

[Lachesis]

Oh, alright. This sounds incredible. Any hint as to when this stuff's going to be released, or is it too far to tell at the moment?

[Wervyn]

It's probably too soon to give any concrete dates yet, though I'll let Exo contradict me if he wants to since he has a clearer picture. But work is being done at a fairly consistent pace, and phase 1 is nearly complete but for a few compatibility issues with complex games that we're still trying to test out. Taoyarin still crashes pretty quickly, for example, but a LOT of other stuff works just fine now.

"But you're staying where you are, there's nothing you can do if you're too scared to try."

[Frobozz]

Kokuchou, on Feb 2 2009, 12:00 AM, said:

Backticks? As in, "I've gotta reach akwardly up to the top-left of my keyboard" backticks? Otherwise, I'm impressed, and in anticipation.

Awkwardly? Not really. I find that hitting that key is pretty easy. Granted most of the time my left hand immediately gravitates to the WSAD keys. Plus I tend to use it when in ZSNES to skip stupid animations.

[nooodl]

Kokuchou, on Feb 2 2009, 06:00 AM, said:

Backticks? As in, "I've gotta reach akwardly up to the top-left of my keyboard" backticks? Otherwise, I'm impressed, and in anticipation.

I also COMPLETELY disagree with changing to backticks, as it's annoying for QWERTY layouts, and impossible for Italian/Belgian/Scandinavian keyboard layouts... But I'm not saying you didn't think it over, most people forget compability for other layouts. Other than this, it's looking sweet so far!

[Wervyn]

I completely agree with the rationale for changing to backticks, and think that if for some reason you've got a keyboard layout that cannot type this character, you should either do something about that yourself, or propose a better quote-like symbol that is not " or '. MZX Robotic is becoming more of a formalized language, and that means that ' (characters) and " (string literals) MEAN different things now. ` is a good option because in many programming languages it does mean to evaluate the enclosed item as a system or program level construct, but not as a keyword. The closest analogy is probably MySQL, where it actually is used to directly clarify that something is a name (e.g. a table or a column name) within the database.

That's what's being done here. Remember that under normal use, you will not need to type many (if any) backticks, because names in the context of names are usually understood as what they are without additional punctuation. That is, the replacement for set "this" to "that", is NOT set `this` to `that`, but set this to that. And thanks to the new color-coding, you will always be able to tell by looking that this is either working as intended, or not parsing correctly. If the program understands something as a variable name, it'll appear in red. If it doesn't, either the entire line or multiple lines of the command will be an invalid dark blue, or it will be some other color appropriate to what it represents. It's only at those times that you need to either change the name or supply backticks.

Now there is a minor wrinkle when it comes to counter interpolation, which is now done exclusively through () for counters and <> for strings. I'm not yet clear on whether the system will require enclosing backticks around variables using a counter name, or if it will parse it naturally. Right now, the first is necessary, and the reason it's an issue to begin with is because of planned syntax overlap with functions. But if functions match first, before counter names, it seems like this could be done. On the other hand, this format for arrays is going to become undesirable in the future, because of the addition of ACTUAL arrays using []. So it may become moot.

And in the worst case, remember that MZX still provides you with a 256 character selector when you press F3. So it'll never be "impossible" for you to type ANY particular character, just a little more clumsy.

My only real complaint with the language specs I've read so far is the replacement of currently natural tasks for built-in counters like HEALTH (e.g. set "HEALTH" to 50, set "mycoins" to "COINS") with built-in functions like get_health() and set_health(value) (e.g. do set_health(50), set mycoins to get_coins()). This threatens to really mess with people, and I think the only reason no one has complained about it besides me yet is not because they don't care (as Exo thinks), but because they don't realize that this is the planned behavior. Still, I could be wrong, it's not like this is impossible to adapt to. The advantage is that it clears up the counter namespace of all those built-ins in favor of functions with names that are less likely to be duplicated. The disadvantage is that it's a major break from most MZXers' coding styles, and requires more typing in most cases.

"The cow says 'moo'."

[ajs]

FWIW we can probably just remap another key to ` in the case that a foreign layout lacks it. Please let me know the details of exactly what layouts don't have the key, and which key would be a good substitute (ideally a key that MZX currently ignores) and we'll come to some kind of solution. Also, I agree with Wervyn over the ` rationale. MZX isn't alone here, MySQL and Bash (and other shells) use this method of escaping.

By the way, though I think Exo plans to remove editor macros as they stand, that would be another way to work around this problem (F-key binds to ` for example).

--ajs.

[Exophase]

Wervyn; I already told you, it'll be like this:

set score() 10

It won't try to distinguish between interpolated names and functions. The former has to have ticks around them.

[nooodl]

Wervyn, on Feb 6 2009, 03:08 AM, said:

And in the worst case, remember that MZX still provides you with a 256 character selector when you press F3. So it'll never be "impossible" for you to type ANY particular character, just a little more clumsy.

That's pretty much what I'm doing right now for < and >, and that's working pretty OK as well. Anyways, you're right. I won't need to type too much backticks in the same way you don't really need to type your strings with "" around them.

ajs: Well, in the current version of MZX - as I said earlier this post - AZERTY's key for <, \, > isn't working correctly.

It's the one left of the W key. I don't really know why this is happening, but feel free to ask any questions you need answers to so you can fix this.

[ajs]

Kinda OT for this thread buuut... http://mzx.devzero.c...mzx282b-x86.zip

This is a build of 2.82b with http://mzx.devzero.c...ey-mapping.diff

Run it, try using the keys that don't work, then pastebin stderr.txt (should contain a bunch of DEBUG: messages). Give me an idea when these dead keys were pressed.

EDIT: BTW could you try pressing the grave combo a few times (maybe AltGr-7?) and see if you get a unique unicode value? If so we could use this as a substitute for backticks. Helps some layouts, anyway. BTW because I wasn't sure fprintf() would be locale friendly, the final column only prints the printable subset of ASCII-7; even if ? is displayed it does NOT mean we can't use that as a key.

--ajs.

[Exophase]

Finally got back to this and fixed up some lingering stuff, I think I'm going to move on to putting together a task list for phase 2.

If anyone is eager to test a crapload of games let me know. You have to follow good testing protocol though.

[Old-Sckool]

Ooh, ooh, PICK ME! PICK ME!

I've been dying to beta test this stuff since this thread started.

[Exophase]

Old-Sckool, on Mar 30 2009, 07:01 PM, said:

Ooh, ooh, PICK ME! PICK ME!

I've been dying to beta test this stuff since this thread started.

Okay, your first job is to find me on IRC.

[Koji]

"I will grant you 3 wishes, but you must CATCH ME FIRST "

[Frobozz]

Just found a rather amusing comment in the debytecode branch.

Quote

Things are back on track... maybe. Want to get "phase 1" wrapped up for real.

Note this is from nine months ago. Also what exactly does "internal testing" mean regarding phase 1 if anyone remembers anything related to this?

[Exophase]

This amuses you, huh? For what it's worth, phase 1 WAS more or less wrapped up, and it's there on SVN. I have a lot of reasons for not working on it since then, and most of them are pretty good. But I've outlined things pretty extensively so someone else is free to continue working on it.

[Terryn]

It means I and a few other people Exo asked were testing the hell out of it at the time.

[Frobozz]

So is it stable enough to try moving to the trunk?

[Terryn]

There were still some pretty major things wrong with the branch when Exo dropped it (like internal saving/loading being completely broken). I guess it depends on how ajs feels about tackling it, considering some major changes to MZX's codebase since Exo's branching.

[Exophase]

Stable or otherwise it definitely has no business being in the trunk, it's an incomplete part of a larger whole and once you go down this path there's no going back. My intention was for it to exist alongside "normal" MZX for a while even after each of the phases were complete. What phase 1 was almost ready for is public release and testing, under the VERY clear understanding that you shouldn't use it for serious new or existing projects, just for testing things. In fact, I thought about not even letting it save worlds, but that'd be too restrictive.

Internal saving/loading isn't completely broken, if that were the case it'd be a lot easier to fix. Rather, it occasionally crashes in ways that are impossible to reproduce. I asked those involved to get some test cases narrowing things down better, they didn't, so we hit a standstill. I simply didn't have the time to try to resolve it myself. Now I'm even busier with things that are much more important, and this has all faded into irrelevance for me anyway.

[ajs]

So how about we start off with me re-basing debytecode on 2.83, doing nightly builds of it, releasing some binaries to folk, and see if we can get some feedback on the changes and maybe fix some stability issues? Do you have any objection to this? Apart from making stuff incompatible, are there any other must-have changes you feel should be made before we put this into wider testing?

BTW I appreciate the time you're taking to comment on these issues.

--ajs.

[Exophase]

I don't believe I had any further issues. Last I remember I was ready to start coding on phase 2. For protection I think we should make MZX incapable of overwriting mainline worlds with debytecode worlds, that should be sufficient for keeping people from doing something completely awful.