The URL Problem (Wheel Reinvention Jam 2024)

For the Handmade Network Wheel Reinvention Jam this year I am looking to investigate alternate methods for URLs, links, and using a browser to navigate between files on the internet. This blog post serves as a short introduction to my thoughts on the topic, with minimal editing or up-front research on the topic or history. As I go through the jam I’d like to do a healthy balance of research, experiment building, and blog writing to present my thoughts so this is more of an exploration rather than an authoritative piece on the subject.

Link to the Jam: https://handmade.network/jam/wheel-reinvention-2024

Link to my Project: https://handmade.network/p/608/orca-links-modern-urls/

The URL Problem

Here's my description of the URL problem, as I understand it (i.e. I haven't done any research into the topic, this is the story I tell myself in my head. I'm sure the reality and history is a lot more complex)

Let's imagine we are at the beginning of the internet. There are like 10 computers connected to each other. If I am using one computer, and I want a file on another computer, I need to somehow distinguish which of the other 9 computers I want to access to get the file. There are essentially two problems that need to get resolved:

• I need to know which computer the file is on. Maybe I just used the computer recently, so I want to see and remember the name of the computer I was accessing so I can tell my computer at home later which computer I am talking about. Much more commonly (in the modern world at least) I was told the file existed by some external source. Like maybe a friend told me that there's a cool game on his computer that I should try out. In this case my friend needs to somehow communicate to me the information about his computer so I can access it later

• Once I know which computer I am talking about, I need to somehow put some information into my computer and run it through some system that will get me connected to and downloading the right file. Essentially, using the information I have in my head, combined with the information and systems on my computer, I need to reliably (and hopefully without too much effort on my part) get connected to the correct computer.

One way to solve this problem is to give every computer in the world a unique identifier. In this hypothetical past, when there were only 10 computers, we could just label each computer with a number 1, 2, 3, etc. and then we could use that number to uniquely talk about each computer. Maybe my friend owns computer 3, he tells me his computer is 3, and on my computer which is 4, I say I want to connect to computer 3. So my computer looks at every computer connected to it and finds the one that reports itself as computer 3. This is pretty simple, and actually would work great if it weren't for a few fundamental problems:

1. There are more than 10 computers in a world. Way too many for a human brain to remember each unique number. Too many for my computer to ask every other connected computer what number it is as it tries to find a particular computer.

2. Humans are not as good at remembering "random" series of numbers as they are at remembering names and words in their own language. I can remember 10 random English words easier than I can 10 random numbers between 0-255, and that's not a fair comparison, because there are more English words than 255. Also some names are more valuable than others, because they are easier to remember, or represent more popular concepts. For example more people may want their computer to be named #1 rather than #2425, because #1 is easier to remember and people have a preference for being #1 in a competitive sense.

3. People all speak different languages, and it's easier to remember words and names in the language(s) you know. This is especially complex if my friend and I speak different languages. His computer might have a name in his language that is hard for me to remember. We need some system where we can use a translation of his name into a name that I can understand and remember.

4. Say my friend buys a new computer between telling me about the game and me trying to download it. This new computer might have a different number than his old one, but it's still his computer, and he likely means for me to get the game from his new computer rather than his old one. It would be nice if he could tell the world "I got a new computer. Please use computer 9 now, not computer 3" without having to communicate with me specifically.

5. Files and information transfer hold value, so someone might want to pretend to be my friend's computer. How can I be sure I got my friends file, not someone else who tried to pretend to be my friend? I somehow need some guarantees that the information my friend told me is enough to confidently route me to his computer, not someone else's.

6. The file on my friends computer might be in some folder. Once I'm connected to his computer, how do I find the file I am looking for? This is sort of a sub-problem, since my friend could include the information about which folder it's in as he's telling me the ID of his computer, but we'll likely want to treat this as a sub-problem, since it's likely easier to solve the problem of finding the right file after we solve the problem of connecting to the correct computer.

7. There might be a lot of people that want this file. Maybe my "friend" is actually Taylor Swift, and the "game" is actually a new album she just finished. Millions of people want to download this album but a single computer is not powerful enough to serve all of those people at the exact same time. For a variety of hardware reasons, we can't have a single physical computer that everyone is downloading from. We need to set up a system where the load of transferring the file is distributed amongst a collection of computers that all have the file. Even if everyone is finding the file using the same information that Taylor told them.

8. Related to #7 Each individual computer may break, or it may go offline for one reason or another. We can't have one computer failure cause the entire system to be inaccessible. We need to have a dynamic system where people get routed to a different computer when the one they were trying to connect to is unavailable

9. People live all over the world, and downloading a file from France while I live in Canada is not always that fast. It would be nice if my friend could manage a computer that is not physically located at his house, so that people that he can share files quickly with people who do not live close to him.

So we have a bunch of hurdles to overcome. Our system of finding and connecting to computers needs to solve all these problems (or at least attempt to solve as many as possible). The system we currently have solves many of these problems, but it arguably doesn't solve all of them. Part of the reason it doesn't solve them is that it was designed in a time where the scale of the internet we have to today was unimaginable. Some fundamental decisions were made that are probably not the best choice, given what we know now.

So let's start thinking about a new solution to this problem. We'll start with iterating over some basic properties we want our solution to have:

1. We want our naming scheme we use for each computer to be something that is easy for both humans and computers to work with. This likely means text, since basically every computer people use has some kind of keyboard that allows them to quickly put in characters in their native language.

2. Given that our naming scheme is textual, we want it to be as versatile as possible. It should work for as many languages as possible, while not sacrificing too much in the way of complexity. So for now lets say we support a larger portion of the unicode characters, but we are still going to choose some small-ish subset. Remember our goal is to balance versatility with other concerns, so allowing every single unicode character is likely going to cause us some grief in terms of complexity later on (for example, imagine if we support all invisible characters in unicode, many completely unique links would look the same to a user)

3. As a single user of the internet, with a computer, I want to be able to choose a name for my computer. Maybe I don't always get the exact name I want, but I should be able to have a decent amount of control and influence on how my computer gets named. As a large business with a well-known name, I want to be able to choose the name for my host of computers that serve files to our customers. I still might not always get the exact name I want, but I will spend a lot of money and/or effort trying to get a particular name that matches my business name. And I also want to make sure nobody else can try and pretend to be my business and distribute files under my name. Both of these customers need to be served by the system, so we need to balance the concerns of both. The current system we have relies largely on a single source of truth that people will pay money to, and in return will get a name uniquely assigned to them. If two people want the same name, they may negotiate selling the name between themselves. Ultimately, whoever wants that name and has the most money will usually win. In our new system, we'd like these single name multiple customer disputes to get resolved a little more gracefully, and preferably with less of a reliance on deep pockets or a single source of truth. For now, let's say we at least want the decision of who gets what name to be a little more collaborative between all the parties involved, and we'd like the result of these disputes to roughly match who wants the name more (especially since that might change over time)

4. As a user of a particular name, I want to be sure that a computer I am connecting to for the first time is the one I expect to be finding. This may mean a longer process of connection, like typing in a longer name and then verifying that the name I typed was correct using some sort of feedback and security mechanisms. But once I am sure I connected to the correct computer, I would like to start using a shorter process to reopen that connection. Maybe a shorter name, maybe a name in my own language, maybe even a name that I gave it myself, that has nothing to do with it's globally registered name. I also want to be sure I don't accidentally visit another new computer when I meant to revisit one I had found previously. Ultimately, there needs to be a collaboration between the user and the browser software they are using to establish a naming pattern that they both understand, which may be separate from the naming pattern used for the actual connection behind the scenes. But this collaboration can't be so complicated that it confuses new users, or prevents communication between users about a particular computer (website) they both visit. So let's assume that our naming scheme is going to be paired with a standard way saving information about names on the clients machine, and building up trust over time for particular computers that the user connects to often.

5. While the single authority aspect is a downside to the current system, I don't think getting entirely away from it is reasonable. We could do something like the block chain where the "authority" is just consensus between machines, but that has it's own problems. For now, let's assume we have a central authority of some kind, but we'd like to make the burden on that authority as light as possible. They shouldn't be required to settle disputes directly between people who want the same name. They should be relied upon to host a secure database and reliable access to all, with a system that allows disputes to get reconciled internally, and over time.

6. The service that hosts the naming information should also support more information than just the textual name and the IP address(es) of the computer(s) it's associated with. It should be able to store and serve information about the owner, date of registration, image icons that represent the site descriptions, and maybe even analytics about how many people have visited that site. All of this information should be editable by the owner of a name, and easily requestable by the users looking up the IP address associate with a name. This doesn't serve as the backbone of security, so verifying the information isn't the highest priority, but it does serve as the user friendly way to investigate the site and determine if it's the site you are looking for, so the system should be set up in a way that the information is commonly used and expected by users, so the owners of the names are encouraged to provide useful and accurate info when in good faith. Bad actors and spoofing should be prevented/handled by other more technical means of security

With all these things in mind, I feel like I'm starting to see a certain shape form of the kind of system that might work. Again all of this is a thought experiment, before I actually do research and implementation attempts, so much of my understanding may change and these proposed aspects may turn out to be entirely wrong. But hopefully this helps set the stage for trying out a potential solution. I'd be happy to hear any feedback about these ideas, I'm sure I'm not nearly the most knowledgeable person on this topic, so I'm probably wrong on a number of accounts here.

My Experience With Jai (Part 3)

Alright, with an actual file converted into Jai, I think we have a good idea of what these conversions are going to look like, so I’ll try to get through a few more in a single post this time.

gy_rectangles.h

gy_rectangles.h is one of the largest files I have in Gylib because it ended up doing more than just “rectangles.” In this context, rectangles are anything in 2D or 3D that has perpendicular flat faces. So that includes AABBs in both 2D (rec) and 3D (box) as well as OBB in both 2D (obb2) and 3D (obb3). I also have integer variants for rec and box called reci and boxi. So that gives us 6 distinct types that are somewhat similar but are actually, in reality, quite different than each other. What I mean by this, is a lot of functions that work just fine on axis-aligned rectangles in 2D do not make sense on an axis-aligned 3D box, or even a rotated 2D rectangle. So although some functions have 6 variants, many of them only have 2-3 variants. Because of this, the file will probably get split someday. But for now, we’ll move on.

As before, HERE is the (mostly) completed gy_rectangles.jai and I’ll point out some of the interesting things that happened with this file

Positive 1: #place and Operator overloads worked wonderfully

Rectangles and boxes are not normally pointed to as “standard” mathematical types for most CS or Linear Algebra discussions. However, I’ve found that they serve as fundamental a role in game development as vectors which means that proper member aliases and some simple operator overloads go a long way. For example, being able to say rectangle.topLeft to get a v2 that represents the x and y members of the rectangle is super useful. And I can still do rectangle.x instead of rectangle.topLeft.x. Also operator overloads being fully available in user space means that I can do things like rec + v2 to indicate I want to “offset” my rectangle by some amount.

Problem 1: No Way to Represent Derived Values (afaik)

This is not something I had in C/C++ but it is something I would find useful and is one of the things I miss from my C# days. Being able to treat simple derived values as if they are actual members of the struct. For example, being able to say rectangle.center as a shorthand for rectangle.topLeft + rectangle.size/2 would be mighty convenient. C# has the option to make “Properties” that have a getter and a setter that define what should happen if the derived member is read or written to. This has a host of downsides, since it can be easily abused to execute an arbitrary amount of code inside what seems to be a harmless read at the calling side. However, if there was some way that I could get a very restricted form of this functionality, I would be a lot happier for it.

gy_directions.h

This file serves as a standard way for me to talk about cardinal (or diagonal) directions in an enumeration like way. Because of the implicit assumptions about handedness and direction order, I feel like this is one of those things that would never make it into a standard library in a satisfactory way. I don’t want to have to specify if my coordinate system is right-handed or which axis is “up” all over the place. I want those sorts of assumptions to be baked into my library of tools and therefore uniform across the code-base. So that’s what this file does, it defines 4 enums: Dir2, Dir2Ex, Dir3 and Dir3Ex. The Dir2’s are for 2D directions (up, down, left, right) and Dir2’s are for 3D directions (up, down, left, right, forward, backward). The “Ex” variants are enumerations that allow for diagonal directions as well as cardinal ones. To be honest, the Dir3Ex enumeration rarely sees use but the Dir2Ex enumeration was widely used in Princess Castle Quest since we had some things that moved diagonally. All of these enumerations are designed to be bitwise or’d together so they can be used like flags.

You can download the completed file HERE.

Positive 1: enum_flags helps implicitly assign bitwise or-able enumeration values

In the old code we had to make sure each value of our enumeration got assigned a proper power of two since we intended each one to be a single bit in a larger field of flags. Jai has a feature called “enum_flags” which allows us to do the following:

Dir2 :: enum_flags u8
{
	Right; //+x ( 1,  0)
	Down;  //+y ( 0,  1)
	Left;  //-x (-1,  0)
	Up;    //-y ( 0, -1)
}

Positive 2: Enumerations can have a forced backing type

Given that enumerations like this can serve as the backbone for many different systems, having the option to specify the size (and signed-ness) of the integer that backs them is pretty useful. It makes serialization a bit less hairy because we get a guarantee that the enumeration won’t accidentally grow in size and push members of a struct following it down. However, the biggest win by far with this feature is the guarantee that the binary NOT operator will behave like we want. Doing something like ~Dir2.Right is guaranteed to give us 0xFE (not 0xFFFFFE or anything else). This is a big deal for bitwise or-able enumerations where we often want to perform the following operations

IsFlagSet(field, bit) -> ((field & bit) != 0)
FlagSet(fieldPntr, bit) -> *fieldPntr |= bit;
FlagUnset(fieldPntr, bit) -> *fieldPntr &= ~bit;
FlagSetTo(fieldPntr, bit, value) *fieldPntr = (*fieldPntr & ~bit) | (value ? bit : 0);

As a side note, I implemented these Flag macros in Jai using the #expand functions thinking they would be important for readability, but it turns out that the type safety and other guarantees the language gives us are enough to make doing the bitwise operations out in plain code feel pretty normal. So the following code feels pretty mundane:

if (directions & .Right)
{
	directions &= ~.Left;
	directions |= .Up;
}

Problem 1: non-pure enums are more obvious because of Type Metadata

This is more of my own problem but these enumerations in C would often include derived or “meta” values that were related to the enumeration but not actually a proper enumeration value. For example Dir2_Count is not supposed to be used as a “valid” value for the enumeration. In fact it directly “conflicts” with Dir2_Left since they both have a value of 4. And now that we have functions like enum_values_as_s64 and enum_names it seemed weird to include them as part of the actual enumeration. Serendipitously we can just pop them out of the enumeration and declare them as constants that have the enumeration type. Or for the _Count case we define them with a regular integer type

Dir2 :: enum_flags u8
{
	Right; //+x ( 1,  0)
	Down;  //+y ( 0,  1)
	Left;  //-x (-1,  0)
	Up;    //-y ( 0, -1)
}
Dir2_None:     Dir2 : 0x00;
Dir2_All:      Dir2 : (Dir2.Right|.Down|.Left|.Up);
Dir2_Count:    u8   : 4;

This makes it possible to do things like:

directionFlags := Dir2_None;
for 0..Dir2_Count-1
{
	side := cast(Dir2)(1 << it);
	if (somethingOnSide(side)) { directionFlags |= side; }
}

Problem 2: My GetDir#Str functions didn’t go away

I imagine this won’t be true for many of my enums but in this case, I opted to keep my hand-typed stringification functions rather than rely on the enum_names function that’s built into the language. This is largely due to the special case handling we do for stuff like .Left|.Right being “Horizontal” or handing Dir2_All like it’s a proper member of the enum. However, there’s also a problem of ergonomics of these enum functions. They are list providers but they don’t help me go from enum value to name in a direct way. If I wanted to use the enum functions to get a name from a enum value I’d have to do something like:

GetDir2Str :: (direction: Dir2) -> string
{
	names := enum_names(Dir2);
	for enum_values_as_s64(Dir2)
	{
		if (it == cast(s64)direction) { return names[it_index]; }
	}
	return UNKNOWN_ENUM_STR;
}

This is not terrible. It’s only a few lines. But a switch case statement feels a little more in-line with how I’d like this function to operate, since the compiler could decide to turn it into a jump table. The real solution here would be if I specifically could ask the compiler to generate a jump table for me. This isn’t important here, but for larger enumerations (like EntityType) this could be a very useful feature. I imagine this might already be possible, given some metaprogramming. I will have to revisit this when I get more familiar with the compile-time code generation features of the language.

Conclusion

Those are all the files I’ve fully converted so far. I will write some more blog posts as I get farther. As you may have noticed, gy_intrinsics.jai is somewhat finished (things like CosF32 are already being used in these files, but I have yet to alias all the “intrinsic” like functions that I use). That and gy_colors.h will probably be next

My Experience With Jai (Part 2)

So if we take a look at our dependency graph in the last post there’s a bunch of things we “should” implement next if we wanted to continue in a top-down order. Things like gy_assert.h, gy_debug.h, gy_intrinsics.h, etc. However, a lot of this stuff can be sorta stubbed out and side-stepped for the time being. And I think that’s for our benefit since it will allow us to get to the meat of the features we want to try converting to Jai really quickly. So next up on the chopping block:

gy_vectors.h

This file contains, as you might expect, vector structs like Vector2, Vector3, Vector4. Generally, since we were working with C in the past, this file tends to be a massive file of manually duplicated code to handle each vector type for each type of operation we’d like to perform with vectors. Moving into Jai, I had two main questions I wanted to answer:

1. Do Vectors already exist in the “standard library” of Jai and if so, do they do all the things we want them to do? If they exist, but don’t do everything we want, can we easily expand the functionality to do everything we want? Or do we need to wholesale replace those implementations with our own?

2. Can and should we try and do polymorphic programming for this kind of problem? At the end of the day, the number of vector variants we actually want is around 6. We want 2D, 3D, and 4D and then we want floating point and integer variants for all 3 of those. Is it worth doing all our programming in a polymorphic way to keep from having 6 of everything?

So we can tackle these questions one at a time. First, do Vector structs already exist. The answer is YES (sorta). In the Math module we have definitions for Vector2, Vector3 and Vector4 (as well as Quaternion and various Matrix sizes). However, on further inspection there is only a limited number of math operations defined for these structures so we at least need to add more functions for these vector implementations to be “complete” in our case. For example, there are dot_product and length functions for all the vector types but there are no functions to round the components of the vector easily (something I do often for UI code and text pixel alignment) or clamp the components between two sets of upper and lower values. Also missing are the integer based vector variants that we want (I call these Vector2i, Vector3i, and Vector4i). So at the very least we’ll need to completely implement those variants and add some functions that operate on the existing structs in common (to us) ways.

Second question: Should we write our structs and functions in a polymorphic way? I’ll admit that I didn’t fully investigate what this would look like before deciding against it. So it may be that my worries are not actually founded. However, the main things that dissuaded me are:

1. We have a very small set of cases that we actually want. And having properly named functions and types for each variant holds some value when we are debugging and looking at the call stack. Seeing that something is a “Vector2i” instead of a Vector{2, s32} might seem like a small distinction, but it makes a big difference when you are dealing with these types all day, every day.

2. We don’t actually need anything above 4 dimensions, nor below 2 dimensions. A lot of functions would degenerate in the 1D case and make implementation a bit noisy (i.e. checking for this edge case and erroring, or early returning). Also a lot of functions that could be generalized to an N-dimensional case are actually only ever used for 3D cases. For example, cross product can be generalized to more dimensions besides 3, but we literally would never use the other variants in our games.

3. Naming our functions specifically to 2D and 3D is often useful. For example Vec2PerpRight is a very common function I use that makes no sense in 3D or 4D. You could make somewhat similar functions that take a couple more parameters than the 2D case but we would basically never use these variants because they don’t make sense to be called “PerpRight.” They would be called something like “RotateAroundAxis” for 3D (and who knows what for 4D)

So we don’t want to do things entirely polymorphic but we do we want to try implementing some “generalizable” operations in a polymorphic way? When I first started writing the file I figured I would say yes to this question. After all, having one VecAdd function seems quite a bit better than 6 VecAdd variants. However, given the brevity of some of the syntax, I found that almost ALL of my functions could be collapsed to a single line and not lose much readability. So I opted to do everything in a bespoke way (mostly for the sake of debugging readability later).

So I went ahead and converted my gy_vector.h to gy_vector.jai. For the sake of brevity, I’m going to upload the result of that process HERE and then point to some of the interesting aspects instead of explaining the process of conversion from beginning to end.

Positive 1: Constant Value Definitions are awesome looking (Curly Bracket Initialization)

One of the things I like having for vectors are “Simple Value Definitions” (see lines 107-181). Stuff like Vec2_One or Vec2_Right. Jai has support for default initialization of a struct which solves half the problem (i.e. Vec2_Zero is probably going to be referenced very rarely). In C these value definitions took the form of #defines that are really just function calls to NewVec2.

#define Vec2_Zero      NewVec2( 0.0f,  0.0f)
#define Vec2_One       NewVec2( 1.0f,  1.0f)
#define Vec2_Half      NewVec2( 0.5f,  0.5f)
#define Vec2_Left      NewVec2(-1.0f,  0.0f)
#define Vec2_Right     NewVec2( 1.0f,  0.0f)
#define Vec2_Up        NewVec2( 0.0f, -1.0f)
#define Vec2_Down      NewVec2( 0.0f,  1.0f)

This meant that our value “definitions” were not actually compile time constants like we wanted them to be. However, the alternative was to do curly bracket initialization, but this came with a HOST of downsides because C’s bracket initialization is under-defined and not very strict.

In jai we have a few more promises from the implementation of this feature. Namely the curly bracket expression is strongly typed, with stuff like “v2.” before the open bracket forcing the expression to always be a v2. And the expression errors if the number of items passed does not match the number of items in the struct exactly. With those things in mind, we’re able to switch to curly bracket initialization for these definitions and reap the benefits of having truly constant values for our vector structures!

Vec2_Zero  :: v2.{ 0, 0 };
Vec2_One   :: v2.{ 1, 1 };
Vec2_Half  :: v2.{ 0.5, 0.5 };
Vec2_Right :: v2.{ 1, 0 };
Vec2_Left  :: v2.{-1, 0 };
Vec2_Down  :: v2.{ 0, 1 };
Vec2_Up    :: v2.{ 0,-1 };

Also, curly bracket initialization allows me to make many of my basic vector functions into single line implementations that are still relatively readable. For example:

Vec2Floor :: (vector: v2) -> v2 { return v2.{ floor(vector.x), floor(vector.y) }; }

Positive 2: I’m getting rid of _t suffix on my Types

For a long time now I’ve opted to add _t to the end of all my types in C. I forget exactly what the main impetus was for this but I suspect it has a lot to do with the fact that types are not treated the same as other constructs in C/C++. So normally my vectors would be defined like so:

union Vector2_t
{
	r32 values[2];
	struct { r32 x, y; };
	struct { r32 width, height; };
};
typedef Vector2_t v2;

So we’d put a _t on the end of the type name to make it clear it was a type. However, for really common mathematical types (like vectors) I would also make an super shorthand alias like “v2” (something I never did for higher level types like Color_t or Textbox_t for example).

With types being treated with more “first-classiness” in Jai, it seems like having this suffix is entirely redundant now, so I’ve opted to try getting rid of them and see where that takes me.

Problem 1: I can’t Augment Vector2, Vector3, and Vector4

If we look at one of our integer vector implementations, there is a feature that I like having that the Math module’s implementation does not have. And that is aliasing x, y, and z components as “width”, “height” and “depth.”

Vector2i :: struct
{
	x, y: s32;
	
	#place x;
	component: [2]s32 = ---;
	#place x;
	width: s32 = ---;
	#place y;
	height: s32 = ---;
}
v2i :: Vector2i;

This can be done by using the #place directive, which is great! However, as far as I can tell there is no way to do this after the structure has been defined. So if we want to actually use the Vector structures from the Math module (and it’s associated functions and operator overloads) we have to forego these aliases for floating point vectors. This is a common case of “If it doesn’t do everything I want out of the box, I can’t use it at all.” This isn’t a deal breaker for now, but I may change my mind later if there ends up being any other missing features that I haven’t noticed yet.

Problem 2: The Unary Negative Operator was missing

This is a small thing to fix, but worth mentioning. There is no definition for negating any of the Vector structs in the Math module, so we had to implement those ourselves.

Problem 3: Dead Code Elimination and Float32 is a “long” name

Since this is the first real file of substance that we’ve started bringing into Jai, there are 2 problems that aren’t Vector specific but are interesting to mention. The first one is that many of the mistakes I made would not get caught by the compiler until the function was actually being called by some external code. At first I solved this by writing a TestVectors function and #running it during compilation. I’m not a TDD proponent so this is very abnormal for me. It ended up wasting enough time that I had to take a brief break and go figure out how to make the compiler more strict. Turns out that the compiler does “Dead Code Elimination” by default, which was definitely my suspicion. There’s actually a standard command-line option -no_dce to turn this off that I had completely skipped over when reading the debug output of “jai -help”. Turning this elimination off definitely helped, but I feel like I still hit a few cases where my function would compile until I called it from somewhere. This might be anecdotal though, I’ll try and keep an example if I come across another instance of this happening. The other “problem” is that I distinctly feel the difference between a 3 letter type name of “s32” and a 7 letter type name for “float32.” I put problem in quotes because it’s entirely aesthetic and it’s not hard for me to fix as a user. But for now I simply noted my preference and continued on with my day.

My Experience With Jai (Part 1)

I recently got access to the beta of a new programming language called “Jai.” My goal is to try converting my existing set of C files that I collectively call “Pig Engine” over to Jai in order to get a sense for how the language works and also find places where my approach to the same problems may differ in Jai (hopefully in a positive way) compared to C/C++.

Along the way I figure it might be useful to keep a running dialogue of my thoughts on various features and how I approach learning and using the language. I don’t use this blog much, so I figure almost no one will read this, but I think it’ll still be a helpful practice for me.

part 1: Reading HOW_To’s

The language beta comes with a folder called “how_to” which has around 70 files worth of examples code and explanations for how the language works. So I figure before I start trying to write anything, I should at least read some of these files. Mostly starting from the beginning (jumping forward to particular sections out of order when I wanted to know something specific about the language) I read through about 15/66 files before switching to proper work in the language.

Although it’s a lot of reading, I found myself pleasently surprised at how many features of the language line up with my experience of the hard problems I encounter while making games.

As a simple example, the 013_enums.jai file explains the syntax for enums and one of the “new” features this language provides is a tag you can put on enums called "#specified.” The whole purpose of the tag is to force the programmer to specify every enum value explicitly (no implicit value assignment) and generally act as a callout that the exact values of this enum are being used for serialization and changing/shifting the values of existing entries is probably a bad idea unless you go change this serialization code. Serializing enums might sound like a random niche problem, but I actually have run into this many times. The choice of whether to serialize the integer value or the string name is a tricky one, particularly if your game goes through an early access period with user generated content that you need backwards compatibility for, while still actively developing the game and changing enums. Having this one small feature really goes a long way to making this particular type of problem recognized as a thing that should be considered. In a lot of ways it’s not much more than a comment with a small amount of compiler backing to enforce specified values, but the fact that it exists in the language means that it can become a standard idea with a commonly understood name, and I think that’s pretty valuable.

Part 2: Converting Gylib (Dependency Order)

Alright so now that we’ve got an understanding of the basic features that the language has, I started trying to convert my common library code into Jai. For dumb reasons, this library is called “Gylib” and it includes everything that I use in basically every project; from Vector structs and math to UTF-8 conversion functions. For a somewhat full list of features you can check out the block diagram for Gylib that’s on the Pig Engine page.

So if we want to “port” Pig Engine to Jai, I figured the best place to start would be this common library. The files are largely organized by “features” but they also have had a dependency order baked into them because C does not allow out of order compilation. And since they were all built in a single header style fashion, I was forced to make sure that they had a #include order that would work. So we’re going to start from the files that are near of the top of the dependency chain and work our way downwards. However, as we go along we may find that a file near the bottom of the chain can actually get split and merged with upper level files now since they used to harbor functions that would fit, logically, in another file. But because they were dependent on a item lower in the chain, they had to be separated and given a name. Here’s an incomplete diagram that roughly shows that dependency order of many of the files:

A good example of something that might get merged is gy_sorting.h. Because the concept of sorting is not uniquely owned by gy_variable_array.h, gy_bucket_array.h and gy_linked_list.h it generally has to get split into another file and then the specific applications of sorting on each data structure type has to get jammed into a very dependent file near the bottom (for example, gy_extras.h, which got missed in the diagram for time constraints but is literally just things that couldn’t fit in upper files because they depend on two or more items in an unsavory way)

So starting from the top we have gy_defines_check.h which mostly translates directly to our module.jai since that’s where we can define our module_parameters. Fortunately some of the defines we’d normally need to enforce ourselves (like WINDOWS_BUILD) are actually nicely handled by compiler constants. So the main things we have left are two defines

DEBUG_BUILD: Enables debug only macros like DebugAssert. This is often tied to whether we want the program to be compiled in an optimized way, but it’s really just “Am I compiling this for me to test right now or to distribute or transfer for someone else to use.” So it’s often nice to control this define ourselves, even if the compiler provides some way to detect if we are compiling for “release”

ASSERT_FAILURE_FUNC: This one is a bit interesting in jai. Essentially it’s nice to have a function run when an assertion fails BEFORE we hit the breakpoint/abort. This function needs to be implemented on a per-application basis but the Gylib library provides all the common Assert() #defines and it’s variants. So we need a way to tell it what function to call when an assertion fails. In C we used to just do extern AssertFailureFunc(…) in the gy_assert.h and that would force you to implement the function elsewhere. In Jai we can actually pass in the function as a properly typed variable

Here’s our module_parameters directive in module.jai

#module_parameters (DEBUG_BUILD := false, ASSERT_FAILURE_FUNC : (message: string) -> bool = null);

Besides that our module.jai #imports a couple common modules like “Basic” and “Math” and then #loads all of our gy_[whatever].jai files.

Next up on the chopping block is gy_std.h which (I hope) is mostly going to be unnecessary. This file mostly handles aliasing standard library functions with our own names (like memcpy becomes MyMemCopy) simply so we can reroute these standard library calls to our implementations or other stdlib implementations in certain contexts (web assembly for example). For the time being, we are going to trust that we can solve this problem later, given the tools that jai provides us when loading modules.

Next we have gy_basic_macros.h which has a bunch of #define function-like macros for common tasks we want function-like syntax for. For example ArrayCount(array) which is just an alias for sizeof(array)/sizeof(array[0]). Luckily arrays in jai come with length info so this is not needed. Also macros in general are actually just functions with the #expand directive attached, so many of the “macros” in this file are going to get bumped to gy_intrinsics.h because the distinction between the two is pointless now (we no longer have a preprocessor that acts like a second language. Macros ARE functions). So yay, another file down.

Next up, gy_types.h. This file mostly deals with #including stdint.h and defining my own naming convention for fixed size types like int32_t becomes i32, uint32_t becomes u32, float becomes r32, and so forth. Jai, luckily, has some built-in, fixed size, types with really similarly short names. Unfortunately i32 is actually s32 in Jai. And r32 is actually float32 (not very short). For now I think we’ll try using this naming convention and forego porting any of gy_types.h. If we want our own naming convention back, we can do a find and replace on our files later to convert from s32 to i32 and float32 to r32.

So that brings us to our first dependency level that has more than 1 file. I think that’s a good place to wrap up this blog post for now (running out of time to write this). In the next post we’ll take a look at vectors, rectangles, directions, and colors and see how these common type implementations map to Jai.

Const Port v1.6

Another version of Const Port is ready. And with this version comes the first release of the OSX version of Const Port. I've done a decent amount of work trying to get the OSX version to run inside an application bundle so hopefully it will be easy to install. That being said I know there are still a lot of things to learn about distributing on OSX. I am largely a windows user I must admit so if anybody has any suggestions I'm open to all critique.

I'm going to try and keep the explanation short here since I don't have a whole ton of time today. There are a couple of major features I'd like to point out in this version. The first is the new and improved Com Menu:

The menu layout has been redesigned to be more friendly to tall windows and a large number of available ports. It handles really small window resolutions quite nicely and it now opens on startup automatically.

Another nice feature that's been added is the input text box. Instead of echoing the characters directly to the port you can now type into an editable text box near the bottom of the screen. This allows for much nicer editing features and relieves the application of the burden of echoing characters and handling backspaces. It also allows us to record previously input lines and allow you to easily recall them using up/down arrows.

There's a lot of other small features and improvements and some major reworkings of some of the core platform functionalities as well. These have both fixed a lot of bugs and added a decent amount more. There's still a long list of things that I'd like to get fixed but I think in the interest of time I'd like to release it as it is. If you run into any bugs feel free to report them on the github, Handmade Network, or as a comment on this post and I will be sure to get them on the list if they aren't already.

Alright enough talk, here's the download links:

Changelog:

• Redesigned the Com Menu to handle various resolutions and long lists of available ports better
• Support for opening COM25-COM256
• Ctrl+W now closes the currently open port
• Escape key closes com menu and popup messages
• Added an input text box to allow better editing features and recall functionality. (Can be disabled in the configuration settings)
• Added support for regular expression triggers that evaluate after every character received
• Added a small running indicator to the bottom left side of the screen
• Added support for pasting from clipboard
• Added support for replacing the matched line with a format string that contains the captured groups in the regular expression
• Fixed a problem where we wouldn't render/respond to operating system events when receiving large amounts of data
• Added better indications for the window not being active
• You can now send binary data to the port by typing in the hexadecimal representation and hitting ctrl+enter (or hold ctrl while pressing the send button)
• Added support for line-wrapping
• Added support for a few for trigger effects (save, new_line, and clear_screen)
• Added better support for keys being held down (repeated key events)
• Added a few temporary check boxes in the overhead menu to toggle various options

Const Port v1.5

Finally I've gotten around to polishing off some new features in Const Port and making a version that can be released. There's a few major features added in this version and a lot of smaller things changed to the existing tool set, so let's get started explaining them.

The main feature I'd like to point out is the ability to run python scripts that interpret the input from the COM port and output data for Const Port to show. There are now a list of configuration options to set exactly how this works and what python script should be run. Then in Const Port if you hit ctrl+P it will attempt to start an instance of python and attach itself to the stdin and stdout so it can send and receive data from it. This means anything printed out by python will show up in the Const Port output window, and if you have the configuration set up correctly, any data received from the COM port will be piped to the python program for processing. There's still a bit of weirdness with how the stdio works in python so you'll have to do sys.stdout.flush() for any output to be pushed to the screen before the script has ended. For now there is an example python script called asciiHex.py that will take the com port data and output the hex values of the characters (spaces and new lines preserved).

Another major feature that has been added is the ability to define and customize your regular expression triggers to a far greater degree. You can now define as many regular expressions as you want each with it's own list of effects that should be applied should it match the line. This allows you to set more complicated formatting and line coloring than ever before. A list of possible effects can be found in the commented example expression. "new_line" and "clear_screen" have not been implemented yet but the rest of the effects work and more will be added as the project progresses.

We've also done a bit of an upgrade on the com menu to support disconnecting from the current port and configuration options for renaming the ports with any alias you want. As you can see in the screenshot above I have named my FTDI cable coming from my controller "PIC32MZ", and the other two ports that show up are for other devices on my system that I don't use so I've named them "System1" and "System2".

Another helpful feature that's been added to the platform layer is the ability to launch the default program to edit a specific file. This allows us to make better use of the gear button on the main menu to open up the GlobalConfig.json file for you to edit quickly and easily. Many of the options in the JSON file will have UI counterparts later so you won't have to leave Const Port in order to change the settings. However, for now this is a good option to make finding and opening the file easier. NOTE: If you're having trouble with it opening the file make sure you have a default program for the .json file type.

The last major thing I'd like to mention briefly is the vastly improved support for different themes. We've added a ton of new configuration color options so almost every piece of the UI color scheme can be set separately. Along with this we've made a mock "Neon" color scheme to show the potential:

So with all of those things out of the way, let's get that download link.

And, as before, there is also a debug version that will open the console window so you can see the debug output of the program if something is going wrong.

As a final note, one of the reasons this version has taken so long to release is a lot of the time has been spent on a OSX platform layer. I was planning on getting the layer usable before this release but we are still having some trouble with making the application bundle work and various configuration options like that so I've decided to release the changes as they stand now and wait till the next version before releasing the OSX variant. That being said, you can go download the source code and compile a fairly usable version for OSX if you would like. The majority of the progam's functionalities are working so it could be a very useful tool. You can download the source code below.

CHangelog:

• Added support for running a python instance with stdin and stdout routed to through Const Port so you can make scripts that interpret the output of the COM port or perform other tasks when certain things happen.
• Changed the way regular expression triggers are defined the configuration options to allow support for any number of triggers each with it's own list of effects that will be applied when they match the line
• Made the settings button open the configuration file for editing with the default program chosen by the operating system.
• Added an info window with a short description of the program that can be accessed by pressing the information button
• Removed the arrow button from the main menu
• Added options to switch the RX and TX LEDs between square and circular modes
• Added more options for how the RX and TX LEDs are sized
• Added a temporary implementation of routing the recieved text to a file without showing it on the screen. This can be started and stopped with the Ctrl+F shortcut. The file will be named after the COM port and saved in the current working directory.
• Added an option to make the application automatically open the saved file for editing when "Save to File" button is pressed
• Fixed some of the formatting problems with different font sizes so the buttons and windows should work a bit better with smaller or larger font sizes
• Added the ability to choose a separate font size for the main text versus the UI text elements
• Made the application regenerate the fonts when the configuration file is reloaded with Ctrl+R
• Added the ability to disconnect from the current port without connecting to another one.
• Added configuration options to allow renaming of ports with a user defined alias
• Added better support for different naming conventions of the platform layer for available ports
• Made the COM menu tabs more dynamic in how they choose to format their text to allow for longer names and various font sizes
• Added a lot of new color options to allow the user to change the UI theme to much greater accuracy
• Added an example "Neon" color scheme
• Made the rendering of the main menu button circles dynamic to allow for greater customization
• Made the rendering of the exit button on the menus dynamic instead of using a predefined image
• Changed the way the main menu's height is chosen to support a greater variety of options for the RX and TX LEDs and font sizes
• Changed the naming convention of the configuration options to be all lower case and underscores between words
• Added a file cursor that will always show at the end of the file. Disabled the hover cursor and hover line options by default
• Added an option to turn on auto-echo of input text  to free up the embedded application the responsibility of echoing the user's input

Const Port v1.4

Lately things have gotten really busy with the Eclipse and family visiting so it's taken me a little while to get this version ready to go.

I've added a couple new big features that I hope people are going to like. The first and foremost feature is configuration files. I've set up a nice system to allow you to change many of the features of Const Port through configuration files found in the Resources/Configuration folder.

There are 3 files currently. GlobalConfig.json, PlatformConfig.json, and RegularExpressions.rgx. All three can be opened and edited by a text editor of your choice. The .json files are parsed using the JSMN (pronounced "Jasmine") library which can be found here. The .rgx file is parsed using a simple custom syntax parser. There's a brief explanation of the file format in the file itself but overall it's a really straightforward way of allowing you to input regular expressions without having to escape them in JSON strings.

There's a large variety of things you can change about the application and I'm adding more options all the time. I would like to make as many things modifiable as possible. Later on I might decide to scale back and get rid of some of the unneeded options or put them in a separate file so they don't confuse people. However, for now I'm just going to keep throwing out new options and see what sticks. So if you have any options you'd like to see added just let me know.

Be warned: One side effect of having so many options is that I haven't tested all the options. I'm sure it's easy to crash the application by putting large numbers or weird combinations of options. Once we have a more solid list of options I will start bounds checking the inputs and confirming that the options are valid. For now however, just be nice with your inputs and you should be fine.

With the addition of options it's now entirely possible to change the color scheme of Const Port to whatever you like. The UI is rendered using a combination of various grays, blacks, and whites. However you can change these to whatever colors you like using a variety of JSON color formats.

The second major addition that was made was regular expression support. (Hence the RegularExpressions.rgx file) I've decided to try and use the C++ library Boost.Regex to allow me to add support for various features that will use regular expressions in the future. The main features right now are hard coded as configuration options in the GlobalConfig.json file. Each of these regular expressions is checked against each line that comes in (after we receive the new line character(s)) and if a match is found we produce some change. These have replaced the hard-coded line coloring methods and opened the line colors up for use with whatever input you'd like to color. You can only define 1 regular expression per color right now and they have to be the Highligh1-Highlight5 colors but this will change in the future to allow you to completely customize the look and feel of your input.

There's a few more features but I think I've gone on long enough. Here's your download link:

The debug version is the same as the regular but with the Console Window enabled so you can see all the debug output of the application. This is useful for debug purposes, especially if the application isn't running on your platform or crashes unexpectadly.

As always the source code can be downloaded from BitBucket:

Screenshots:

This one is just Sublime Text but it's an example of the GlobalConfig.json file

Changelog:

• Added JSON configuration files in the Resources/Configuration folder that contain a variety of configuration options for Const Port. Edit these at any time and then use the Ctrl+R keybinding to reload the configuration
• Added a RegularExpressions.rgx file in the Resources/Configuration folder. This file uses a custom syntax and is used in conjunction with the configuration files to allow you to easily define regular expressions without having to escape them in JSON strings.
• Replaced the hard-coded line coloring to be defined by the user using Regular Expressions
• Added an option for a regular expression that increments a generic counter. Use this to have Const Port count occurrences of some line for you
• Added an option for a regular expression that automatically adds a line break
• Fixed a bug where we weren't able to open COM ports above COM9
• Did some work to try and get Const Port  to run on Windows 7. Still need to do more work on this but I've made some improvements on the required OpenGL version

Const Port v1.3

It's time for another release of Const Port! Version 1.3 adds some cool new features and fixes a host of small bugs. As before you can download the compiled x64 Windows version:

or download the source code from the BitBucket Repository and compile it yourself:

Screenshots:

Changelog:

• The way the buffer is sized and worked with has been completely revamped to allow dynamically sized lines. This allows us to do a lot of cool things. The first of which is Time Elapsed Banners
• Added Time Elapsed Banners which will automatically show up after 15s or more have passed between lines. The minimum time is configurable in the code but cannot be configured after build in this version. Configuration options are coming soon.
• Horizontal scrolling has been added. You can use the arrow keys left and right or Shift+Scroll Wheel. There is no horizontal scrollbar in this version
• Added a Clear button in the upper right hand corner that does the same thing as Ctrl+Shift+C
• Added an End of Buffer button below the scrollbar
• Status messages in the bottom bar can now be delivered from anywhere in the program
• Tx and Rx indicators have been added in the upper right corner allowing you to easily see when data is sent or received
• Save to File button now appears when text is selected. Currently the output file is fixed in the program's directory with a timestamp appended on the end. Ctrl+S does the same thing as pressing the button
• Line marks are displayed slightly different. Thick line marks make the line spacing between lines a little larger to make the break a little more clear
• Escape key now closes the COM menu. Enter key hits the Connect button.
• Added support for COM13-COM24.
• Added a single frame flip right after the window is created to allow for a blank window to show even if the application DLL isn't loaded or rendered correctly. Also allows for the window to show sooner
• Fixed a problem with empty lines being left with a timestamp of 0

Const Port v1.2

Const Port version 1.2 is now finished. I added a 1.1 version to the bitbucket repository but there was only a few random features added so I didn't go through a full release announcement process. 

Version 1.2 is mainly focused around adding the ability to configure the UART settings (Baud Rate, Parity, etc.) through the COM menu. There's also a lot of other smaller features and fixes added on the side. See the changelog below for a full list of features.

The updated COM menu

Changelog:

[1.2]

• Reformatted the COM menu to include options for Baud Rate, Num Bits, Parity mode, and Stop Bits
• Changes to the COM menu are not applied until you press the Connect button
• Ctrl+[1-9]  now open COM port 1-9 rather than just an index in the list of available COM ports
• COM settings are now grouped in a single structure that can be passed around
• Re-enabled the colorful selection for fun (4th of July?)
• Changed the way COM ports are enumerated internally
• Set the supported OpenGL version to 3.0. Modified the shaders to work accordingly

[1.1]

• Added simple line marks. You can click on the line number gutter to add marks. Shift+click makes the marks slightly thicker.
• Ctrl+M marks the last line received (Ctrl+Shift+M for thicker mark)
• A readout in the status menu indicates time since last line was received
• When text is selected the readout indicates how many characters are selected
• Removed max size limit on the window
• Disabled the TOPMOST option when compiled for release mode
• Connected COM port is displayed in the window title and in the status bar

Const Port v1.0

So over the past year I've been working as a firmware developer and I've been frustrated with the lack of quality tools for tasks that I perform daily. I've toyed with the idea of making my own version of these tools but many of them are out of the scope of a one person development team.

However, I got the idea one day to make a better serial COM port reading application. I've tried using Putty, Tera Term, Real Term and many others, and they all have certain perks but each one comes with a heavy load of quirks and design problems. Most of them try hard to be an all-in-one terminal application that supports many types of text communications. Putty is a great tool if all you are doing is SSHing into a remote server, however when it comes to data stream reading it doesn't provide most of the features I would like. And some of the features I would expect of even the simplest text editor it has sacrificed at the hand of trying to be an All-In-One Application. 

So I decided to start making my own application. Since I spend most of my day reading debug output from my embedded applications it would be a huge time saver to have something that provides me with the functionality I need. And while I'm at it, it will provide a great platform for experimentation with features that I think would be fun or exciting.

I started working on it mid last week. I started by stripping the platform layer I had from my last project and some GLFW/GLEW code I had from the project before that and started creating a simple architecture. Like all of my projects recently it supports dynamically reloaded code with a clean line between application and platform layer. With OpenGL as the rendering technology we should be able to port it nicely to other platforms later, though for now it only works on windows. The whole thing is hand written, even the math functions for Matrices and Vectors are my own. The project compiles in C++ using the MSVC compiler however most of the code is written to C like standards.

After about a week of work I'm surprised how far the project has come. Much of the basic functionality came together quickly and I've even had time to add some of the nice features I've wanted to try. There's still a lot of things to be done before I can say it's releasable, however I feel like it already at a point that I can use it for real use at work.

The name is temporary, and likely to be changed in the future

I figure now that it is somewhat functional I will begin releasing versions for people to download if they would like. Keep in mind that the program DOES NOT support many of the features you might expect from a finished application. For example, currently you can only open a COM port at 115200, 8n1 settings since the UI doesn't have a place for configuration yet. If you'd like to change it, you have to recompile the application. However, since I am releasing the source code you are welcome to do so. You'll have to compile with GLEW and GLFW but other than that you should be able to get it working with only some minor path tweaks to the build.bat file.

So with all that said, if you'd still like to download it and check it out you can get a windows (x64) compiled version here:

Or you can find the source code on BitBucket: 

Since the application is so unfinished I can't guarantee that I'll be able to get to any bug reports any time soon. You can still log them on bitbucket if you'd like though. I also won't accept any pull requests. However if you have any feature suggestions I'm happy to hear them, you can email me at robbitay@gmail.com