Web Page Changes to avoid ESP code knowing about UI

I started this thread on discord but it will be easier to discuss here. The settings pages get the settings data injected in the GetV() javascript function. The GetV() javascript function is built up and injected into the page using ESP code. This is done in xml.cpp using the getSettingsJS and sappends functions. This tightly couples the form fields to server side rendered logic. As WLED evolves, this complicates future changes. I propose eliminating the injection of content from the ESP side into pages. A better approach may be to expose the cfg.json data or similar to the browser for data binding. I have been prototyping a number of these changes.

  1. Allows the gz compression of all pages, javascript, stylesheets to reduce flash size. For example ā€œsettings_leds.htmā€ is about 23KB, gzipped it goes to 6.8 KB. This is the biggest setting file.
  2. Allows extracting javascript and style sheets into separate files for source control. Allows for linting of these sources and cdata.js can still inline/combine them if required to reduce web requests on esp8266.
  3. Allows generating all the C code to serve up each of the files, checking etags, etc - WLED/cdata.js at 28daefc856a293682f10c4c210da92b813e60982 Ā· pbolduc/WLED Ā· GitHub
  4. Can simply the cdata.js script to remove the writeChunks parts and replace with writeHtmlGzipped
  5. cdata.js can dynamically scan the user interface directory to call writeHtmlGzipped for each web site file
  6. Enhance or change cdata.js script to use a task runner like Grunt. I know grunt is not the cool way things are done. However, will be less complex than using webpack. For example, projects like node-red (GitHub - node-red/node-red: Low-code programming for event-driven applications) use Grunt. The developer experience would be the same ā€˜npm run buildā€™ or ā€˜npm run devā€™
  7. Allows creation of better etags based on the content and file modification time instead of WLED version - WLED/cdata.js at 28daefc856a293682f10c4c210da92b813e60982 Ā· pbolduc/WLED Ā· GitHub
  8. Enables better development experience by using a local web server that can proxy data and web socket requests to a another WLED instance - WLED/wled.js at change-settings-pages Ā· pbolduc/WLED Ā· GitHub
3 Likes

This would allow much more than what you wrote.
And replacing xml.cpp is a desired end effect.

I think itā€™s a great idea.

would it make sense to design a single ā€œ/configurationā€ json endpoint that would return the contents of cfg.json? Allow callers to determine what their configuration editor needs and submit the changes back. Ensuring changes to the cfg.json are versioned so that updates do not break saved previous versions. ie you flash from 0.12 to 0.14, the old configuration shouldnā€™t break the ui. However, similar to tasmota, you have to provide an upgrade path and not try to support all older configuration versions indefinitely.

Without hobbling over to the computer to check, I think most values are already either in /json/state and /json/info. Iā€™d suggest adding /json/config pulling the values that can be changed from info to config and info would be truly informational; version, uptime, WiFi signal, etc. And state would just be about effects, palettes, colors etc.

There is /json/cfg but is intended just for saving configuration.
Reading should be done by loading cfg.json and parsing it.

This is pretty much 100% of what Iā€™m planning, so Iā€™m super glad we all agree this is the way to go forward with settings. Iā€™ve wanted to do this ever since storing settings in a JSON file rather than EEPROM, so now is the perfect time!

@pbolduc Thank you for all your work prototyping some of this! :sparkles: In csdata.js, we can most likely handle this just like the index page, having separate HTML/CSS/JS sources, and merging, minifying and gzipping them upon request. Perhaps even multiple .js sources would be beneficial, since right now LED settings alone sports ~400 lines of JS.

Iā€™ve also been thinking about how to bind JSON values and settings fields. In UI settings, I even already experienced with auto-generating all form fields based on the JSON. My conclusion was that I donā€™t really think itā€™s worth it, since there are too many factors not defined in the cfg.json file (what are the upper and lower bounds of this number? Is it a float? How many characters may this string have? Should this option even be displayed to the user without conversion?)

Right now I think the easiest way would be to still add all form fields (except obviously usermod settings, since they are ESP-provided in the u object of cfg.json) hard-coded in HTML and bind them to JSON via their ID or HTML data attribute matching the JSON structure (e.g., the field for the WiFi SSID would be nw_ins_ssid). This should be rather lightweight, after fetching the JSON, JS can easily look for the right fields and populate them. Similarily, when changing a value, an oninput listener could be used to first validate and then modify the value of the JSON. This is obviously pretty simplified, but just a concept for now. What do you think about it?

The technical change from ESP-injected values to JSON + static minified gzipped page will also coincide with a major settings UI overhaul, together forming the mainline 0.14 feature.

GET /json/cfg just returns cfg.json, they are thus interchangable, but the former is a more ā€œofficialā€ endpoint.

Here are a few thoughts I had about the new page already:

Iā€™ve also just made the first commit to the new_settings branch, though it isnā€™t noteworthy just yet.

1 Like

It would be great if we can try to focus on using ES6 modules. Also we can use some create some helper modules that make things easier. For example, I took a portion of the micro jquery like framework chibi to simplify down the cfg.js file. When using modules, things stop being on global scope and things like tree shaking and minification can be used to make the code really small. We can stop trying to code small to keep the javascript small and allow the minifier to do itā€™s job. here is a quick non-finished example,


import { lang } from './cfg_lang.js'

function setLabel(elm) {
	const id = elm.id;
	const label = lang.labels[id];
	elm.textContent = label ? label : id;
}

//startup, called on page load
function S() {
  $('.l').each(setLabel); //populate labels
}

//toggle between hidden and 100% width (screen < ? px) 
//toggle between icons-only and 100% width (screen < ?? px)
//toggle between icons-only and ? px (screen >= ?? px)
function menu() {

}

S();
1 Like

My stance on modules/frameworks was always a capital, bold NO, unless it was something that could not practically be implemented quickly (so far this was only iro.js (colorpicker) and rangetouch, which ā€œmagicallyā€ fixed an issue with the sliders in the UI on iOS).

They have the potential to make development easier, but at the cost of binary size, which is not acceptable in WLEDā€™s use case. This might sound unconventional, and I know the resulting code can get unwieldy quickly, but I love programming in vanilla JS.

Chibi framework seems like something I could make an exception for though. 3kb gzipped sounds reasonable. It might even have the potential to save more space than it takes up.

Maybe we can make our own modules to split up the code into several JS files - that would definitely make it easier to maintain. I will read up on ES6 modules as Iā€™m not yet very familiar with the concept :slight_smile:

Having less globals would definitely be great both for maintaining and size since cryptic ā€œmanual minificationā€ names like sLC could be replaced with something more descriptive like busLengthSum and still minified to maybe even single-letter variable names.

There is a bunch of extra stuff in chibi to support old browsers like IE8 and FF < 8. All this stuff can be yanked out and we can use modern browser features. I have started to create a minimal version with functions we may need and remove the old browser work arounds.

This is an example of dynamically creating the menu: WLED/cfg.js at new-settings-es6 Ā· pbolduc/WLED Ā· GitHub

Reuse of common UI manipulation will be worth the use. I used Babel Ā· The compiler for next generation JavaScript and UglifyJS 3: Online JavaScript minifier to convert and minimize the js. The current dom.mjs (module js file), gzip compresses to about 570 bytes (mjs = 2,880 bytes, ugilfy = 1,169 bytes, gzip = 570 bytes)

In my example, I am loading the language labels using a module, however, I think you may be wanted to fetch the UI language dynamically from storage. This way the language could be changed dynamically by uploading a UI language json file following the accept language headers.

cfg_lang-de.js cfg_lang-en.js cfg_lang-fr.js

There could be a way to fetch these files from github via the browser and save to flash. It could also be challenging as new UI options are added to have a customized language file updated in sync. May be useful to use English as the default compiled in language and override any language elements found in flash language file. Also providing an ā€œupdate translation fileā€ feature where the browser pulls from github and posts new language file to be saved to flash.

May also want to consider instead of mapping UI elements directly to translated words, it may make sense to map a normalized words key to the translated word. So if you wanted to remap the word Schedules to another language, you do something like (sorry if Google Translate messed this up) this in the translation file. Only those words that need to be translated need to be listed in the language specific file.

ā€œSchedulesā€: ā€œZeitplƤneā€

It could be: UI Element maps to normalized ā€œEnglish Valueā€, normalized ā€œEnglish Valueā€ may map to language specific translation. If no translation mapping is found, the normalized ā€œEnglish Valueā€ would be used. This may be how were thinking already.

1 Like

I should also note that using type=module <script src="cfg.js" type="module"></script> will not work when loading files using file:/// only via http://

This is why I have tried to build the WLED Dev Server on express for development purposes.

Have a prototype of translation into other languages in my branch GitHub - pbolduc/WLED at new-settings-es6

I tagged the required elements that need translation using class ā€˜l10nā€™. Per Localization vs. Internationalization, Localization is sometimes written in English as l10n, so it seemed fitting and unlikely to conflict with something else. Using this, we find all nodes with this class, look up the text in the node for a matching item in the translation table. If found, it is replaced. If not, nothing changes. This way not every UI element needs a translation, only those that need replacing. The main mark up stays in English, which makes it easier to maintain cause the words are there.

1 Like

When we serve up strings.json WLED should wrap it in a self executing function like in my poc. This way, the file can be loaded from a script tag at the top of the page. This should ensure the translations are loaded right away and are available on the window object. I tried to do a fetch from javascript, but I always got a UI update flash. Even with self executing function, if you refresh quickly, you can still see update flash but it is only for around 0.2ms. It would probably get worse as the DOM grows.

Just picking up on this again, sorry for the long idle period. I have started familiarizing myself with ES6 modules and believe they would be an amazing fit for this project since functions and vars are not exported unless explicitly stated and thus will do wonders for minifiability and also make developing easier by modularizing code into different files.

Thank you for the great POC @pbolduc! If this is OK for you, I would merge your changes back and use it as the new starting point.

Now we just need to find a way to have npm run build / cdata.js neatly minify all HTML/JS+modules/CSS into a single cfg.html file / gzipped C array for serving by the ESP. Perhaps webpack is a good option.

One thing I still need to figure out regarding internationalization is how to add variables to localized strings, perhaps printf-style, and to differentiate between singular and plural in the translations.
One way might be JS template strings.
That way, e.g. ā€œfor most effects, ~${current}A is enoughā€ could be localized to ā€œfĆ¼r die meisten Effekte reich${current===1.0?ā€œtā€:ā€œenā€} ~${current}A ausā€.

IMO xml.cpp is/will still be useful but used as a cfg.js generator instead of AsyncWebServer substitution callback.
My fork has this implemented and could also allow usermods to communicate form data/validation to settings pages.

As far as localization goes, there are plenty of existing approaches. I would recommend to make JSON files with translations which can be uploaded to file system. If the file does not exist just use default language (En).

I fail to see a good benefit of this that would justify the overhead of keeping xml.cpp instead of just fetching /cfg.json for all settings, like you have already done for Usermod settings.
The plan is to get rid of xml.cpp/set.cpp (excluding HTTP API) ever since Filesystem config was added :slight_smile:

Usermod settings metadata is a valid point, Iā€™d propose a separate JSON /umcfg endpoint with a structure that could look like this to solve that challenge:

{"Temperature":{
  "read-interval-s":{"name":"Reading interval","unit":"s","type":"number","min":1,"max":1000,"step":1}
  "degC":{"name":"Celsius degrees","type":"checkbox"}
}}

Fully agreed. Translation files can either be served from FS or GitHub.
Just the variable part is what I need to figure out:
To generate, based on a var, the strings

There are no trees
There is one tree
There are 2 trees

supporting something like

`There ${n===1?"is":"are"} ${n===0?"no":n===1?"one":n} tree${n===1?"":"s"}`

is easy to implement, but not having inline JS would both be way easier for translators and pose less of a security risk of arbitrary JS somehow ending up in a translation file.
Some syntax like this would probably be optimal, as it saves space and is easier for translators to deal with:
There {n?is|are} {n?no|one|n} tree{n?|s}
(in case of one pipe left would be for value 1, right for other values, in case of two pipes, left would be for 0, middle for 1, and right for all other values of n)
Could even be simplified down to:
There {is|are} {no|one|n} tree{|s}
in case there is only a single var.
And in case some var just needs to be inserted without singular/plural adjustment, that is easy too:
WLED version {n} (Build {m}) is installed

Iā€™ve done localization in one of my first (mostly solo) projects in 1999.

Will look up the code and post it.

1 Like
/**************************************
* koncnica - return word ending (plural form) for literal numbers
*--------------------------------------
* @parameter integer - input number
* @parameter string - comma separated entries wor word endings
* @returns string - word ending corresponding to input number
* ex: 1 zmaj - echo $zmaji . " zmaj(" . koncnica($zmaji," ,a,i,ev") . ")"
* ex: 2 zmaj(a) - echo $zmaji . " zmaj(" . koncnica($zmaji," ,a,i,ev") . ")"
* ex: 3 zmaj(i) - echo $zmaji . " zmaj(" . koncnica($zmaji," ,a,i,ev") . ")"
* ex: 7 zmaj(ev) - echo $zmaji . " zmaj(" . koncnica($zmaji," ,a,i,ev") . ")"
**************************************/
function koncnica($kolicina=0, $koncnice=" ,a,e,ov")
{
	$koncnice = explode(",", $koncnice);
	switch ( $kolicina % 100 ) {
		case 1:  $Koncnica = $koncnice[0]; break;
		case 2:  $Koncnica = $koncnice[1]; break;
		case 3:
		case 4:  $Koncnica = $koncnice[2]; break;
		default: $Koncnica = $koncnice[3]; break;
	}
	return trim( $Koncnica );
}

Example uses Slovenian language.

Oh okā€¦ seems like my idea of singular and plural was too simple :sweat_smile:

So youā€™re telling me there are 3 plural forms, one for quantity 1, another for 2, then another for 3-4 and another for all other values, but then for some reason 101 is singular again?
Maybe I should start learning Slovenian, this sounds super interesting :smile: