Web Page Changes to avoid ESP code knowing about UI

I started this thread on discord but it will be easier to discuss here. The settings pages get the settings data injected in the GetV() javascript function. The GetV() javascript function is built up and injected into the page using ESP code. This is done in xml.cpp using the getSettingsJS and sappends functions. This tightly couples the form fields to server side rendered logic. As WLED evolves, this complicates future changes. I propose eliminating the injection of content from the ESP side into pages. A better approach may be to expose the cfg.json data or similar to the browser for data binding. I have been prototyping a number of these changes.

  1. Allows the gz compression of all pages, javascript, stylesheets to reduce flash size. For example “settings_leds.htm” is about 23KB, gzipped it goes to 6.8 KB. This is the biggest setting file.
  2. Allows extracting javascript and style sheets into separate files for source control. Allows for linting of these sources and cdata.js can still inline/combine them if required to reduce web requests on esp8266.
  3. Allows generating all the C code to serve up each of the files, checking etags, etc - WLED/cdata.js at 28daefc856a293682f10c4c210da92b813e60982 · pbolduc/WLED · GitHub
  4. Can simply the cdata.js script to remove the writeChunks parts and replace with writeHtmlGzipped
  5. cdata.js can dynamically scan the user interface directory to call writeHtmlGzipped for each web site file
  6. Enhance or change cdata.js script to use a task runner like Grunt. I know grunt is not the cool way things are done. However, will be less complex than using webpack. For example, projects like node-red (GitHub - node-red/node-red: Low-code programming for event-driven applications) use Grunt. The developer experience would be the same ‘npm run build’ or ‘npm run dev’
  7. Allows creation of better etags based on the content and file modification time instead of WLED version - WLED/cdata.js at 28daefc856a293682f10c4c210da92b813e60982 · pbolduc/WLED · GitHub
  8. Enables better development experience by using a local web server that can proxy data and web socket requests to a another WLED instance - WLED/wled.js at change-settings-pages · pbolduc/WLED · GitHub
3 Likes

This would allow much more than what you wrote.
And replacing xml.cpp is a desired end effect.

I think it’s a great idea.

would it make sense to design a single “/configuration” json endpoint that would return the contents of cfg.json? Allow callers to determine what their configuration editor needs and submit the changes back. Ensuring changes to the cfg.json are versioned so that updates do not break saved previous versions. ie you flash from 0.12 to 0.14, the old configuration shouldn’t break the ui. However, similar to tasmota, you have to provide an upgrade path and not try to support all older configuration versions indefinitely.

Without hobbling over to the computer to check, I think most values are already either in /json/state and /json/info. I’d suggest adding /json/config pulling the values that can be changed from info to config and info would be truly informational; version, uptime, WiFi signal, etc. And state would just be about effects, palettes, colors etc.

There is /json/cfg but is intended just for saving configuration.
Reading should be done by loading cfg.json and parsing it.

This is pretty much 100% of what I’m planning, so I’m super glad we all agree this is the way to go forward with settings. I’ve wanted to do this ever since storing settings in a JSON file rather than EEPROM, so now is the perfect time!

@pbolduc Thank you for all your work prototyping some of this! :sparkles: In csdata.js, we can most likely handle this just like the index page, having separate HTML/CSS/JS sources, and merging, minifying and gzipping them upon request. Perhaps even multiple .js sources would be beneficial, since right now LED settings alone sports ~400 lines of JS.

I’ve also been thinking about how to bind JSON values and settings fields. In UI settings, I even already experienced with auto-generating all form fields based on the JSON. My conclusion was that I don’t really think it’s worth it, since there are too many factors not defined in the cfg.json file (what are the upper and lower bounds of this number? Is it a float? How many characters may this string have? Should this option even be displayed to the user without conversion?)

Right now I think the easiest way would be to still add all form fields (except obviously usermod settings, since they are ESP-provided in the u object of cfg.json) hard-coded in HTML and bind them to JSON via their ID or HTML data attribute matching the JSON structure (e.g., the field for the WiFi SSID would be nw_ins_ssid). This should be rather lightweight, after fetching the JSON, JS can easily look for the right fields and populate them. Similarily, when changing a value, an oninput listener could be used to first validate and then modify the value of the JSON. This is obviously pretty simplified, but just a concept for now. What do you think about it?

The technical change from ESP-injected values to JSON + static minified gzipped page will also coincide with a major settings UI overhaul, together forming the mainline 0.14 feature.

GET /json/cfg just returns cfg.json, they are thus interchangable, but the former is a more “official” endpoint.

Here are a few thoughts I had about the new page already:

I’ve also just made the first commit to the new_settings branch, though it isn’t noteworthy just yet.

1 Like

It would be great if we can try to focus on using ES6 modules. Also we can use some create some helper modules that make things easier. For example, I took a portion of the micro jquery like framework chibi to simplify down the cfg.js file. When using modules, things stop being on global scope and things like tree shaking and minification can be used to make the code really small. We can stop trying to code small to keep the javascript small and allow the minifier to do it’s job. here is a quick non-finished example,


import { lang } from './cfg_lang.js'

function setLabel(elm) {
	const id = elm.id;
	const label = lang.labels[id];
	elm.textContent = label ? label : id;
}

//startup, called on page load
function S() {
  $('.l').each(setLabel); //populate labels
}

//toggle between hidden and 100% width (screen < ? px) 
//toggle between icons-only and 100% width (screen < ?? px)
//toggle between icons-only and ? px (screen >= ?? px)
function menu() {

}

S();
1 Like

My stance on modules/frameworks was always a capital, bold NO, unless it was something that could not practically be implemented quickly (so far this was only iro.js (colorpicker) and rangetouch, which “magically” fixed an issue with the sliders in the UI on iOS).

They have the potential to make development easier, but at the cost of binary size, which is not acceptable in WLED’s use case. This might sound unconventional, and I know the resulting code can get unwieldy quickly, but I love programming in vanilla JS.

Chibi framework seems like something I could make an exception for though. 3kb gzipped sounds reasonable. It might even have the potential to save more space than it takes up.

Maybe we can make our own modules to split up the code into several JS files - that would definitely make it easier to maintain. I will read up on ES6 modules as I’m not yet very familiar with the concept :slight_smile:

Having less globals would definitely be great both for maintaining and size since cryptic “manual minification” names like sLC could be replaced with something more descriptive like busLengthSum and still minified to maybe even single-letter variable names.

There is a bunch of extra stuff in chibi to support old browsers like IE8 and FF < 8. All this stuff can be yanked out and we can use modern browser features. I have started to create a minimal version with functions we may need and remove the old browser work arounds.

This is an example of dynamically creating the menu: WLED/cfg.js at new-settings-es6 · pbolduc/WLED · GitHub

Reuse of common UI manipulation will be worth the use. I used Babel · The compiler for next generation JavaScript and UglifyJS 3: Online JavaScript minifier to convert and minimize the js. The current dom.mjs (module js file), gzip compresses to about 570 bytes (mjs = 2,880 bytes, ugilfy = 1,169 bytes, gzip = 570 bytes)

In my example, I am loading the language labels using a module, however, I think you may be wanted to fetch the UI language dynamically from storage. This way the language could be changed dynamically by uploading a UI language json file following the accept language headers.

cfg_lang-de.js cfg_lang-en.js cfg_lang-fr.js

There could be a way to fetch these files from github via the browser and save to flash. It could also be challenging as new UI options are added to have a customized language file updated in sync. May be useful to use English as the default compiled in language and override any language elements found in flash language file. Also providing an “update translation file” feature where the browser pulls from github and posts new language file to be saved to flash.

May also want to consider instead of mapping UI elements directly to translated words, it may make sense to map a normalized words key to the translated word. So if you wanted to remap the word Schedules to another language, you do something like (sorry if Google Translate messed this up) this in the translation file. Only those words that need to be translated need to be listed in the language specific file.

“Schedules”: “Zeitpläne”

It could be: UI Element maps to normalized “English Value”, normalized “English Value” may map to language specific translation. If no translation mapping is found, the normalized “English Value” would be used. This may be how were thinking already.

1 Like

I should also note that using type=module <script src="cfg.js" type="module"></script> will not work when loading files using file:/// only via http://

This is why I have tried to build the WLED Dev Server on express for development purposes.

Have a prototype of translation into other languages in my branch GitHub - pbolduc/WLED at new-settings-es6

I tagged the required elements that need translation using class ‘l10n’. Per Localization vs. Internationalization, Localization is sometimes written in English as l10n, so it seemed fitting and unlikely to conflict with something else. Using this, we find all nodes with this class, look up the text in the node for a matching item in the translation table. If found, it is replaced. If not, nothing changes. This way not every UI element needs a translation, only those that need replacing. The main mark up stays in English, which makes it easier to maintain cause the words are there.

1 Like

When we serve up strings.json WLED should wrap it in a self executing function like in my poc. This way, the file can be loaded from a script tag at the top of the page. This should ensure the translations are loaded right away and are available on the window object. I tried to do a fetch from javascript, but I always got a UI update flash. Even with self executing function, if you refresh quickly, you can still see update flash but it is only for around 0.2ms. It would probably get worse as the DOM grows.