I'm in the process of setting up a solid approach to loading YUI and other JavaScript components to a page and looking for some guidance on the approach(s) Yahoo! takes on it's various properties.

Some issues I’ve been thinking about related YUI Loading:

1. One concern I have is that YUI 3 introduces a level of granularity that makes loading of individual [non-aggregated] JavaScript files out of the question. e.g. I have a YUI 3 based app running and there are 20 HTTP requests for script files on the page, 19 of which are loaded via the YUI Loader.
   
   
2. Another concern I have is around ill-effects of combo-handling too much; i.e. if I load the YUI 3 modules via the Loader as a combined file on one page of my app, another page may have a slightly different set of required YUI modules therefore the client isn't able to leverage what was downloaded on page 1, and the code for most modules will be downloaded twice.
   
   
3. Related to combo-handling, is possible caching issues around the use of query-string-parameters for specifying modules. We know from the past that older browsers and proxies don't handle caching of resources with query string parameters all that well. What's the teams thoughts on this issue? Maybe the YUI Loader should have support for a custom module-separator when constructing combo-file-urls?
   
   
4. I’m also unsure of the best approach for loading my custom YUI3-based modules that are external to the library. Since external modules are not (and can't be when using the Yahoo CDN) combined into the single-combo file with the other YUI modules, I need some way to minimized requests when loading them as well.

After trying to talk through this loading stuff the best approach I could derive is sort of a hybrid using standard script tags and dynamically loaded scripts both using combo-handling.

It seems one way to deal with my second concern is to "seed" the page with more than the yui base module (yui/yui-min.js) and include all the common components used on every page of the application, setting sort of a base-line of modules. These modules with their dependecies could be written as a single script tag leveraging combo-handling.

Since there's bound to be some variations of required YUI modules between pages, the first, big-seed, script could be followed by the page's script file which would create a new YUI instance listed with it's dependencies. Any modules which are not already on the page from the big-seed script, the YUI Loader would add dynamically. The dynamically loaded scripts could also utilize combo handling to limit the impact on # of HTTP requests.

This setup would end up having 3 http requests for script files: 1 for the big-seed file, another for the page's JavaScript file, and the 3rd from the page-specific dependencies dynamically loaded via the YUI loader.

I would feel pretty good about where things are at this point. But still unsure about the best approach to add-in my custom YUI-based modules to the mix, and be ensured that my combo-handled resources will be cached properly.

I'm looking to you guys for suggestions. What's the standard formula at Yahoo! to get the YUI and app-specific scripts loaded right and fast?

Hello Eric,

I do believe we have a lot of room for improvements on the YUI Loader, but these concerns of yours are really hard to address, mainly because each project requires a different approach.

So, I will give me my point of view on your 4 bullets:

#1 ------:
I think we can tackle this defining a "lazyload" feature per YUI instance. So at the moment you create a new instance, defining new modules and using "use" to load the requirements, the system should wait a little bit before actually inserting the script tag in the page, just to make sure that the next block is not another instantiation process. If it's in fact another instance, we can merge the requirements, inserting a single script tag, instead of 2, and so on. I guess the first step should be delaying the loading process until the "domready" or "load". Merging all the initial instances thru a single script tag instead of 19 scripts tags. Here is an example:

YUI({modules: {a:{fullpath:'a.js'}}}).use ('a', 'node', 'io', function(Y) {});

...HTML CODE IN THE MIDDLE...

YUI({modules: {b:{fullpath:'b.js'}}}).use ('b', 'dd', 'event', function(Y) {});

The optimized result should be:
1. yui.js
2. combo?node+event+io+dd+other
3. a.js
4. b.js

instead of:
1. yui.js
2. combo?node-io-other
3. a.js
4. combo?event-dd-other
5. b.js

One of my side projects (bootstrap: http://github.com/caridy/bootstrap/tree/master) will address some of this issues, wrapping he loader functionality, and providing a similar API for yui 2.x as well. Feel free to dig in.

#2. ----------
I don't have a solution for this in my mind. For now we are focus on each page as an individual cacheable asset. But yes, reusing combos from page to page will be great. Probably the solution for #1 can help a little bit on this too. And the other option is the hybrid solution. Instead of including yui.js you can insert a combo url with a base set of components used across all the pages, and complementing the requirements with the loader per instance.

#3. ------
I don't think this is a real problem. Mainly because we are targeting A-Grade browsers in which the combo works great.

#4. ------
Definitely this is something we can improve, especially if you host a combo-like solution. Loader should be able to group modules by combo instances. So, when you define a custom module, you can specify a custom combo, and loader should compute the dependencies for that particular combo. This is especially important for the new granularity approach.
Another approach should be a hosted solution, in which developers can host scripts and use them thru the current combo, combining the files with the YUI files. This is a little more complex, and I think it's a good option for certified plugins and extensions. I mention this approach to Eric Miraglia few weeks ago.

-------------------
P.S.: sorry for the huge delay in my answer for your last comment on my blog

Best Regards,
Caridy

This is an interesting idea, and I thought about this a bit when dealing with the concurrent loader request problem. While there may be some applications that could benefit from being built this way, I have doubts that this approach could be used generically. Reducing the total request time isn't the goal -- the goal is to give the user the best experience, which means we need to get the content and the UI enhancements available to the user as soon as we can. If the delay you impose in order to combine multiple requests is longer than the time it takes to do a request, the end result may actually be worse for the user even if the aggregate time for pulling the resources is lower.

One main goal is for my applications to have portability around YUI hosting locations; specifically, I'm looking to have [server-side] runtime configuration setting whether YUI is pulled from the Yahoo! CDN or our own servers. I wouldn’t want to use a custom build of YUI or a custom hosting solution which wouldn’t also work when pulling from the CDN.

Adam, I see your point about the introduction of inter-module roll-up files coming once it’s determined the common sets of modules people are using together.

I imagine that most stuff I’m building will have a JavaScript file for each page which will act as the “main” program keeping everything under one YUI instance. Making me not too worried about how to deal with multiple YUI instances on the same pages loading a similar set of modules. Adam I believe I saw some commits go through related to a bug on this issue. I’m guessing that the YUI library will need to have a pretty solid-level of support around multiple YUI instances in conjunction with Caja-style sand-boxed components living together on the page.

Two areas of improvement on the Loader would be:

1. moduleSeparator property that would default to “&” but be configurable per YUI instance. This would help support different combo-handling strategies other than query-string parameters.
   
   
2. Pull out the modules metadata into a JSON file which could be injected into the Loader at build time, but would also be copied over as file during the build process as well. This would allow server-side code to parse the metadata about the YUI module dependencies.

I was hoping to get some details from you guys at YUI about how Yahoo! FEs implement script loading on different types of YUI properties…

Since Yahoo! properties have a combination of YUI, property-specific modules, and page-specific runtime code I’m very interested in the different approaches used for single-page Ajax-y applications like Yahoo! Mail, vs. JavaScript-enhanced properties like Yahoo! Sports.

I've added an enhancement request to: Add combo-module-separator property to YUI Loader to support combo-handling strategies other than query-strings.